The cracks in the Byzantium and the Silurian city both led to the end of the universe, or to the time and place that caused it, at least (which is how the Doctor was able get a temporal fix on when and where it happened, and retrieved that 'object' that is implied to be involved in it). The other cracks that Zero and the fish-people passed through, were between points in the universe.
I see it this way - when you strike a piece of toughened glass with a hammer, the greatest concentration of cracks is around the point of impact. The cracks spread outwards from that point.