My brain genius theory is that Crystal Tower itself is one single structure existing in every (possible) shard at once.