Might be a bit of a strange thought, but what if each shard isn't just a single world, but an entire universe? So, each and every shard would have it's own world where Midgardsormir originated on? In some of these universe-shards, he'd go to the world that's the universe's main world.