Oh the time differences themselves don't bother me one bit, I am completely aware their main purpose is for their plot device utility, and I think they are rather clever plot device that can allow for a lot of potential creative routes.
I am more just befuddled in trying to extrapolate the implications of what that means for how the Shards would interact with the rest of the cosmos/outer-space (even if it is all purely hypothetical stuff, that will never really be relevant in-game), if the notion that the Shards don't have their own outer-space/cosmos synced to their unique plane of reality is true, and that there is only one outer-space/cosmos that is synced to the The Source.