For the Load time what they can do is ONLY integrate this Giant Storage Chest in the Mog Garden, like this when you go to any Mog House its not even there and it wont load.
But if you really need your items and you want to take the time to change jobs, prepare your gears etc you go to your Mog Garden and you access it.

You come back after a longday of farming, events, quests, missions etc and you end up with so many of same items in different storage areas..so
do you know how many duplicated items everyone has in mog safe 1 mog safe 2, locker, case, satchet, inventory etc..

But if we had a giant storage we could toss all of them in there and auto sort it to get stacks and then sort them in different bags..