I believe the way the servers are set up, each Mog Wardrobe you have actually has an increased server load, which is why it takes so long for your items to show up every time you zone.