They've mentioned that inventory attributes to load times whenever you enter or exit an instance. I believe they said retainer stuff counts too, though I may be mistaken on some other issue.
So if your system is rather low end, or a PS3; perhaps having little amount of junk and gear is lowering the memory load on your character, causing smoother transitions.