I had the exact same thought every time I'm on my way to the retainer. Working as intended, however, so it's unlikely. Just keep the stuff you commonly use in your inventory for now.