Well~ consider the possible story base behind that. Who's to say your retainer is suited to traveling out to meet you at a camp. They're simple servants who live in the city. That could be dangerous for them.

Anyways, I agree that they should be left to towns. It would be convent in a camp, but it encourages a certain lack of depth I find hard to describe. I feel like it would also nurture various other bad habits from all sorts of groups, including the RMT. I imagine a whole system of them making low value, easy item drop farming. Or possibly even base craft item farming.

Another issue I'd find with it is the speed of rare item hoarding. I like the competition of a long wait NM, as well as the simplicity of the new lower tier NM's they've added. But if there were retainers at every camp, I can imagine the ability for someone to horde an NM in which they have the ability to quickly attack, defeat, obtain, store and get back again.