Ok lets have a think.
140 regular invintory slots
70 saddle bag
175 per retainer, 2 retainers
-------
50 weapon slots
50 ring slots
35 slots for everything else (315 total for other gear slots)
415 total gear slots from the armory chest
That is the total amount of inventory someone could have without buying more retainers, we can exclude the armory though as it doesn't factor into where bait can be held.
The main issue I see from this is not the fact that you have 560 total slots to put things in, it is the fact that while fishing, questing, or doing most things in the game you can't have access to your retainers which is a large portion of the potential slots you have to fill. At the end of the day you only have your bags and maybe the saddle bag as your "real" inventory and the rest should be considered storage. Even then though I would argue that someone could still say the saddle bag should be lumped in with retainers as you can't hand in quest items from your SB, you can craft items with mats from your SB or do a number of other things. Sure the SB goes with you where you go (other than instanced content) but it still has many restrictions on it.
Now here is where it gets kinda messy, with all of the different currencies in the game someone could be put off by the fact that there is just such a stupidly low amount of inventory space to begin with. You have Eureka and Bozja with their own items and currencies within their zone but you also have things like Bozjan coins which can be put in your retainer or SB but you would have ZERO access to them while actually in Bozja/Zad. This is bad design and needs to be changed. Of that I think everyone can agree. The fact of the matter with that is you could potentially use up 5 inventory slots of just the different types of coins. Then you have things like relic items, sure you can always put them in your SB when you are done farming but you can't complete quests while they are in your SB. Depending on which expansion you are farming for items in the amount could be a few (EW) to a **** ton (ShB).
I could go on but I'm sure I'm close to the character limit by now but I'll just say I don't think it is a matter or having too few or too many slots more that it is the fact that the retainer and SB don't act just like regular inventory slots. If anything I would like for them to link retainer and SB into your regular bags so that there are just multiple tabs you can switch through, much like how the Key Items & Crystals tab works now.