Having caught everything, I keep on hand exactly two types of bait. (versatile lure and metal spinner)
I'd much rather have inventory that's more useful than for storing things I will literally never touch again.
OP:
I likewise have a one retainer holding onto materia + relic items. First of all, you shouldn't be keeping all those crafter souls... actually I don't even know if they allow those to remain useful. You're only ever supposed to be allowed to have 3 crafters designated as specialists, and if you ever need to change that, you need to trade in the designated soul with a separate stackable item to swap specialist. The stackable item is all you ever need (and that rarely), just make things specialist and stop holding onto whatever you have in regards to that.
Materia: Grade 1-3 combat materia are good for ARR relics. Grade 4 can be nice for that, but will often be unnecessary (unless you're trying to give relics "good" stats, but there is zero point to that at this time). Sell/trash all of your gathering/crafting materia. These are only useful at "endgame" stages, and only for the highest tier hardest items. Most players will find no use or value from them. And even if you do find you need/want them, they are far, far easier to reacquire with scrips you can get in abundance.
Faded scrolls can start piling up, definitely worth getting to those eventually if you feel more inventory crunch. But it is a pain getting to them as soon as you get them, so having room to stash a few is nice. Just don't go stashing too many copies, or holding onto ones for scrolls you've already made; very few are really worth selling at all.
Sell all your seeds. Especially if you're just holding onto whatever you gathered, it's clearly easy to gather them. Only a handful are likely of any real use/value, and only if you're actually gardening. Apartments can't do much fancy with that anyway, the important stuff is all garden bed cross breeding.
Your crafting mats are probably your biggest problem, especially if you're going with the approach of wanting to craft everything in the crafting log. Your inventory is going to suffer for this, and for no good reason. Check marks and achievements, all so you're left with piles of items nobody wants. You either realize crafting everything is a bad idea and only keep materials for crafts you might actually use (minions, bardings, Extreme trial weapons), or you decide you're going to dedicate to completing the crafting log, and hopefully recognize you are almost certainly underestimating the undertaking.
Eventually, you're probably going to have to start asking yourself if you want to buy one extra retainer to start storing all of your raid tokens, as it very much sounds like you'll end up trying to hold onto all of them. I did. One extra retainer isn't too burdensome, personally. I don't think I could possibly find enough use to justify more than that, tho.