Did I mess up my first retainer?

[Niwashi]

If you were previously just using your retainer to store stuff and not for ventures, then he/she didn't have any levels to lose, since it's ventures that give your retainers XP. They don't level just because you are.

My advice, leave the retainer you've set to Botanist as a Botanist, but get a second retainer (you'll want another retainer's worth of storage space as you get into crafting anyway) and make that one your Bard retainer. Send them both out on ventures to level them up until they catch up to you in level. (The longer 18 hour exploration ventures in particular give a lot of XP.)

They won't be able to exceed your level in their respective classes, though. Nor in the targeted hunting/mining/botany/fishing ventures will they be able to gather items you haven't gathered yourself first (though in exploration ventures they can). So there's an extra incentive to be sure you gather each item in your gathering log at least once.