Our killing primals isn't unique and only something we can do. The primals have been defeated before without us and our blessing. What makes us unique is that we are immune to being tempered, not that we can kill primals. The company of heroes who we had to do all those quests for back when we were still an unknown had put Titan down. It's even stated in the Ifrit cutscenes that the Flames and Blades could put down Ifrit, just that more lives would be lost as they have to kill anyone that gets tempered.

Also our level isn't an argument, levels are a game mechanic that is only something that is there as a baseline to set content difficulty. Even using that argument just a quick glance through the regions near a nation show guards come in more than a single level range, so who is to say the ones inside the palace wouldn't be of a level that would make them at least mobs in an equal level dungeon to us. The imperial unit inside the Keeper of the Lake are from the same base we ride through, and yet they are certainly stronger than those who are at best an annoyance when they hit us and slow down our ride.