One thing you gotta understand is even if we beat the primals that doesn't mean we have the power of 1000 men, we might have the power of 100 or less but the one mayor advantage we had with the primals was our immunity to being tempered i bet many npc could of beat a primal with immunity to temper, and our power of light doesn't work on normal humans.
As for our passiveness as much as i hate it, it was for the best because monetarist have the most power they can easly bend the facts and make people doubt us and treat the scions as criminals, even then gridania and limsa have their hands tied because going into war with uldah to defend us would be the worst outcome as they are all expecting garleans to swoop in at any moment and crush eorzea.