蛮族 ("banzoku"), yes. Literally "barbarian people" ("people" in the sociological sense). And "Primals" are 蛮神 ("banshin"), which literally translates to "barbarian god" or "god of the barbarians".
You are correct that Garleans refer to everyone else (Eorzeans and Othardians) as "savages", which is synonymous with "barbarians". Etymologically, "barbarian" originally meant "the people who are not Greeks", to contrast with the "Greek" identification with "polites", ie "citizen"/"people from the city". Then the Romans took that term and used it to refer to "uncivilized" peoples, ie anyone who was not a citizen of Rome.
So I see it as a sort of parallel development of xenophobia: the Eorzeans use "banzoku" to refer to any culture that is not a citizen of one of the city-states, while the Garleans use "banzoku" to refer to anyone who is not a citizen of Garlemald.
This also happens (deliberately) to correspond to racial lines, rather than simply territorial ones, in both RL and FFXIV. The term "banzoku" or "barbarian" was almost never used in a strict dictionary sense, but much more commonly as an insult or slur. Hence the negative connotations of "beast tribe".
Yeah, the devs have been wanting to change the "Beast Tribe" parts of the UI to simply "Tribe" for a while. They've just announced being able to do so recently, while bringing up the Dwarves on the First as an illustrative example.
The reasoning they gave was, well, the same as this thread: it's rude and prejudiced, and it still makes sense to change racially problematic behaviour even when that racism is directed at fictional races.
