Let's be honest, they may not be Elezen tall, but Elezen, Highlander, Roegadyn and Hrothgar are relatively rare anyways, so 7/10 times, the only ones taller than a male Viera would be a male Au Ra.
I felt a similar way when it turned out Hroths were considerably shorter than Roes, but I still tower over everybody else.