They seem to be rare because of taboo as stated in game, but I also wouldn't doubt there is some physical/cultural challenges with some of the races as well.

We know that Elezen and Hyur can procreate from Hilda, as well as Hyur and Au Ra (from the story in Sui no Sato of an Au Ra with no scales). Hyur and Garleans can as well as Arenvald is mixed race. Meaning that reasonably Elezen, Au Ra, Garleans, and Hyurs can all intermingle.

The other races I have a hard time imagining either because of their extreme statures, isolationist cultures, and some of their certain attributes being too different from Hyurs. Would a Miqo'te/Hyur hybrid have human ears on top of their head? Or cat ears on the side? Or Gods forbid a pair of both?

As an aside, I simply have the headcannon that Miqo/Viera + Hyur/Elezen/Garlean hybrids having hearing issues.