I think the thing that throws me the most and why I think they might not be related to the Ancients is that Byregot looks very much like a roegadyn. And if you decide to just chalk that up to artistic license and that he's just an overly-muscular god, then why does Rhalgr look like an elezen complete with the ears when none of his depictions nor even the statue that you fight him on have the same elezen ears?