tbf, Byregot's used to be in W Shroud which... we can't just visit anymore
let me iiiiiin, Yoshidaaaaaa near one of the old camps, so it was displaced during the 7UC... somehow! I don't think it was moved by a spoken race since
that looks mighty heavy. It could have been elsewhere long ago too, which lends itself to the dialogue post-raid that they've been around for a real long time, but the earth being rent asunder so many times, they either appear elsewhere or through the Twelve's own ability they manifest elsewhere. Some may have some obvious connections (Gridania is still influenced by Amdapor with Nophica, Ul'dah the twin gods, Limsa and Llymlaen) but others, like you said, may not... at least ones we can't/haven't sussed out yet

Whether we get that info or not, I enjoy theorizing about it haha.