getting my data just fine from neolobby02 (199.91.189.74). Maybe it's just certain ones?

Check it via netstat or resmon to see which IP you are connecting to when sitting at the character select screen.

These are the registered IP's for the three CA lobby's:

neolobby02.ffxiv.com (199.91.189.74)
neolobby04.ffxiv.com (199.91.189.93)
neolobby06.ffxiv.com (199.91.189.106)