Not noticeable in other parts of the game (I can speak fine with NPCs with a half second before message windows open).
I also went and played leap of faith in goldsaucer to see other players and no lag.

I'll try to see if this only happens during peak hours, I only usually play PVP during peak for a faster queue.
Also, those ports are fine since I can play ok in non-peak hours.