When you say your connection to the datacenter is 20ms... are you testing against an address starting with 199? If not, you are hitting the wrong server. If it's 184.107.107.176, that is a webserver....hosting EIDOS.com and other sites related to their game library, as well as Square-Enix sites.

You can find the proper IP by pulling up resmon, expanding the networking section, and finding FFXIV in the list. At least two connections should be there. 124 addresses are to Japan (idle for the most part), and the 199 IP is to the game server you are using.

Many have been researching this, working with various ISP's, (their own, as well as ones in-between them and SE) to try to pinpoint and (hopefully) relieve some of the choking points in route to SE's servers. Unfortunately for many, things have not improved yet, as many are still pointing fingers at someone else even though they've been shown testing that shows their hardware is not managing the congestion well.

If you do manage to gather evidence showing problems with packet loss in route to your actual game server, please forward them to both your ISP and SE. Hopefully, if enough data is gathered someone will finally take action one of these days.