There could be something going on along your routing. It has been a common problem document here frequently, but not isolated to FFXIV (D3, WoW, and LoL have had issues recently on TWC forums as well as Netflix and Google/YouTube). It comes and goes year after year because of how markets continually get oversold and networks are poorly maintained.

Might spot some signs of congestion if you run a series of tracerts to the IP your client is using... if so, check some traces to other services to see if the same thing happens. If you find a match, forward the info to your ISP's tier3 support. They can and should conduct an investigation and forward their findings on to the proper people to address it. Your ISP is responsible for the agreements put in place to carry your data across to other ISP's networks, and thus have a means to affect changes where needed. It's hard to get them to take action though without good data to back you up when you contact them. If you can show good traces and bad traces to your game server as well as good and bad ones to another service, it lays out a pretty good case that they are having an issue with congestion control that needs to be addressed.