Bad lag again with NTT's NA server network

[Ahumaya]

Chiming in to report that 5 of my 8 man static are experiencing this issue, and that so far our only resolution has been some form of VPN, mudfish for most of us.

As with everyone else, my trace-route identifies a significant increase in latency after entering the NTT infrastructure, as shown below:

Tracing route to 204.2.29.122 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.13.37.1
2 <1 ms <1 ms <1 ms 192.168.100.1
3 2 ms 3 ms 4 ms 162-239-116-1.lightspeed.tulsok.sbcglobal.net [162.239.116.1]
4 2 ms 2 ms 4 ms 76.200.2.190
5 * * * Request timed out.
6 7 ms 11 ms 4 ms 12.83.46.1
7 10 ms 15 ms 15 ms 32.130.25.65
8 9 ms 10 ms 8 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * * * Request timed out.
10 87 ms 96 ms 86 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
11 94 ms 95 ms 96 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 99 ms 95 ms 94 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
13 * 97 ms 98 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 95 ms * * 204.2.29.241
15 94 ms 93 ms 95 ms 204.2.29.122

Trace complete.

I have asked people who are experiencing this issue to report via the tool reported in the tweet regarding this issue.

What is very, deeply concerning however, is the significant lack of communication regarding this incident. Quite literally the only public acknowledgement of any issue at all is the tweet last Wednesday, which merely states that they are "aware" of an issue, and asking for impacted users to submit a report. There has been no follow-up, no confirmation of work, no identification of cause, and outside of select emails that some users claim to have received from either AT&T, or NTT, or Square Enix that it is being "worked on", we have zero actual response to an issue impacting many.

As of when I add this reply, I am the 501st reply to a thread of over 51 pages, of which there has been 0 responses from Square Enix or from Technical Support. It's as if we're shouting into the void with no response. In any other system, for any other company, a quality issue of this magnitude would be a breach of SLA, but even then there would be some form of acknowledgement that's not in the form of a tweet.

While VPNs can route around the issue that NTT is experiencing, that should not be the accepted solution. Square Enix has a contract with NTT to host their servers within NTT's Datacenters, and they (and CBU3 specifically) need to make sure that NTT resolves issues which negatively impact Square Enix's customers. To not do so, and to not even address the issue, is not only an embarrassment, but also a massive breach of trust.

As far as the limited investigations that we as users can perform, this is not an issue with AT&T, or any other ISP which is affected. Global backbone traffic for AT&T is not affected, and speeds and latency outside of NTT are as expected for AT&T. What we can see is a significant issue within the infrastructure that NTT provides for their Datacenters which must be addressed. Passing blame to or from doesn't serve anyone.

I love this game and have spent a lot of time with it, and I enjoy playing the game with the large number of friends I have made playing it, but when a huge portion of my FC can't play the game comfortably without using a VPN, and when those who play on console can't even use a VPN, it makes me question why I would want to continue to support a service and a company which can't even speak openly to its paying customers to say that there is a problem, and that they are working on it.

Peace and love, and I wish you all the best.

[MoonmoonMoonmoon]

Well here we are... 50 pages into a thread about AT&T issues and NTT. It's not SE's fault, but they're the ones that have to turn the screw on getting it fixed unfortunately since they are the major customer being impacted from NTT and AT&T's perspective. I don't think either company would deem it worth their time to listen to the little folks like us.

SO it's encouraging to hear that SE is aware and working to get it fixed. Maybe the 6.4 patch update will allow NTT to switch over to better nodes since the servers will be forced offline anyway? Hard to say. Here's to a big huff of Hopium. It smells sweet.

[Ionplasma]

I was going to resub for 6.4 but I'll wait until they fix it much to the dismay of my static. I should not have to pay for an VPN on top of the game subscription to just play.

It's ridiculous on how this issue have persisted for years and becomes disruptive every few months. NDA is not an excuse for lack of communication from Square, there's no way an NDA would cover over some problematic network node that is clearly visible over the Internet to the world.

Here's my traceroute from AT&T on the east coast:
traceroute to neolobby04.ffxiv.com (204.2.29.7), 64 hops max, 52 byte packets
1 opnsense (192.168.5.1) 3.856 ms 3.041 ms 3.431 ms
2 192.168.1.254 (192.168.1.254) 3.615 ms 4.634 ms 3.904 ms
3 104-13-92-1.lightspeed.tukrga.sbcglobal.net (104.13.92.1) 4.488 ms 4.696 ms 4.563 ms
4 107.212.169.40 (107.212.169.40) 4.968 ms 5.284 ms 4.835 ms
5 * * *
6 32.130.17.63 (32.130.17.63) 7.632 ms 19.246 ms 8.446 ms
7 192.205.36.158 (192.205.36.158) 6.340 ms 6.253 ms 5.482 ms
8 ae-8.r25.atlnga05.us.bb.gin.ntt.net (129.250.4.66) 7.085 ms 8.977 ms 8.271 ms
9 ae-6.r21.dllstx14.us.bb.gin.ntt.net (129.250.4.116) 21.630 ms 21.782 ms 21.710 ms
10 * * *
11 ae-2.r24.lsanca07.us.bb.gin.ntt.net (129.250.7.69) 105.722 ms 103.375 ms 98.035 ms <--- this is the problematic node in NTT
12 ae-5.a01.scrmca03.us.bb.gin.ntt.net (129.250.2.7) 109.419 ms 108.035 ms 107.932 ms
13 ae-1.a00.scrmca03.us.bb.gin.ntt.net (129.250.4.76) 112.734 ms 111.568 ms 111.180 ms
14 xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net (128.241.2.18) 115.420 ms 108.570 ms 112.163 ms
15 * * 204.2.29.241 (204.2.29.241) 111.744 ms
16 * * *

[Pat00tSack]

Here's hoping it gets fixed pretty soon.. If not, I'll be left in the dust for hard content cause I'm 99% sure people in my FC won't wait for SE to give me game privileges again to do new savage stuff.

[Huo]

Some people are suggesting paying money for a VPN or purchasing new internet (lol). We should not have to pay extra money to "get around" an obvious problem. Someone needs to take responsibility. Fix the problem. That's it.

[Uno_Amaretto]

Here is the grim, undeniable reality:

SE is a multi-billion dollar company.
If everyone in this thread were to unsubscribe, they would lose a few thousand dollars a month.
Meanwhile they know that FF16 and FF7R2 are going to sell millions and millions of copies.

You tell me how much motivation they have to care about us.

Amazing game. Terrible company.

[Okieeomi]

Hello everyone,

Thank you for your cooperation as we look into the recent network delays and/or packet loss experienced by some users playing on the North American data centers.

While our investigation is still ongoing, we've been able to confirm that the issues are caused by a network connection somewhere between our servers and those affected, rather than the FFXIV servers or equipment themselves.

At this time, we've forwarded a request to the relevant network carrier to locate the root of the issues; however, we believe it may expedite the investigation if those affected were to notify their internet service provider, which we understand some of you have already done. We apologize for making another request of those experiencing the recent network issues, but would be grateful if you could also report the situation to your internet service provider so that they are aware as well.

[Uno_Amaretto]

While our investigation is still ongoing, we've been able to confirm that the issues are caused by a network connection somewhere between our servers and those affected, rather than the FFXIV servers or equipment themselves.

Yeah, that was confirmed on like page 2 of this 51 page thread. Amazing investigation.

[Janyd]

Hello everyone,

Thank you for your cooperation as we look into the recent network delays and/or packet loss experienced by some users playing on the North American data centers.

While our investigation is still ongoing, we've been able to confirm that the issues are caused by a network connection somewhere between our servers and those affected, rather than the FFXIV servers or equipment themselves.

At this time, we've forwarded a request to the relevant network carrier to locate the root of the issues; however, we believe it may expedite the investigation if those affected were to notify their internet service provider, which we understand some of you have already done. We apologize for making another request of those experiencing the recent network issues, but would be grateful if you could also report the situation to your internet service provider so that they are aware as well.

Way ahead of you, Okieeomi, but thanks for reinforcing what we knew.

[ntt_routing_is_an_issue]

Well here we are... 50 pages into a thread about AT&T issues and NTT. It's not SE's fault, but they're the ones that have to turn the screw on getting it fixed unfortunately since they are the major customer being impacted from NTT and AT&T's perspective. I don't think either company would deem it worth their time to listen to the little folks like us.

SO it's encouraging to hear that SE is aware and working to get it fixed. Maybe the 6.4 patch update will allow NTT to switch over to better nodes since the servers will be forced offline anyway? Hard to say. Here's to a big huff of Hopium. It smells sweet.

Unfortunately that's not quite how it works. This problem will persist after the patch drop. Not that I want to see the game lose subs, but the only way to really voice the issue is with the wallet. And even then they likely wouldn't bat an eye unless it was every AT&T customer.