Bad lag again with NTT's NA server network

[ntt_routing_is_an_issue]

Cross posting for visibility

More fun with graphs now that SE has acknowledged there are some network issues in NA. First the route map:



Hop 4 is leaving AT&T's network, hop 5 obviously is entering NTT's network. First we look at our traffic statistics from AT&T to NTT:



There are a couple of spikes, but pretty comfortable 20ms on average. Then we get to hop 6, which is another NTT node in Dallas:



Here we start seeing maximums of almost 2 second latency, but still no packet loss. However, when we look at hop 7 which is the long haul from Dallas to Los Angeles (I previously thought this was San Francisco as I only had the device name to go off of) and we start to see a massive problem:



From the 14th at 2030 CST we start seeing ~30% packet loss which lasts roughly 12 hours. There's a short window where it stabilizes until 0900 on the 15th then starts dropping packets again for two straight days. This resolves at roughly 1300 on the 17th, then starts back up at 1800.

I understand the SE NOC can not feasibly monitor service quality from every players origination point to the game servers, but when the support staff just copy and paste the same 3 boxes over and over again blaming our ISP's for the issue and never escalating it, it's pretty annoying.

[Lahtori]

My big question is why now? I've only been in North Texas for 10 months, but in that time I've always had AT&T Fiber and had ZERO problems in game. So what happened on Monday with AT&T and/or NTT that changed the latency for myself and countless others? I'm not an engineer and know very little about networking (enough to troubleshoot client side (my side) problems), but seems to me "something" was changed and it broke. Shouldn't be that difficult to figure out what and get a fix / rollback to when it wasn't an issue.

Shrug. I guess I'm looking into a VPN. I don't necessarily want to go 4 days without playing now. Already cancelled raid Tuesday, don't want to cancel another. And it's not looking good with patch day in 5 days and savage a week after.

I submitted my tracert, spoke with AT&T, not much more I can do.

[Janyd]

Hmm, this looks promising.

Usually, this would be the starting time for the latency to start showing itself, according to the patterns we've noted but I just ran a traceroute, and here's what I've got (starting at the NTT nodes in Dallas):

8 28 ms 26 ms 26 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * 25 ms * ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
10 * * * Request timed out.
11 60 ms 84 ms 64 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 63 ms 64 ms 61 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
13 72 ms 73 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 * 64 ms * 204.2.29.241
15 64 ms 68 ms 68 ms 204.2.29.122


Ran a Ping command as well:

Pinging 204.2.29.122 with 32 bytes of data:
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=60ms TTL=49
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=58ms TTL=49

Ping statistics for 204.2.29.122:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 58ms, Maximum = 60ms, Average = 59ms

[MoonmoonMoonmoon]

My trace route shows that it's hopping to Los Angeles and then Sacramento.

ignoring the first few hops to avoid doxxing myself, let's just say that I have a pretty direct connection to AT&T via my fiber connection... here's my tracert output this morning.

4 3 ms 3 ms 3 ms 71.154.103.72
5 * * * Request timed out.
6 13 ms 15 ms 15 ms 32.130.25.65
7 15 ms 33 ms 12 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
8 16 ms 13 ms 11 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
9 49 ms 46 ms 50 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
10 58 ms 56 ms 78 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
11 64 ms 54 ms 60 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
12 53 ms 52 ms 60 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * 52 ms 204.2.29.241
14 59 ms 63 ms 63 ms 204.2.29.7

This is pretty in-line with what you're getting from your tracert, too

[brnslpy]

My big question is why now? I've only been in North Texas for 10 months, but in that time I've always had AT&T Fiber and had ZERO problems in game. So what happened on Monday with AT&T and/or NTT that changed the latency for myself and countless others? I'm not an engineer and know very little about networking (enough to troubleshoot client side (my side) problems), but seems to me "something" was changed and it broke. Shouldn't be that difficult to figure out what and get a fix / rollback to when it wasn't an issue.

Shrug. I guess I'm looking into a VPN. I don't necessarily want to go 4 days without playing now. Already cancelled raid Tuesday, don't want to cancel another. And it's not looking good with patch day in 5 days and savage a week after.

I submitted my tracert, spoke with AT&T, not much more I can do.

I've had AT&T Fiber ever since I started playing FFXIV a few years ago. This ATT+NTT node problem has been long term. But historically, it might be a problem once a quarter for 6 hours one night, then be fine for weeks. You think, network maintenance no problem, right?

But in the past year, it's gotten more common, like once a month for 6-12 hours. Most I've experienced before is like "oh its Friday night, lag all night, then Saturday morning it's fine"

This current problem is the worst I've seen where it's lasted days which is kind of ridiculous for two Tier 1 / Tier 2 ISPs to have noticeable congestion like that and not try and re-route it. /vent

[Cythrawll]

Shouldn't be that difficult to figure out what and get a fix / rollback to when it wasn't an issue.

Can't really rollback network topology like that. it's not software. NTT probably has to reroute traffic due to hardware or routing locations that straight up don't exist anymore and didn't expect congestion at their Sacramento node to get so bad.

It really is a congestion issue. That node is fine if you test it during off hours. I tested 2am central cuz curious last night and it was fine.

[MoonmoonMoonmoon]

Shrug. I guess I'm looking into a VPN.

Proton VPN has a free tier that worked amazingly for me last night. My latency was a little high for my spoiled/pampered tastes but it was perfectly manageable at 100ms.

[Tomoken]

It's showing up clean for me now without VPN (have been playing the game with PIA + Michigan). Looks to be back to normal unless it's just working for now...?

[brnslpy]

Hmm, this looks promising.

Usually, this would be the starting time for the latency to start showing itself, according to the patterns we've noted but I just ran a traceroute, and here's what I've got (starting at the NTT nodes in Dallas):

8 28 ms 26 ms 26 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * 25 ms * ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
10 * * * Request timed out.
11 60 ms 84 ms 64 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 63 ms 64 ms 61 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
13 72 ms 73 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 * 64 ms * 204.2.29.241
15 64 ms 68 ms 68 ms 204.2.29.122


Ran a Ping command as well:

Pinging 204.2.29.122 with 32 bytes of data:
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=60ms TTL=49
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=58ms TTL=49

Ping statistics for 204.2.29.122:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 58ms, Maximum = 60ms, Average = 59ms

This does look promising. Been watching pings all morning. 65ms average (and normal behavior for me)

Right around 11:10AM EDT, pings jumped to 110ms (sigh) but so far are hanging around 70ms with a few spikes but a lot different than the last few days where it stayed at 120ms with packet loss.

Traceroutes on two of our problem nodes, 128.241.2.18 and 129.250.7.57, are also responding better than before (70-80ms down from 120ms)

Fwiw, after getting run around with ATT support last night, getting disconnected from their chat, and getting a phone call, they did say "should be fixed by Thursday morning" so lets hope so *fingers crossed*

[Ath192]

Cross posting for visibility

More fun with graphs now that SE has acknowledged there are some network issues in NA. First the route map:



Hop 4 is leaving AT&T's network, hop 5 obviously is entering NTT's network. First we look at our traffic statistics from AT&T to NTT:



There are a couple of spikes, but pretty comfortable 20ms on average. Then we get to hop 6, which is another NTT node in Dallas:



Here we start seeing maximums of almost 2 second latency, but still no packet loss. However, when we look at hop 7 which is the long haul from Dallas to Los Angeles (I previously thought this was San Francisco as I only had the device name to go off of) and we start to see a massive problem:



From the 14th at 2030 CST we start seeing ~30% packet loss which lasts roughly 12 hours. There's a short window where it stabilizes until 0900 on the 15th then starts dropping packets again for two straight days. This resolves at roughly 1300 on the 17th, then starts back up at 1800.

I understand the SE NOC can not feasibly monitor service quality from every players origination point to the game servers, but when the support staff just copy and paste the same 3 boxes over and over again blaming our ISP's for the issue and never escalating it, it's pretty annoying.

The only thing that gives me pause is that yesterday we confirmed someone from Optimum was not having the same issues and he posted his Tracert that showed his traffic going through the exact same nodes you posted. With a 50ish ms response.

Here's that section on my friend, not sure if Dallas, but in Texas with Optimum. Not hitting that same node we're all showing issues with.

7 * * * Request timed out.
8 58 ms 52 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
9 54 ms 56 ms 56 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
10 61 ms 79 ms 54 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
11 54 ms 54 ms 54 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * 59 ms * 204.2.29.241
13 55 ms 57 ms 53 ms 204.2.29.6

This is also mine last night around 3 or 4 am when things were smooth enough to play. But nowhere near that same jump they're trying to say is expected going from Texas to Sacramento for that same node at that time.

8 8 ms 8 ms 9 ms ae-4.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.239]
9 68 ms 55 ms 57 ms ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
10 59 ms 61 ms 59 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
11 60 ms 60 ms 60 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * * * Request timed out.
13 60 ms 59 ms 59 ms 204.2.29.6

I can confirm no loses either this morning but until we hit 5-9 PM with no issues I won't be convinced.