Bad lag again with NTT's NA server network

[Ath192]

If I were to put money on it, the issue is between ae-5.a00.dllstx14.us.bb.gin.ntt.net and ae-2.r20.dllstx14.us.bb.gin.ntt.net. I have two internet connections coming in, and on my spectrum line, I do not hit ae-5.a00.dllstx14.us.bb.gin.ntt.net and have not had issues in ~3 weeks. AT&T routing looks like:

Hop 4: 32.130.25.65
Hop 5: ae-5.a00.dllstx14.us.bb.gin.ntt.net
Hop 6: ae-2.r20.dllstx14.us.bb.gin.ntt.net

While spectrum looks like:

Hop 9: as2914.cr11-dal3.ip4.gtt.net
Hop 10: ae-2.r20.dllstx14.us.bb.gin.ntt.net
Hop 11: ae-2.r24.lsanca07.us.bb.gin.ntt.net

Here is the first post showing none of those though:

It's happening again. NTT having bad packet loss and lag all within their network. Please get this fixed once and for all.

Current packet loss.


Past 12 hours graphs. Can see when it start about 10:30am EST


Past 2.5 months.

(NTT made a change in early April that caused the average pings to increase by 5-10ms)

Still the only thing I can find in common so far is that people who are having problems are going through a 129.250.4.9 or a 129.250.4.239 address. If anyone can prove this is wrong lmk. You will also see that the known working optimum tracert does not include either of these.


*Update, first packet drop of the day recorded 10:45 AM -11 AM CST*

Reply from 204.2.29.86: bytes=32 time=120ms TTL=50
Reply from 204.2.29.86: bytes=32 time=110ms TTL=50
Reply from 204.2.29.86: bytes=32 time=121ms TTL=50
Reply from 204.2.29.86: bytes=32 time=107ms TTL=50
Reply from 204.2.29.86: bytes=32 time=114ms TTL=50
Reply from 204.2.29.86: bytes=32 time=109ms TTL=50
Reply from 204.2.29.86: bytes=32 time=115ms TTL=50
Reply from 204.2.29.86: bytes=32 time=118ms TTL=50
Reply from 204.2.29.86: bytes=32 time=122ms TTL=50
Reply from 204.2.29.86: bytes=32 time=117ms TTL=50
Reply from 204.2.29.86: bytes=32 time=117ms TTL=50
Reply from 204.2.29.86: bytes=32 time=120ms TTL=50
Reply from 204.2.29.86: bytes=32 time=111ms TTL=50
Request timed out.
Reply from 204.2.29.86: bytes=32 time=113ms TTL=50
Reply from 204.2.29.86: bytes=32 time=121ms TTL=50
Reply from 204.2.29.86: bytes=32 time=117ms TTL=50
Reply from 204.2.29.86: bytes=32 time=104ms TTL=50
Reply from 204.2.29.86: bytes=32 time=112ms TTL=50
Reply from 204.2.29.86: bytes=32 time=113ms TTL=50
Reply from 204.2.29.86: bytes=32 time=115ms TTL=50
Reply from 204.2.29.86: bytes=32 time=115ms TTL=50
Reply from 204.2.29.86: bytes=32 time=121ms TTL=50
Reply from 204.2.29.86: bytes=32 time=119ms TTL=50
Reply from 204.2.29.86: bytes=32 time=113ms TTL=50
Reply from 204.2.29.86: bytes=32 time=108ms TTL=50

[Lahtori]

My big question is why now? I've only been in North Texas for 10 months, but in that time I've always had AT&T Fiber and had ZERO problems in game. So what happened on Monday with AT&T and/or NTT that changed the latency for myself and countless others? I'm not an engineer and know very little about networking (enough to troubleshoot client side (my side) problems), but seems to me "something" was changed and it broke. Shouldn't be that difficult to figure out what and get a fix / rollback to when it wasn't an issue.

Shrug. I guess I'm looking into a VPN. I don't necessarily want to go 4 days without playing now. Already cancelled raid Tuesday, don't want to cancel another. And it's not looking good with patch day in 5 days and savage a week after.

I submitted my tracert, spoke with AT&T, not much more I can do.

[brnslpy]

My big question is why now? I've only been in North Texas for 10 months, but in that time I've always had AT&T Fiber and had ZERO problems in game. So what happened on Monday with AT&T and/or NTT that changed the latency for myself and countless others? I'm not an engineer and know very little about networking (enough to troubleshoot client side (my side) problems), but seems to me "something" was changed and it broke. Shouldn't be that difficult to figure out what and get a fix / rollback to when it wasn't an issue.

Shrug. I guess I'm looking into a VPN. I don't necessarily want to go 4 days without playing now. Already cancelled raid Tuesday, don't want to cancel another. And it's not looking good with patch day in 5 days and savage a week after.

I submitted my tracert, spoke with AT&T, not much more I can do.

I've had AT&T Fiber ever since I started playing FFXIV a few years ago. This ATT+NTT node problem has been long term. But historically, it might be a problem once a quarter for 6 hours one night, then be fine for weeks. You think, network maintenance no problem, right?

But in the past year, it's gotten more common, like once a month for 6-12 hours. Most I've experienced before is like "oh its Friday night, lag all night, then Saturday morning it's fine"

This current problem is the worst I've seen where it's lasted days which is kind of ridiculous for two Tier 1 / Tier 2 ISPs to have noticeable congestion like that and not try and re-route it. /vent

[Cythrawll]

Shouldn't be that difficult to figure out what and get a fix / rollback to when it wasn't an issue.

Can't really rollback network topology like that. it's not software. NTT probably has to reroute traffic due to hardware or routing locations that straight up don't exist anymore and didn't expect congestion at their Sacramento node to get so bad.

It really is a congestion issue. That node is fine if you test it during off hours. I tested 2am central cuz curious last night and it was fine.

[MoonmoonMoonmoon]

Shrug. I guess I'm looking into a VPN.

Proton VPN has a free tier that worked amazingly for me last night. My latency was a little high for my spoiled/pampered tastes but it was perfectly manageable at 100ms.

[Janyd]

Hmm, this looks promising.

Usually, this would be the starting time for the latency to start showing itself, according to the patterns we've noted but I just ran a traceroute, and here's what I've got (starting at the NTT nodes in Dallas):

8 28 ms 26 ms 26 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * 25 ms * ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
10 * * * Request timed out.
11 60 ms 84 ms 64 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 63 ms 64 ms 61 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
13 72 ms 73 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 * 64 ms * 204.2.29.241
15 64 ms 68 ms 68 ms 204.2.29.122


Ran a Ping command as well:

Pinging 204.2.29.122 with 32 bytes of data:
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=60ms TTL=49
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=58ms TTL=49

Ping statistics for 204.2.29.122:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 58ms, Maximum = 60ms, Average = 59ms

[brnslpy]

Hmm, this looks promising.

Usually, this would be the starting time for the latency to start showing itself, according to the patterns we've noted but I just ran a traceroute, and here's what I've got (starting at the NTT nodes in Dallas):

8 28 ms 26 ms 26 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * 25 ms * ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
10 * * * Request timed out.
11 60 ms 84 ms 64 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 63 ms 64 ms 61 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
13 72 ms 73 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 * 64 ms * 204.2.29.241
15 64 ms 68 ms 68 ms 204.2.29.122


Ran a Ping command as well:

Pinging 204.2.29.122 with 32 bytes of data:
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=60ms TTL=49
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=58ms TTL=49

Ping statistics for 204.2.29.122:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 58ms, Maximum = 60ms, Average = 59ms

This does look promising. Been watching pings all morning. 65ms average (and normal behavior for me)

Right around 11:10AM EDT, pings jumped to 110ms (sigh) but so far are hanging around 70ms with a few spikes but a lot different than the last few days where it stayed at 120ms with packet loss.

Traceroutes on two of our problem nodes, 128.241.2.18 and 129.250.7.57, are also responding better than before (70-80ms down from 120ms)

Fwiw, after getting run around with ATT support last night, getting disconnected from their chat, and getting a phone call, they did say "should be fixed by Thursday morning" so lets hope so *fingers crossed*

[Tomoken]

It's showing up clean for me now without VPN (have been playing the game with PIA + Michigan). Looks to be back to normal unless it's just working for now...?

[Janyd]

Bummer, just ran a few Ping commands, and the latency on both of them jumped up to 98-105ms.

[Janyd]

At this rate, wouldn't it behoove SE to demand that NTT take those faulty nodes offline until fixes can be made?