Bad lag again with NTT's NA server network

[Kaitlen]

I'm based in Tokyo and have been on Hyperion for the duration of the game since 2.0.
I rarely have any issue playing unless there's some locally induced problem.
The only real benefit is that I'm within the NTT network the entire time (OCN is provider on NTT).
There was some packet loss going on once my connection hit the next hop after crossing the ocean (#9-#11 below), but it seems to be gone now.

(Tracing/Pinging a Primal DC IP) Taken at 11:45am JST 5/18/23.

1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 1 ms 2 ms 118.23.27.227
3 2 ms 2 ms 2 ms 118.23.27.21
4 4 ms 4 ms 4 ms 118.23.4.5
5 4 ms 2 ms 2 ms 180.8.126.49
6 3 ms 3 ms 4 ms 122.1.245.209
7 3 ms 3 ms 3 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 5 ms 3 ms 3 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 112 ms 112 ms 116 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 * 112 ms * ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 128 ms 115 ms * ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 115 ms 114 ms 114 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 115 ms 204.2.29.86

Ping Stats:
Ping statistics for 204.2.29.86:
Packets: Sent = 321, Received = 321, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 143ms, Average = 116ms

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Edit: Added 2nd Sample from 1:29pm JST - seems to have stabilized from early this morning.

Tracing route to 204.2.29.86 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 2 ms 2 ms 118.23.27.227
3 3 ms 2 ms 3 ms 118.23.27.21
4 4 ms 3 ms 3 ms 118.23.4.5
5 6 ms 6 ms 4 ms 180.8.126.49
6 2 ms 3 ms 3 ms 122.1.245.209
7 3 ms 4 ms 4 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 3 ms 2 ms 3 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 112 ms 110 ms 110 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 113 ms 119 ms 115 ms ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 119 ms 137 ms 119 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 115 ms 115 ms 115 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 117 ms 204.2.29.86

Ping statistics for 204.2.29.86:
Packets: Sent = 2517, Received = 2517, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 157ms, Average = 118ms

[Kaitlen]

I'm based in Tokyo and have been on Hyperion for the duration of the game since 2.0.
I rarely have any issue playing unless there's some locally induced problem.
The only real benefit is that I'm within the NTT network the entire time (OCN is provider on NTT).
There was some packet loss going on once my connection hit the next hop after crossing the ocean (#9-#11 below), but it seems to be gone now.

(Tracing/Pinging a Primal DC IP) Taken at 11:45am JST 5/18/23.

1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 1 ms 2 ms 118.23.27.227
3 2 ms 2 ms 2 ms 118.23.27.21
4 4 ms 4 ms 4 ms 118.23.4.5
5 4 ms 2 ms 2 ms 180.8.126.49
6 3 ms 3 ms 4 ms 122.1.245.209
7 3 ms 3 ms 3 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 5 ms 3 ms 3 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 112 ms 112 ms 116 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 * 112 ms * ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 128 ms 115 ms * ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 115 ms 114 ms 114 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 115 ms 204.2.29.86

Ping Stats:
Ping statistics for 204.2.29.86:
Packets: Sent = 321, Received = 321, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 143ms, Average = 116ms

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Edit: Added 2nd Sample from 1:29pm JST - seems to have stabilized from early this morning.

Tracing route to 204.2.29.86 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 2 ms 2 ms 118.23.27.227
3 3 ms 2 ms 3 ms 118.23.27.21
4 4 ms 3 ms 3 ms 118.23.4.5
5 6 ms 6 ms 4 ms 180.8.126.49
6 2 ms 3 ms 3 ms 122.1.245.209
7 3 ms 4 ms 4 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 3 ms 2 ms 3 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 112 ms 110 ms 110 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 113 ms 119 ms 115 ms ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 119 ms 137 ms 119 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 115 ms 115 ms 115 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 117 ms 204.2.29.86

Ping statistics for 204.2.29.86:
Packets: Sent = 2517, Received = 2517, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 157ms, Average = 118ms

Replying with an update from my post yesterday. I took another sample as of 10:25am JST (6:25pm PST).
I have the unique view of being inside the NTT network for the entire path and it does reflect some consistent issues once my connection hits the US side. (hop #9)
Hops #2-#6 are also NTT hops, but just no RDNS entry.

Tracing route to 204.2.29.86 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 2 ms 2 ms 118.23.27.227
3 2 ms 2 ms 2 ms 118.23.27.21
4 7 ms 8 ms 5 ms 118.23.4.5
5 3 ms 3 ms 7 ms 180.8.126.49
6 29 ms 19 ms 18 ms 122.1.245.209
7 12 ms 13 ms 14 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 19 ms 23 ms 27 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 * 122 ms 126 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 151 ms 157 ms 139 ms ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 * 140 ms * ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 118 ms 116 ms 114 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 117 ms 204.2.29.86

Despite the issues noted above, I'm not experiencing packet loss today and no issues in-game.
(I'm usually on during Japan daytime (tyvm wfh!) and partially in the evening hours)
Ping Sample from 10:25AM-10:35AM.

Ping statistics for 204.2.29.86:
Packets: Sent = 740, Received = 740, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 157ms, Average = 119ms

I typically get a "Good" connection most days.

[Ath192]

Guys the only thing I can find in common so far is that people who are having problems are going through a 129.250.4.9 or a 129.250.4.239 address.

Can anyone parse through all the postings here and/or confirm any differently?

ATT Wireless – Drops

1 3 ms 3 ms 15 ms dsldevice.attlocal.net [192.168.1.254]
2 4 ms 4 ms 3 ms 108-218-196-1.lightspeed.snantx.sbcglobal.net [108.218.196.1]
3 7 ms 2 ms 3 ms 99.71.0.56
4 * * * Request timed out.
5 * * * Request timed out.
6 20 ms 13 ms 47 ms 32.130.25.67
7 38 ms 19 ms 13 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
8 * 13 ms 13 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]

9 * 88 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
10 94 ms * 105 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
11 109 ms 102 ms 111 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
12 100 ms 102 ms 126 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * 94 ms 204.2.29.241
14 95 ms 97 ms 98 ms 204.2.29.87


Spectrum - Drops
1 2 ms 11 ms 3 ms 192.168.0.1
2 16 ms 16 ms 9 ms 072-177-160-001.res.spectrum.com [72.177.160.1]
3 27 ms 34 ms 27 ms lag-63.snavtxuu01h.netops.charter.com [24.28.133.145]
4 27 ms 19 ms 18 ms lag-50.snavtxuu01r.netops.charter.com [24.175.33.176]
5 34 ms 18 ms 16 ms lag-24.mcr11snantxvy.netops.charter.com [24.175.32.216]
6 24 ms 23 ms 82 ms lag-23.rcr01dllatx37.netops.charter.com [24.175.32.146]
7 26 ms * 26 ms lag-14.dllstx976iw-bcr00.netops.charter.com [66.109.6.88]
8 * 28 ms 24 ms lag-0.pr3.dfw10.netops.charter.com [66.109.5.121]
9 28 ms 38 ms 64 ms ae3.cr7-dal3.ip4.gtt.net [208.116.212.9]
10 * 132 ms * ae8.cr11-dal3.ip4.gtt.net [89.149.139.205]
11 26 ms 26 ms 25 ms as2914.cr11-dal3.ip4.gtt.net [69.174.23.138]
12 * 36 ms 53 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]

13 57 ms 57 ms 53 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
14 155 ms 85 ms 79 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
15 69 ms 69 ms 64 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
16 62 ms 67 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
17 * * * Request timed out.
18 67 ms 72 ms 70 ms 204.2.29.87

Optimum - Works
7 * * * Request timed out.
8 58 ms 52 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
9 54 ms 56 ms 56 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
10 61 ms 79 ms 54 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
11 54 ms 54 ms 54 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * 59 ms * 204.2.29.241
13 55 ms 57 ms 53 ms 204.2.29.6

Even in this example where its gone now, those addresses are not present:

I'm based in Tokyo and have been on Hyperion for the duration of the game since 2.0.
I rarely have any issue playing unless there's some locally induced problem.
The only real benefit is that I'm within the NTT network the entire time (OCN is provider on NTT).
There was some packet loss going on once my connection hit the next hop after crossing the ocean (#9-#11 below), but it seems to be gone now.

(Tracing/Pinging a Primal DC IP) Taken at 11:45am JST 5/18/23.

1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 1 ms 2 ms 118.23.27.227
3 2 ms 2 ms 2 ms 118.23.27.21
4 4 ms 4 ms 4 ms 118.23.4.5
5 4 ms 2 ms 2 ms 180.8.126.49
6 3 ms 3 ms 4 ms 122.1.245.209
7 3 ms 3 ms 3 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 5 ms 3 ms 3 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 112 ms 112 ms 116 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 * 112 ms * ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 128 ms 115 ms * ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 115 ms 114 ms 114 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 115 ms 204.2.29.86

Ping Stats:
Ping statistics for 204.2.29.86:
Packets: Sent = 321, Received = 321, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 143ms, Average = 116ms

[Zhexos]

Greetings everyone,

The link to the form to submit your report has been updated.

We apologize for the inconvenience.

[TaleraRistain]

Just because a couple of us were keeping track of this, things have cleared up again around 2am eastern time for me. Seeing the same drop on that 57 node to the 60s ms and game is playable.

10 16 ms 17 ms 16 ms ae-4.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.239]
11 60 ms 59 ms 59 ms ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
12 63 ms 64 ms 63 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
13 64 ms 64 ms 64 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 63 ms * 66 ms 204.2.29.241
15 64 ms 65 ms 63 ms 204.2.29.6

I think in a previous post it starts acting up again around 10 or 11 eastern in the morning. I won't be up to check at that time but I'll check as soon as I can after that.

[Janyd]

Aye, the fact that this period of latency is a clockwork thing (from 10/11AM to 2AM Eastern) each day so far is very disconcerting but hopefully that will give the powers that be a clue as to what can be done to resolve the matter. See why that is, figure out a solution, and figure out why our connections with FFXIV suddenly act up on that schedule.

[ntt_routing_is_an_issue]

Cross posting for visibility

More fun with graphs now that SE has acknowledged there are some network issues in NA. First the route map:



Hop 4 is leaving AT&T's network, hop 5 obviously is entering NTT's network. First we look at our traffic statistics from AT&T to NTT:



There are a couple of spikes, but pretty comfortable 20ms on average. Then we get to hop 6, which is another NTT node in Dallas:



Here we start seeing maximums of almost 2 second latency, but still no packet loss. However, when we look at hop 7 which is the long haul from Dallas to Los Angeles (I previously thought this was San Francisco as I only had the device name to go off of) and we start to see a massive problem:



From the 14th at 2030 CST we start seeing ~30% packet loss which lasts roughly 12 hours. There's a short window where it stabilizes until 0900 on the 15th then starts dropping packets again for two straight days. This resolves at roughly 1300 on the 17th, then starts back up at 1800.

I understand the SE NOC can not feasibly monitor service quality from every players origination point to the game servers, but when the support staff just copy and paste the same 3 boxes over and over again blaming our ISP's for the issue and never escalating it, it's pretty annoying.

[MoonmoonMoonmoon]

My trace route shows that it's hopping to Los Angeles and then Sacramento.

ignoring the first few hops to avoid doxxing myself, let's just say that I have a pretty direct connection to AT&T via my fiber connection... here's my tracert output this morning.

4 3 ms 3 ms 3 ms 71.154.103.72
5 * * * Request timed out.
6 13 ms 15 ms 15 ms 32.130.25.65
7 15 ms 33 ms 12 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
8 16 ms 13 ms 11 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
9 49 ms 46 ms 50 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
10 58 ms 56 ms 78 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
11 64 ms 54 ms 60 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
12 53 ms 52 ms 60 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * 52 ms 204.2.29.241
14 59 ms 63 ms 63 ms 204.2.29.7

This is pretty in-line with what you're getting from your tracert, too

[Ath192]

Cross posting for visibility

More fun with graphs now that SE has acknowledged there are some network issues in NA. First the route map:



Hop 4 is leaving AT&T's network, hop 5 obviously is entering NTT's network. First we look at our traffic statistics from AT&T to NTT:



There are a couple of spikes, but pretty comfortable 20ms on average. Then we get to hop 6, which is another NTT node in Dallas:



Here we start seeing maximums of almost 2 second latency, but still no packet loss. However, when we look at hop 7 which is the long haul from Dallas to Los Angeles (I previously thought this was San Francisco as I only had the device name to go off of) and we start to see a massive problem:



From the 14th at 2030 CST we start seeing ~30% packet loss which lasts roughly 12 hours. There's a short window where it stabilizes until 0900 on the 15th then starts dropping packets again for two straight days. This resolves at roughly 1300 on the 17th, then starts back up at 1800.

I understand the SE NOC can not feasibly monitor service quality from every players origination point to the game servers, but when the support staff just copy and paste the same 3 boxes over and over again blaming our ISP's for the issue and never escalating it, it's pretty annoying.

The only thing that gives me pause is that yesterday we confirmed someone from Optimum was not having the same issues and he posted his Tracert that showed his traffic going through the exact same nodes you posted. With a 50ish ms response.

Here's that section on my friend, not sure if Dallas, but in Texas with Optimum. Not hitting that same node we're all showing issues with.

7 * * * Request timed out.
8 58 ms 52 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
9 54 ms 56 ms 56 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
10 61 ms 79 ms 54 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
11 54 ms 54 ms 54 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * 59 ms * 204.2.29.241
13 55 ms 57 ms 53 ms 204.2.29.6

This is also mine last night around 3 or 4 am when things were smooth enough to play. But nowhere near that same jump they're trying to say is expected going from Texas to Sacramento for that same node at that time.

8 8 ms 8 ms 9 ms ae-4.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.239]
9 68 ms 55 ms 57 ms ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
10 59 ms 61 ms 59 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
11 60 ms 60 ms 60 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * * * Request timed out.
13 60 ms 59 ms 59 ms 204.2.29.6

I can confirm no loses either this morning but until we hit 5-9 PM with no issues I won't be convinced.

[ntt_routing_is_an_issue]

If I were to put money on it, the issue is between ae-5.a00.dllstx14.us.bb.gin.ntt.net and ae-2.r20.dllstx14.us.bb.gin.ntt.net. I have two internet connections coming in, and on my spectrum line, I do not hit ae-5.a00.dllstx14.us.bb.gin.ntt.net and have not had issues in ~3 weeks. AT&T routing looks like:

Hop 4: 32.130.25.65
Hop 5: ae-5.a00.dllstx14.us.bb.gin.ntt.net
Hop 6: ae-2.r20.dllstx14.us.bb.gin.ntt.net

While spectrum looks like:

Hop 9: as2914.cr11-dal3.ip4.gtt.net
Hop 10: ae-2.r20.dllstx14.us.bb.gin.ntt.net
Hop 11: ae-2.r24.lsanca07.us.bb.gin.ntt.net