yes please, "tracert 204.2.29.87" no quotes on both and post please.Quote:
Originally Posted by Kestaa
Bad lag again with NTT's NA server network
Printable View
-
05-18-2023, 10:49 AMAth192
-
05-18-2023, 10:57 AMKestaaupdated my previous post with results but here they are again jicQuote:
Originally Posted by Ath192
ATT Wireless
Tracing route to 204.2.29.122 over a maximum of 30 hops
4 * * * Request timed out.
5 * * * Request timed out.
6 13 ms 16 ms 23 ms 32.130.25.67
7 30 ms 12 ms 14 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
8 11 ms 10 ms 14 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
9 90 ms 90 ms 89 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
10 99 ms 101 ms 102 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
11 99 ms 97 ms 102 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
12 105 ms 105 ms * xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * 98 ms 97 ms 204.2.29.241
14 100 ms * 101 ms 204.2.29.122
Trace complete.
Spectrum Wireless
Tracing route to 204.2.29.122 over a maximum of 30 hops
4 49 ms * 19 ms lag-50.mcr11snavtxuu.netops.charter.com [24.175.33.180]
5 23 ms 24 ms 22 ms lag-23.rcr01hstqtx02.netops.charter.com [24.175.32.156]
6 18 ms * * lag-416.hstqtx0209w-bcr00.netops.charter.com [66.109.9.88]
7 19 ms * 87 ms lag-800.pr0.hou50.netops.charter.com [66.109.5.236]
8 * * * Request timed out.
9 * 140 ms * dls-b24-link.ip.twelve99.net [62.115.123.106]
10 * * * Request timed out.
11 30 ms 24 ms 25 ms dls-b23-link.ip.twelve99.net [62.115.138.65]
12 25 ms 24 ms 31 ms ntt-ic-325660.ip.twelve99-cust.net [62.115.156.249]
13 29 ms 27 ms 30 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
14 * 115 ms 64 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
15 67 ms 74 ms 68 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
16 65 ms 64 ms 60 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
17 73 ms 73 ms 79 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
18 67 ms * * 204.2.29.241
19 68 ms 70 ms 71 ms 204.2.29.122 -
05-18-2023, 10:59 AMAth192Ok, now you are going to do the following, "ping 204.2.29.87 -t" no quotes, let it run for 5 minutes or so, then do CTRL + C to cancel and post the end, how many packets were sent, and how many were received/lost. One time on ATT and the other on Spectrum.Quote:
Originally Posted by Kestaa
-
05-18-2023, 11:03 AMKestaaLooks like I was using a different tracert than you (grabbed min from the old post) here are the results for both from 204.2.29.87Quote:
Originally Posted by Ath192
Spectrum
tracert 204.2.29.87
Tracing route to 204.2.29.87 over a maximum of 30 hops
4 27 ms 19 ms 18 ms lag-50.snavtxuu01r.netops.charter.com [24.175.33.176]
5 34 ms 18 ms 16 ms lag-24.mcr11snantxvy.netops.charter.com [24.175.32.216]
6 24 ms 23 ms 82 ms lag-23.rcr01dllatx37.netops.charter.com [24.175.32.146]
7 26 ms * 26 ms lag-14.dllstx976iw-bcr00.netops.charter.com [66.109.6.88]
8 * 28 ms 24 ms lag-0.pr3.dfw10.netops.charter.com [66.109.5.121]
9 28 ms 38 ms 64 ms ae3.cr7-dal3.ip4.gtt.net [208.116.212.9]
10 * 132 ms * ae8.cr11-dal3.ip4.gtt.net [89.149.139.205]
11 26 ms 26 ms 25 ms as2914.cr11-dal3.ip4.gtt.net [69.174.23.138]
12 * 36 ms 53 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
13 57 ms 57 ms 53 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
14 155 ms 85 ms 79 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
15 69 ms 69 ms 64 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
16 62 ms 67 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
17 * * * Request timed out.
18 67 ms 72 ms 70 ms 204.2.29.87
Trace complete.
ATT
Tracing route to 204.2.29.87 over a maximum of 30 hops
4 * * * Request timed out.
5 * * * Request timed out.
6 20 ms 13 ms 47 ms 32.130.25.67
7 38 ms 19 ms 13 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
8 * 13 ms 13 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
9 * 88 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
10 94 ms * 105 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
11 109 ms 102 ms 111 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
12 100 ms 102 ms 126 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * 94 ms 204.2.29.241
14 95 ms 97 ms 98 ms 204.2.29.87
Trace complete. -
05-18-2023, 11:04 AMAth192No worries, thank you for your service, do the ping test now, remember let it run for 5 minutes each and clearly label them here. We'll see if Spectrum doesnt drop and ATT does. "ping 204.2.29.87 -t" no quotes, let it run for 5 minutes or so, then do CTRL + C to cancel and post the end, how many packets were sent, and how many were received/lost. One time on ATT and the other on Spectrum.Quote:
Originally Posted by Kestaa
-
05-18-2023, 11:29 AMKestaaQuote:
Originally Posted by Ath192
ping 204.2.29.87 -t As "on the min" as I could with timing. fwiw both routers are not in the same room as my pc. Spectrum router was fluctuating between 3-5 bars for testing. ATT at a full 5 the entire time of testing. Im not a techy but hopefully this data can help out.
Spectrum Non Fiber Wireless
9:17- 9:22
Ping statistics for 204.2.29.87:
Packets: Sent = 185, Received = 155, Lost = 30 (16% loss),
Approximate round trip times in milli-seconds:
Minimum = 57ms, Maximum = 302ms, Average = 91ms
ATT Fiber Wireless 9:24-9:29
Ping statistics for 204.2.29.87:
Packets: Sent = 192, Received = 163, Lost = 29 (15% loss),
Approximate round trip times in milli-seconds:
Minimum = 93ms, Maximum = 127ms, Average = 97ms -
05-18-2023, 11:29 AMSonicRoseWhatever these numbers mean
I'm East Coast, NA
My service provider is ATT and I used the info Square provided to get this. Should be with a Fiberoptic wire, and I have a wired connection.
Tracing route to 204.2.29.122 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 2 ms 3 ms 2 ms 23-116-200-1.lightspeed.toldoh.sbcglobal.net [23.116.200.1]
3 4 ms 4 ms 1 ms 71.153.70.124
4 * * * Request timed out.
5 * * * Request timed out.
6 15 ms 13 ms 12 ms 32.130.17.77
7 19 ms 12 ms 12 ms cgcil402igs.ip.att.net [12.122.132.197]
8 62 ms 70 ms 63 ms 192.205.32.194
9 62 ms 65 ms 61 ms ae-4.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.239]
10 60 ms 60 ms 56 ms ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
11 * * 117 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 * 117 ms 118 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 * 118 ms * 204.2.29.122
15 * 117 ms 118 ms 204.2.29.122 -
05-18-2023, 11:32 AMAth192It does thanks, Spectrum is affected it seems.Quote:
Originally Posted by Kestaa
-
05-18-2023, 11:35 AMKestaaAnecdotal: The rubberbanding has not been nearly as bad on spectrum as att. It does however dc me more. But at least its smoother over all. Still cant macro craft on either though :\Quote:
Originally Posted by Ath192
-
05-18-2023, 11:35 AMLinaGiggles
This is what I got when I did ticket and such. I don't know if I should even make new thread or just at least post here on this update.
Greetings,
Thank you for keeping in touch with the SQUARE ENIX Support Center.
We apologize for the confusion and inconvenience you are still having experience with. Unfortunately this is a known issue, at the moment is under investigation and procedure, we advise to please report this on the forum as a bug and to stay on the lookout for any update on the Lodestone. We can only ask for you patience for the time being.
We thank you for understating this matter. If you ever need further assistance, do not hesitate to contact us. Have a great day and please take care.
Thank you,
Veronica
SQUARE ENIX Customer Support
_____________
If you need any additional assistance with this concern, please reply to this email directly. If you need assistance with a new concern, please visit the SQUARE ENIX Support Center and submit a new support ticket at: https://support.na.square-enix.com. Additionally, we have a survey that you may fill out to provide your customer service feedback and suggestions once your ticket has been closed. -
05-18-2023, 11:44 AMJanyd
Thanks for sharing the ticket reply from SE, Lina.
Yes, you can use this forum thread to keep the reports and traceroutes coming.
The best we can do is keep doing what we're doing here, and if you want to use a VPN as a temporary workaround, I highly recommend you do so if you can spare the cash. -
05-18-2023, 11:46 AMJonobangsame issues, bad lag every time i play
I am midwest NA,
Tracing route to 204.2.29.122 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 20 ms 22 ms 20 ms 76-255-24-1.lightspeed.mdsnwi.sbcglobal.net [76.255.24.1]
3 20 ms 21 ms 20 ms 76.229.12.62
4 * * * Request timed out.
5 26 ms 33 ms 28 ms 32.130.17.75
6 25 ms 30 ms 30 ms cgcil402igs.ip.att.net [12.122.133.161]
7 75 ms 76 ms 74 ms 192.205.32.194
8 75 ms 75 ms 79 ms ae-4.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.239]
9 91 ms 80 ms 78 ms ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
10 * 132 ms 131 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
11 133 ms 132 ms 133 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * 131 ms 132 ms 204.2.29.241
13 132 ms * 132 ms 204.2.29.122
Trace complete.
My traceroute results, hope they fix it soon :( -
05-18-2023, 11:53 AMAllison_Grimvault
Lag is as bad as ever STILL and on the 3rd day. Game is unplayable.
-
05-18-2023, 11:53 AMKaitlen
I'm based in Tokyo and have been on Hyperion for the duration of the game since 2.0.
I rarely have any issue playing unless there's some locally induced problem.
The only real benefit is that I'm within the NTT network the entire time (OCN is provider on NTT).
There was some packet loss going on once my connection hit the next hop after crossing the ocean (#9-#11 below), but it seems to be gone now.
(Tracing/Pinging a Primal DC IP) Taken at 11:45am JST 5/18/23.
1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 1 ms 2 ms 118.23.27.227
3 2 ms 2 ms 2 ms 118.23.27.21
4 4 ms 4 ms 4 ms 118.23.4.5
5 4 ms 2 ms 2 ms 180.8.126.49
6 3 ms 3 ms 4 ms 122.1.245.209
7 3 ms 3 ms 3 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 5 ms 3 ms 3 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 112 ms 112 ms 116 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 * 112 ms * ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 128 ms 115 ms * ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 115 ms 114 ms 114 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 115 ms 204.2.29.86
Ping Stats:
Ping statistics for 204.2.29.86:
Packets: Sent = 321, Received = 321, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 143ms, Average = 116ms
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Edit: Added 2nd Sample from 1:29pm JST - seems to have stabilized from early this morning.
Tracing route to 204.2.29.86 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms RT-AC88U-2710 [192.168.1.254]
2 2 ms 2 ms 2 ms 118.23.27.227
3 3 ms 2 ms 3 ms 118.23.27.21
4 4 ms 3 ms 3 ms 118.23.4.5
5 6 ms 6 ms 4 ms 180.8.126.49
6 2 ms 3 ms 3 ms 122.1.245.209
7 3 ms 4 ms 4 ms ae-6.r02.tokyjp05.jp.bb.gin.ntt.net [120.88.53.21]
8 3 ms 2 ms 3 ms ae-3.r30.tokyjp05.jp.bb.gin.ntt.net [129.250.3.23]
9 112 ms 110 ms 110 ms ae-4.r25.snjsca04.us.bb.gin.ntt.net [129.250.5.78]
10 113 ms 119 ms 115 ms ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146]
11 119 ms 137 ms 119 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
12 115 ms 115 ms 115 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * * Request timed out.
14 115 ms 115 ms 117 ms 204.2.29.86
Ping statistics for 204.2.29.86:
Packets: Sent = 2517, Received = 2517, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 115ms, Maximum = 157ms, Average = 118ms -
05-18-2023, 12:02 PMLinaGigglesYeah for now ..I got VPN.. I shouldn't have to or any of us shouldn't have to but its best way I can play for now. :S <3 Thanks for the suggestion.Quote:
Originally Posted by Janyd
-
05-18-2023, 12:29 PMAth192
Guys the only thing I can find in common so far is that people who are having problems are going through a 129.250.4.9 or a 129.250.4.239 address.
Can anyone parse through all the postings here and/or confirm any differently?
ATT Wireless – Drops
1 3 ms 3 ms 15 ms dsldevice.attlocal.net [192.168.1.254]
2 4 ms 4 ms 3 ms 108-218-196-1.lightspeed.snantx.sbcglobal.net [108.218.196.1]
3 7 ms 2 ms 3 ms 99.71.0.56
4 * * * Request timed out.
5 * * * Request timed out.
6 20 ms 13 ms 47 ms 32.130.25.67
7 38 ms 19 ms 13 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
8 * 13 ms 13 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
9 * 88 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
10 94 ms * 105 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
11 109 ms 102 ms 111 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
12 100 ms 102 ms 126 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * 94 ms 204.2.29.241
14 95 ms 97 ms 98 ms 204.2.29.87
Spectrum - Drops
1 2 ms 11 ms 3 ms 192.168.0.1
2 16 ms 16 ms 9 ms 072-177-160-001.res.spectrum.com [72.177.160.1]
3 27 ms 34 ms 27 ms lag-63.snavtxuu01h.netops.charter.com [24.28.133.145]
4 27 ms 19 ms 18 ms lag-50.snavtxuu01r.netops.charter.com [24.175.33.176]
5 34 ms 18 ms 16 ms lag-24.mcr11snantxvy.netops.charter.com [24.175.32.216]
6 24 ms 23 ms 82 ms lag-23.rcr01dllatx37.netops.charter.com [24.175.32.146]
7 26 ms * 26 ms lag-14.dllstx976iw-bcr00.netops.charter.com [66.109.6.88]
8 * 28 ms 24 ms lag-0.pr3.dfw10.netops.charter.com [66.109.5.121]
9 28 ms 38 ms 64 ms ae3.cr7-dal3.ip4.gtt.net [208.116.212.9]
10 * 132 ms * ae8.cr11-dal3.ip4.gtt.net [89.149.139.205]
11 26 ms 26 ms 25 ms as2914.cr11-dal3.ip4.gtt.net [69.174.23.138]
12 * 36 ms 53 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
13 57 ms 57 ms 53 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
14 155 ms 85 ms 79 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
15 69 ms 69 ms 64 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
16 62 ms 67 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
17 * * * Request timed out.
18 67 ms 72 ms 70 ms 204.2.29.87
Optimum - Works
7 * * * Request timed out.
8 58 ms 52 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
9 54 ms 56 ms 56 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
10 61 ms 79 ms 54 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
11 54 ms 54 ms 54 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * 59 ms * 204.2.29.241
13 55 ms 57 ms 53 ms 204.2.29.6
Even in this example where its gone now, those addresses are not present:
Quote:
Originally Posted by Kaitlen
-
05-18-2023, 12:38 PMJanydI completely agree. It's not the ideal workaround, and none of us here should have to use it all the time, but it's at least a workaround. For now.Quote:
Originally Posted by LinaGiggles
Ideally, things will get sorted ASAP.
As for those Spectrum and AT&T traceroutes... guess this means the node situation between Dallas and California is worse than we thought, if I'm reading this right. -
05-18-2023, 01:42 PMZhexos
Greetings everyone,
The link to the form to submit your report has been updated.
We apologize for the inconvenience. -
05-18-2023, 03:22 PMTaleraRistain
Just because a couple of us were keeping track of this, things have cleared up again around 2am eastern time for me. Seeing the same drop on that 57 node to the 60s ms and game is playable.
10 16 ms 17 ms 16 ms ae-4.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.239]
11 60 ms 59 ms 59 ms ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
12 63 ms 64 ms 63 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
13 64 ms 64 ms 64 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 63 ms * 66 ms 204.2.29.241
15 64 ms 65 ms 63 ms 204.2.29.6
I think in a previous post it starts acting up again around 10 or 11 eastern in the morning. I won't be up to check at that time but I'll check as soon as I can after that. -
05-18-2023, 04:12 PMJanyd
Aye, the fact that this period of latency is a clockwork thing (from 10/11AM to 2AM Eastern) each day so far is very disconcerting but hopefully that will give the powers that be a clue as to what can be done to resolve the matter. See why that is, figure out a solution, and figure out why our connections with FFXIV suddenly act up on that schedule.
-
05-18-2023, 11:27 PMntt_routing_is_an_issue
Cross posting for visibility
More fun with graphs now that SE has acknowledged there are some network issues in NA. First the route map:
https://cdn.discordapp.com/attachmen...8627/image.png
Hop 4 is leaving AT&T's network, hop 5 obviously is entering NTT's network. First we look at our traffic statistics from AT&T to NTT:
https://cdn.discordapp.com/attachmen...6968/image.png
There are a couple of spikes, but pretty comfortable 20ms on average. Then we get to hop 6, which is another NTT node in Dallas:
https://cdn.discordapp.com/attachmen...6669/image.png
Here we start seeing maximums of almost 2 second latency, but still no packet loss. However, when we look at hop 7 which is the long haul from Dallas to Los Angeles (I previously thought this was San Francisco as I only had the device name to go off of) and we start to see a massive problem:
https://cdn.discordapp.com/attachmen...4488/image.png
From the 14th at 2030 CST we start seeing ~30% packet loss which lasts roughly 12 hours. There's a short window where it stabilizes until 0900 on the 15th then starts dropping packets again for two straight days. This resolves at roughly 1300 on the 17th, then starts back up at 1800.
I understand the SE NOC can not feasibly monitor service quality from every players origination point to the game servers, but when the support staff just copy and paste the same 3 boxes over and over again blaming our ISP's for the issue and never escalating it, it's pretty annoying. -
05-19-2023, 12:09 AMLahtori
My big question is why now? I've only been in North Texas for 10 months, but in that time I've always had AT&T Fiber and had ZERO problems in game. So what happened on Monday with AT&T and/or NTT that changed the latency for myself and countless others? I'm not an engineer and know very little about networking (enough to troubleshoot client side (my side) problems), but seems to me "something" was changed and it broke. Shouldn't be that difficult to figure out what and get a fix / rollback to when it wasn't an issue.
Shrug. I guess I'm looking into a VPN. I don't necessarily want to go 4 days without playing now. Already cancelled raid Tuesday, don't want to cancel another. And it's not looking good with patch day in 5 days and savage a week after.
I submitted my tracert, spoke with AT&T, not much more I can do. -
05-19-2023, 12:15 AMJanyd
Hmm, this looks promising.
Usually, this would be the starting time for the latency to start showing itself, according to the patterns we've noted but I just ran a traceroute, and here's what I've got (starting at the NTT nodes in Dallas):
8 28 ms 26 ms 26 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * 25 ms * ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
10 * * * Request timed out.
11 60 ms 84 ms 64 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 63 ms 64 ms 61 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
13 72 ms 73 ms 76 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 * 64 ms * 204.2.29.241
15 64 ms 68 ms 68 ms 204.2.29.122
Ran a Ping command as well:
Pinging 204.2.29.122 with 32 bytes of data:
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=60ms TTL=49
Reply from 204.2.29.122: bytes=32 time=59ms TTL=49
Reply from 204.2.29.122: bytes=32 time=58ms TTL=49
Ping statistics for 204.2.29.122:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 58ms, Maximum = 60ms, Average = 59ms -
05-19-2023, 12:19 AMMoonmoonMoonmoon
My trace route shows that it's hopping to Los Angeles and then Sacramento.
ignoring the first few hops to avoid doxxing myself, let's just say that I have a pretty direct connection to AT&T via my fiber connection... here's my tracert output this morning.
4 3 ms 3 ms 3 ms 71.154.103.72
5 * * * Request timed out.
6 13 ms 15 ms 15 ms 32.130.25.65
7 15 ms 33 ms 12 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
8 16 ms 13 ms 11 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
9 49 ms 46 ms 50 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
10 58 ms 56 ms 78 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
11 64 ms 54 ms 60 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
12 53 ms 52 ms 60 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
13 * * 52 ms 204.2.29.241
14 59 ms 63 ms 63 ms 204.2.29.7
This is pretty in-line with what you're getting from your tracert, too -
05-19-2023, 12:19 AMbrnslpyI've had AT&T Fiber ever since I started playing FFXIV a few years ago. This ATT+NTT node problem has been long term. But historically, it might be a problem once a quarter for 6 hours one night, then be fine for weeks. You think, network maintenance no problem, right?Quote:
Originally Posted by Lahtori
But in the past year, it's gotten more common, like once a month for 6-12 hours. Most I've experienced before is like "oh its Friday night, lag all night, then Saturday morning it's fine"
This current problem is the worst I've seen where it's lasted days which is kind of ridiculous for two Tier 1 / Tier 2 ISPs to have noticeable congestion like that and not try and re-route it. /vent -
05-19-2023, 12:20 AMCythrawllCan't really rollback network topology like that. it's not software. NTT probably has to reroute traffic due to hardware or routing locations that straight up don't exist anymore and didn't expect congestion at their Sacramento node to get so bad.Quote:
Originally Posted by Lahtori
It really is a congestion issue. That node is fine if you test it during off hours. I tested 2am central cuz curious last night and it was fine. -
05-19-2023, 12:21 AMMoonmoonMoonmoonProton VPN has a free tier that worked amazingly for me last night. My latency was a little high for my spoiled/pampered tastes but it was perfectly manageable at 100ms.Quote:
Originally Posted by Lahtori
-
05-19-2023, 12:24 AMTomoken
It's showing up clean for me now without VPN (have been playing the game with PIA + Michigan). Looks to be back to normal unless it's just working for now...?
https://i.ibb.co/S625Hg0/2023-05-18-...otter-Free.png -
05-19-2023, 12:28 AMbrnslpyThis does look promising. Been watching pings all morning. 65ms average (and normal behavior for me)Quote:
Originally Posted by Janyd
Right around 11:10AM EDT, pings jumped to 110ms (sigh) but so far are hanging around 70ms with a few spikes but a lot different than the last few days where it stayed at 120ms with packet loss.
Traceroutes on two of our problem nodes, 128.241.2.18 and 129.250.7.57, are also responding better than before (70-80ms down from 120ms)
Fwiw, after getting run around with ATT support last night, getting disconnected from their chat, and getting a phone call, they did say "should be fixed by Thursday morning" so lets hope so *fingers crossed* -
05-19-2023, 12:46 AMAth192The only thing that gives me pause is that yesterday we confirmed someone from Optimum was not having the same issues and he posted his Tracert that showed his traffic going through the exact same nodes you posted. With a 50ish ms response.Quote:
Originally Posted by ntt_routing_is_an_issue
I can confirm no loses either this morning but until we hit 5-9 PM with no issues I won't be convinced.Quote:
Originally Posted by TaleraRistain
-
05-19-2023, 12:53 AMntt_routing_is_an_issue
If I were to put money on it, the issue is between ae-5.a00.dllstx14.us.bb.gin.ntt.net and ae-2.r20.dllstx14.us.bb.gin.ntt.net. I have two internet connections coming in, and on my spectrum line, I do not hit ae-5.a00.dllstx14.us.bb.gin.ntt.net and have not had issues in ~3 weeks. AT&T routing looks like:
Hop 4: 32.130.25.65
Hop 5: ae-5.a00.dllstx14.us.bb.gin.ntt.net
Hop 6: ae-2.r20.dllstx14.us.bb.gin.ntt.net
While spectrum looks like:
Hop 9: as2914.cr11-dal3.ip4.gtt.net
Hop 10: ae-2.r20.dllstx14.us.bb.gin.ntt.net
Hop 11: ae-2.r24.lsanca07.us.bb.gin.ntt.net -
05-19-2023, 12:58 AMAth192Here is the first post showing none of those though:Quote:
Originally Posted by ntt_routing_is_an_issue
Quote:
Originally Posted by Krojack
Still the only thing I can find in common so far is that people who are having problems are going through a 129.250.4.9 or a 129.250.4.239 address. If anyone can prove this is wrong lmk. You will also see that the known working optimum tracert does not include either of these.
*Update, first packet drop of the day recorded 10:45 AM -11 AM CST*
Reply from 204.2.29.86: bytes=32 time=120ms TTL=50
Reply from 204.2.29.86: bytes=32 time=110ms TTL=50
Reply from 204.2.29.86: bytes=32 time=121ms TTL=50
Reply from 204.2.29.86: bytes=32 time=107ms TTL=50
Reply from 204.2.29.86: bytes=32 time=114ms TTL=50
Reply from 204.2.29.86: bytes=32 time=109ms TTL=50
Reply from 204.2.29.86: bytes=32 time=115ms TTL=50
Reply from 204.2.29.86: bytes=32 time=118ms TTL=50
Reply from 204.2.29.86: bytes=32 time=122ms TTL=50
Reply from 204.2.29.86: bytes=32 time=117ms TTL=50
Reply from 204.2.29.86: bytes=32 time=117ms TTL=50
Reply from 204.2.29.86: bytes=32 time=120ms TTL=50
Reply from 204.2.29.86: bytes=32 time=111ms TTL=50
Request timed out.
Reply from 204.2.29.86: bytes=32 time=113ms TTL=50
Reply from 204.2.29.86: bytes=32 time=121ms TTL=50
Reply from 204.2.29.86: bytes=32 time=117ms TTL=50
Reply from 204.2.29.86: bytes=32 time=104ms TTL=50
Reply from 204.2.29.86: bytes=32 time=112ms TTL=50
Reply from 204.2.29.86: bytes=32 time=113ms TTL=50
Reply from 204.2.29.86: bytes=32 time=115ms TTL=50
Reply from 204.2.29.86: bytes=32 time=115ms TTL=50
Reply from 204.2.29.86: bytes=32 time=121ms TTL=50
Reply from 204.2.29.86: bytes=32 time=119ms TTL=50
Reply from 204.2.29.86: bytes=32 time=113ms TTL=50
Reply from 204.2.29.86: bytes=32 time=108ms TTL=50 -
05-19-2023, 01:01 AMntt_routing_is_an_issueSure, and it's completely plausible that the nodes in my route map have an issue as well as other nodes. My stuff is going through Dallas, the post you linked is going through Chicago. It's an overall NTT issue.Quote:
Originally Posted by Ath192
Further, in your quote, they hop from Chicago to San Jose, I hop from Dallas to Los Angeles. The only explanation is that *something* in NTT's network is causing the issue. -
05-19-2023, 01:12 AMAth192Well I tried to follow up with NTT to press a bit more but this person is at least making it the ISP service's issue.Quote:
Originally Posted by ntt_routing_is_an_issue
---------------------------------------------------------------
Hey Victor,
Can you check this out for me if you have time?
In your email:
Your ping test you have a Response time of:
PING 204.2.29.87 (204.2.29.87): 56 data bytes
64 bytes from 204.2.29.87: icmp_seq=0 ttl=57 time=40.044 ms
In my tracert I have a response time of:
11 116 ms 117 ms * ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7] <<< The ms increase between Hop 8 and 11 is expected as the traffic traverses the NTT network between Dallas, TX to Sacramento, CA
Isn’t that a little strange that we are both from Texas, and getting vastly different ms response?
Also, I ran two different Tracerts to a hop earlier in the same route and get this:
Tracing route to ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
over a maximum of 30 hops:
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 29 ms 30 ms 30 ms 32.130.25.67
8 26 ms 30 ms 26 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * 26 ms 26 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
10 112 ms 115 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
11 141 ms 123 ms 142 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 72 ms 71 ms 73 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
In the full route, I get this:
Tracing route to 204.2.29.87 over a maximum of 30 hops
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 30 ms 31 ms 31 ms 32.130.25.67
8 31 ms 27 ms 31 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
9 * * * Request timed out.
10 111 ms * * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
11 118 ms 117 ms 118 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
12 * 175 ms 117 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
13 119 ms 118 ms 119 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
14 * * * Request timed out.
15 * 117 ms 117 ms 204.2.29.87
So that doesn’t match the description of the original email where:
8 40 ms 36 ms 30 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237] <<< This is the ATT node where the traffic is picked up by NTT
9 * * * Request timed out. <<< This is a node that is not responding to ICMP, does not indicate packet loss or latency
10 * * * Request timed out. <<< This is a node that is not responding to ICMP, does not indicate packet loss or latency
11 116 ms 117 ms * ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7] <<< The ms increase between Hop 8 and 11 is expected as the traffic traverses the NTT network between Dallas, TX to Sacramento, CA
12 118 ms 117 ms 116 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76] <<< The ms stayed consistent
13 117 ms 119 ms * xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18] <<< The ms stayed constsent
It seems like Hops 9 and 10 are:
9 * 26 ms 26 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
10 112 ms 115 ms * ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
And yet we can see it cannot always display them reliably. Can you see if there is something strange going on there? Or is this normal behavior?
-----------------------------------------------
Good day,
Per my previous email, I do suggest contacting your upstream provider to determine the reliability issue you are working on.
See inline for response, in red, below:
---
Victor Nadalalicea
Network Analyst, Global IP NOC
Isn’t that a little strange that we are both from Texas, and getting vastly different ms response?
>>> Not strange at all, but more expected. Pings and traceroutes are different in that sense yes. Traceroute calculates how many hops it takes to its destination, and how long each hop takes. With Ping, this app just calculates the time per echo reply responses. Therefore, both numbers will always result in different ms readings. You should never receive the same results in ms.
So that doesn’t match the description of the original email where:
8 40 ms 36 ms 30 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237] <<< This is the ATT node where the traffic is picked up by NTT
9 * * * Request timed out. <<< This is a node that is not responding to ICMP, does not indicate packet loss or latency
10 * * * Request timed out. <<< This is a node that is not responding to ICMP, does not indicate packet loss or latency
11 116 ms 117 ms * ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7] <<< The ms increase between Hop 8 and 11 is expected as the traffic traverses the NTT network between Dallas, TX to Sacramento, CA
12 118 ms 117 ms 116 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76] <<< The ms stayed consistent
13 117 ms 119 ms * xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18] <<< The ms stayed constsent
>>> Traceroute to different destination IP addresses, you should expect different paths, because the destination network address is different and depending how the route path algorithms are calculated will determine the path it takes. The above traceroutes to two different destination addresses are expected.
And yet we can see it cannot always display them reliably. Can you see if there is something strange going on there? Or is this normal behavior?
>>> We only have visibility within our network, and I have shown there is no issue that would cause your reliability issues.
-----------------------------------------------------------
I don't think they're losing sleep over this. -
05-19-2023, 01:23 AMJanyd
Bummer, just ran a few Ping commands, and the latency on both of them jumped up to 98-105ms.
-
05-19-2023, 01:23 AMntt_routing_is_an_issueThis is largely what I expect. When I was experiencing this on both AT&T (5 gig fiber) and Spectrum (1 gig cable) SE told me to call the ISP. I knew what the response from AT&T and Spectrum would be, and I agree with them since as I leave both their networks I have good latency and no packet loss. AT&T eventually escalated me to some engineers in Dallas who confirmed what I already knew, the issue is inside NTT. This is evidenced by the fact that I have no problems with the first NTT hop in Dallas. The second hop, also in Dallas, can from time to time introduce some latency, but the packet loss occurs when transiting to NTT in California. By this time, we are already inside NTT's network and it completely outside of our ISP's control. NTT announces their routes, and the ISP uses what is broadcast. It's just a stall tactic that has unfortunately become common in the industry to save face. Also interesting that I have been complaining about this since early April and it wasn't until some site wrote an article about it that SE finally calls NTT to work the issue.Quote:
Originally Posted by Ath192
-
05-19-2023, 01:30 AMJanyd
At this rate, wouldn't it behoove SE to demand that NTT take those faulty nodes offline until fixes can be made?
-
05-19-2023, 01:31 AMAth192IDK that SE will get anywhere. Lets hope they do though. I don't see the difference between them telling an intern to see what's up and us doing the same.Quote:
Originally Posted by ntt_routing_is_an_issue
PIA has decent prices I guess....
SE peeps! just move the Datacenter to Dallas plz. xD -
05-19-2023, 01:35 AMntt_routing_is_an_issueSE has more pull since they are the customer of NTT. If random Joe calls NTT they are going to brush it off, but if SE calls it's an issue because there are SLA's in place.Quote:
Originally Posted by Ath192
-
05-19-2023, 01:35 AMJanyd
SE datacenter in Texas? Wouldn't that be nice?!
As a fellow Texan (San Antonio/South Texas region), I'd be down for that.
Other MMORPGs have had, or currently have, Texas-based datacenters. Not naming any games in particular, but suffice it to say they are not made by Square Enix.
Regardless, moving the NA datacenters out of California and to a more centralized part of the US would help a lot of us out significantly.