Bad lag again with NTT's NA server network

So far so good, let's see if it lasts come reset day when people start logging back in lol

Yeah Rekts. Let's hope. Tho yeah, I DID find a bug on my island (the programming kind..) which I reported, which I wouldn't of before since I wasn't doing much crafting which means no furniture on my island to speak of until yesterday. :)

Connection so far appears fine again, though I have been noticing transient 100ms spikes that last for small seconds of time before returning to normal. I've been raiding and doing things off-vpn and no true problems persist. So, that's nice, maybe.

No more issues here as well so far.

Same, all's good so far.

Remember, there is hotfix maintenance early Tuesday morning (2AM to 6AM Eastern), so we'll see...

Stay vigilant.

Quote:

---

3 5 ms 3 ms 4 ms 71.155.29.140
4 * * * Request timed out.
5 10 ms 7 ms 7 ms 32.130.25.65
6 15 ms 12 ms 9 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
7 11 ms 11 ms 28 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
8 * * * Request timed out.
9 43 ms 43 ms 43 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
10 42 ms 43 ms 44 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
11 42 ms 43 ms 44 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * * * Request timed out.
13 45 ms 43 ms 43 ms 204.2.29.6

---

Either the lsanca07 node is completely broken (timed out), or today my traffic is not routed to that node. More or less, this is a normal tracert for me. Cautious optimism may be allowed!

Quote:

---

Originally Posted by Janyd

hotfix maintenance

---

Most likely to fix the knockback resist exploit. SE fixing bugs in their game =/= NTT's node issues.

Keep the pitchforks sharp and the torches burning, however.
Still not buying FF16 or F7R2 until this is 100% confirmed fixed.

Page 101 and counting!

If anyone was going fanfest should still bring up the node issues because they can't run the severs for 10 extra years with this problem.

Knockback Resist Exploit?

Quote:

---

Originally Posted by ArcadiaP

Knockback Resist Exploit?

---

This exploit here: https://na.finalfantasyxiv.com/lodes...851b0067609498

There's also a separate knockback problem in the 6.4 Trial (normal/extreme), so I'd imagine that's on the to-do list.

I figure that's what's being addressed first and foremost. They're also likely working on a client-side fix for sound issues I've seen around.

As for the nodes, still holding steady at 58ms to 60ms on my end today.

It's plausible that AT&T changed the routing up a bit. That or some nodes being offline may be a blessing in disguise.

Quote:

---

Originally Posted by Lahtori

Either the lsanca07 node is completely broken (timed out), or today my traffic is not routed to that node. More or less, this is a normal tracert for me. Cautious optimism may be allowed!

---

Keep in mind routers along the way don't always respond to pings, and its completely normal.

No replies today, is it finally over?

Quote:

---

Originally Posted by burgerfall

No replies today, is it finally over?

---

Personally I'm reserving judgement (and not resubscribing) until I see a solid week without this issue, but there does seem to have been some sort of potential change starting on Friday. Could be that they fixed something, could also be that the simultaneous release of Diablo 4 early access and Street Fighter 6 drew enough traffic away that the congestion isn't bad enough to cause problems.

Fingers crossed, but I'm not necessarily holding out hope at this point. It'd be nice to play my silly explosion cat again, though.

I agree, plus lately, since the ping issue, my game would freeze a bit not sure if it's my end or the severs. but ya.

I'm cautiously optimistic. I'm still not getting the kind of (good) performance I did before this issue started. But it's a thousand times better than it was at the worst, so I'll take it for now.

I'm still having consistency issues, personally speaking. It seems to get turbo bad around 830pm CST and just be flaky other times.

Quote:

---

laurayka@yshtola  ~  traceroute neolobby02.ffxiv.com
traceroute to neolobby02.ffxiv.com (204.2.29.6), 30 hops max, 60 byte packets
1 172.29.176.1 (172.29.176.1) 0.189 ms 0.182 ms 0.179 ms
2 10.10.10.1 (10.10.10.1) 3.056 ms 3.563 ms 1.409 ms
3 192.168.1.254 (192.168.1.254) 7.145 ms 6.423 ms 6.623 ms
4 108-226-52-1.lightspeed.fyvlar.sbcglobal.net (108.226.52.1) 7.876 ms 7.639 ms 7.870 ms
5 75.14.128.90 (75.14.128.90) 9.284 ms 9.508 ms 9.505 ms
6 * * *
7 * * *
8 * * *
9 32.130.25.65 (32.130.25.65) 21.901 ms 17.874 ms 21.130 ms
10 ae-5.a00.dllstx14.us.bb.gin.ntt.net (129.250.8.237) 16.351 ms 22.525 ms 23.702 ms
11 ae-2.r20.dllstx14.us.bb.gin.ntt.net (129.250.4.9) 31.911 ms 31.908 ms 31.905 ms
12 ae-2.r24.lsanca07.us.bb.gin.ntt.net (129.250.7.69) 97.653 ms 97.649 ms *
13 ae-5.a01.scrmca03.us.bb.gin.ntt.net (129.250.2.7) 55.068 ms 55.066 ms 56.319 ms
14 ae-1.a00.scrmca03.us.bb.gin.ntt.net (129.250.4.76) 61.233 ms 56.371 ms 60.339 ms
15 xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net (128.241.2.18) 57.992 ms 57.724 ms 58.996 ms
16 204.2.29.241 (204.2.29.241) 54.629 ms 54.323 ms *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

---

Been running checks all weekend that were good and just now about 9pm EDT and things are clear

6 9 ms 15 ms 13 ms cgcil402igs.ip.att.net [12.122.133.161]
7 12 ms 8 ms 10 ms 192.205.32.194
8 13 ms 21 ms 15 ms ae-4.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.239]
9 55 ms 59 ms 55 ms ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
10 58 ms 58 ms 57 ms ae-4.a00.scrmca03.us.bb.gin.ntt.net [129.250.7.57]
11 58 ms 66 ms 62 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * * * Request timed out.
13 59 ms 58 ms 58 ms 204.2.29.6

I'm still a bit gun shy to try doing EX without a VPN but I may try tonight

The issue is mostly resolved for me at least. I live in the Atlanta, GA area, and pings have been pretty stable, even improved by a few ms. I have gotten a minor lag spike a couple of times, but I got those before all this mess started.

Quote:

---

3 4 ms 3 ms 3 ms 71.155.29.140
4 * * * Request timed out.
5 7 ms 7 ms 7 ms 32.130.25.65
6 26 ms 27 ms 24 ms ae-5.a00.dllstx14.us.bb.gin.ntt.net [129.250.8.237]
7 26 ms 13 ms 10 ms ae-2.r20.dllstx14.us.bb.gin.ntt.net [129.250.4.9]
8 82 ms 85 ms 86 ms ae-2.r24.lsanca07.us.bb.gin.ntt.net [129.250.7.69]
9 42 ms 43 ms 44 ms ae-5.a01.scrmca03.us.bb.gin.ntt.net [129.250.2.7]
10 44 ms 43 ms 43 ms ae-1.a00.scrmca03.us.bb.gin.ntt.net [129.250.4.76]
11 44 ms 43 ms 44 ms xe-0-0-5-0.a00.scrmca03.us.ce.gin.ntt.net [128.241.2.18]
12 * * * Request timed out.
13 45 ms 43 ms 43 ms 204.2.29.6

---

Today got some response from lsanca07 and its still in the 80ms. At least the other nodes seem to be consistent and normal.

Yeah, same for me, Lahtori. Averaging around 58 to 60 ms at the Aether DC on the end of my traceroutes.

Was able to do Normal reclears of Anabaseios and roulettes with no issues, with the VPN turned Off for testing.

It feels like it's back to the way it was before all this mess, at least for the moment.

Still not sure if AT&T did some rerouting, or if the launch of Diablo 4 and Street Fighter 6 were both factors in easing traffic to those nodes, or if it's something else, but so far so good.

Give it another few days before we can call it "solved" at least.

Has the audio issue been solved in this latest patch? I was one of the ones where the audio cut-out after 30-40 minutes. I've help back on progressing quest line as i would like to hear the sound. The issues was recognized two weeks ago - and it was said it would take two weeks to resolve - but i have not heard any further comment from SE as to whether the issue has been resolved....

Quote:

---

Originally Posted by felgryn

Has the audio issue been solved in this latest patch? I was one of the ones where the audio cut-out after 30-40 minutes. I've help back on progressing quest line as i would like to hear the sound. The issues was recognized two weeks ago - and it was said it would take two weeks to resolve - but i have not heard any further comment from SE as to whether the issue has been resolved....

---

Yes, it was part of the hotfixes that were included in this week's maintenance.

https://na.finalfantasyxiv.com/lodes...d378cf14ec4020

I ran EX last night multiple times without the VPN and with no lag. Still not going to say it's fixed until this is consistent over longer than it's been.

Still no issues, but as others have said I'm not going to call it "fixed" until we get a true stress test, which fortunately will probably be a little while.

Quote:

---

Originally Posted by Janyd

Yes, it was part of the hotfixes that were included in this week's maintenance.

https://na.finalfantasyxiv.com/lodes...d378cf14ec4020

---

Great News! Thanks Janyd, i didn't catch that one sentence in the update - back to the quest line!

So It seems that it has eased up. So much so that nord VPN is actually make my connection far worse, especially in frontlines.

So I get to save $12 again!

Everything looks like it's working now, which is good.

While we will likely never know what the true root cause is, as no one involved will ever have any reason to reveal the cause of a 3+ week issue, we can make a few inferences, and a reasonable idea as to what happened, based on the evidence provided in this thread.

It looks most likely that NTT had a dying node. They were resistant to investigate and resolve this issue, as the replacement of the node would be a cost measured both in a large amount of time and a large amount of money. During this slow death, the node would thrash the very moment it became saturated, causing excessive latency, and worse packet loss.

After about 3 weeks, it is very likely that the node simply died, and automatic network routing within the NTT infrastructure re-routed around the failed node (rather than through it), and thus the issue was resolved. And as the device has entirely failed, NTT will need to replace it, and in the meantime the automatic reroute around it has resolved the immediate issue.

Looking at logs, the specific node which was causing the vast majority of issues within the NTT infrastructure no longer appears in traceroutes, which adds credence to this theory. But alas, we will very likely never know.

Much love, glad we got through this.

Good thread, see you guys next time! Maybe we can get the next one to 200 pages! :cool:

Because this can and most likely will happen again as long as all the NA data centers are centralized in one location.

even if the latency issues died out. we should bring up this topic if anyone going to fanfest. because this could happen to anyone who has other ISP's and it is more annoying to deal with. Plus this wouldn't fly on other games with node issues. if Square wants xiv to be online then they gotta fix there severs nodes for the next 10 years. because this game will be active for that long.

Quote:

---

Originally Posted by Ahumaya

Everything looks like it's working now, which is good.

While we will likely never know what the true root cause is, as no one involved will ever have any reason to reveal the cause of a 3+ week issue, we can make a few inferences, and a reasonable idea as to what happened, based on the evidence provided in this thread.

It looks most likely that NTT had a dying node. They were resistant to investigate and resolve this issue, as the replacement of the node would be a cost measured both in a large amount of time and a large amount of money. During this slow death, the node would thrash the very moment it became saturated, causing excessive latency, and worse packet loss.

After about 3 weeks, it is very likely that the node simply died, and automatic network routing within the NTT infrastructure re-routed around the failed node (rather than through it), and thus the issue was resolved. And as the device has entirely failed, NTT will need to replace it, and in the meantime the automatic reroute around it has resolved the immediate issue.

Looking at logs, the specific node which was causing the vast majority of issues within the NTT infrastructure no longer appears in traceroutes, which adds credence to this theory. But alas, we will very likely never know.

Much love, glad we got through this.

---

You might be right, Ahumaya. I haven't noticed that node come up in my traceroutes since the start of the month.

Hopefully, the node kicking the bucket was the "fix" for the moment.

I just resubbed yesterday, after all the good reports here. For the most part, the game plays smoothly. But, not in alliance raids. Skills still hang, then rapidly pop off. It's playable, but I don't think it's really fixed on NTT's end. Just better. I'll take it one month at a time. If, when the "new" wears off of Diablo 4 and SF6, things get bad again, then we'll know it was player absence, not NTT, that made things better.

I'm on AT&T, and before the NTT fix, I could avoid issues by using a VPN. However after fix, I now get periodic lag spikes both on and off VPN (I haven't kept track of how often, but it's like once every 30 mins maybe). I'm not sure if this is still an NTT issue, I haven't tried (or really have the tools handy) to do the deeper time-based analysis others have done here.

Over the last few days, my AT&T Internet was acting up, so I had a tech stop by yesterday to look into it.
Turns out, some recent severe weather blew the cover off and there were exposed wires. Suffice it to say, that was a quick fix with a new wire cover and a thorough check of the wires and nearby nodes.

It took an hour-plus of work, but the tech did their job well.

Quote:

---

Originally Posted by Janyd

Over the last few days, my AT&T Internet was acting up, so I had a tech stop by yesterday to look into it.
Turns out, some recent severe weather blew the cover off and there were exposed wires. Suffice it to say, that was a quick fix with a new wire cover and a thorough check of the wires and nearby nodes.

It took an hour-plus of work, but the tech did their job well.

---

Just wanted to thank you for this, I did end up running and logging several pings to various dns servers (google, cloudflare, etc), as well as a couple FFXIV servers. I found that when I started having lag, my pings across the board were bad. I contacted AT&T and they isolated the issue to my modem, so they're sending a replacement. Thanks for the tip!

It never hurts to call them now that the bad node has been kicked; I've had decent luck with that as long as I was specific and it was their system that was going wonky. Their tech support is NOT the best and with us, since we're Uverse (we get phone/internet/cable though our gateway), when they think we need a new gateway, they always send the bloody wrong gateway. We have to have a tech come with one to make sure we get the right one. (And since they tend to reset the connection with us, which potentially could get the brother that works from home and plays WoW VERY mad, I try to be careful when I call. As I kept saying, rather not call over ONE game.)

But yeah, at this point, if you're still getting lag, ping more than XIV stuff then call them. BE SPECIFIC on what's going wonky and they will likely fix it.

Just updating that I've personally continued to raid without a VPN and am not experiencing any issues. Everything is as it used to be. Is it permanent? No idea. Anyone notice how the issue magically went away on a new month? Who knows what July has in store for us.

I've been playing for the last 3 days without a VPN with no issues. About once a night, I'll see one person shoot across the screen but I think it's something on their end. I think this may be resolved.

I don't think this problem ever got resolved, because I still see the same packet loss and warnings coming from the same nodes from weeks prior.
These are the nodes carrying traffic between the NTT Nodes from Dallas-Fort Worth to Los Angeles.

https://i.gyazo.com/9818ccd702910e48...ed4ea8c3ab.png
In fact, over the past few weeks, my ping has went from 67 ms to 85 ms now on average. Not even ExitLag has been able to mitigate this problem.

I would go try and use mudfish if your latency is bad. Exitlag is only popular because streamers shove it down everyone's throat. Mudfish, any node should be fine, I personally used US West (Fremont, CA - Aliyun). No sub needed and it's cheap.

Quote:

---

Originally Posted by ArcadiaP

I would go try and use mudfish if your latency is bad. Exitlag is only popular because streamers shove it down everyone's throat. Mudfish, any node should be fine, I personally used US West (Fremont, CA - Aliyun). No sub needed and it's cheap.

---

I have Mudfish as well and it's not helping the problem either.

Fun fact: Mudfish, on certain games (like Maplestory), is actually worse because it tethers to all connections and chews through all of your data.