FIX THE LAG and 90000 Errors petition!

[Cervani]

This is only happening to me when I use my hotspot at work to connect to the game. I never drop connection when I'm at home. It's odd too, because I get error 90000 exactly once an hour and once it happens once it will happen at the exact same time every hour. So if it happened at 5:10, I will guaranteed be kicked off the game at 6:10, 7:10 etc etc. I did run a tracert, and it looks exactly like squids:

Tracing route to 199.91.189.28 over a maximum of 30 hops

1 * * * Request timed out.
2 60 ms 58 ms 59 ms 224.sub-66-174-162.myvzw.com [66.174.162.224]
3 44 ms 68 ms 64 ms 17.sub-66-174-63.myvzw.com [66.174.63.17]
4 63 ms 65 ms 46 ms 194.sub-69-83-70.myvzw.com [69.83.70.194]
5 65 ms 64 ms 68 ms 113.sub-69-83-83.myvzw.com [69.83.83.113]
6 65 ms 66 ms 66 ms 241.sub-69-83-70.myvzw.com [69.83.70.241]
7 71 ms 65 ms 65 ms 210.sub-69-83-70.myvzw.com [69.83.70.210]
8 72 ms 71 ms 72 ms 0.sub-69-83-83.myvzw.com [69.83.83.0]
9 75 ms 74 ms 67 ms 0.sub-69-83-83.myvzw.com [69.83.83.0]
10 67 ms 64 ms 67 ms 209.sub-69-83-66.myvzw.com [69.83.66.209]
11 72 ms 70 ms 75 ms ae57.bar2.Philadelphia1.Level3.net [4.30.46.5]
12 88 ms 85 ms 85 ms ae-6-6.ebr1.NewYork1.Level3.net [4.69.133.170]
13 84 ms 79 ms 78 ms ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]
14 85 ms 84 ms 79 ms ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]

15 84 ms 79 ms 80 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
16 85 ms 86 ms 85 ms 192.34.76.10
17 85 ms 84 ms 81 ms 199.91.189.242
18 92 ms 91 ms 86 ms 199.91.189.28

Edit: It just happened again, this time while playing League of Legends so I think I can pretty much confirm it's my hotspot doing it.

[ZohnoReecho]

your hotspot may have a DHCP with a 1h lease time

[Jadi]

Hi all. I'm a super genius network engineer with 15 years experience working for some of the largest tech companies in the business. /wave Hi.

Your pings mean nothing, forget about them. Don't post them. They just don't give any meaningful info. Why? Because the traffic is ICMP. Why more? Because ICMP isn't UDP and what is going on is ISP's are incorrectly applying a low priority protocol scheduling to the UDP game traffic, NOT ICMP traffic. They do this because they mistake the traffic as P2P traffic, specifically torrent traffic.. and yes, it's similar so it's easy to see how this occurred. ICMP usually has high scheduling.

SE's ability to control the protocol scheduling priority of your ISP is very limited. They basically contact them and ask for a whitelisting. This is difficult and tedious for them. I mean you've called your ISP right? Their customer service is shit and you actually pay them, SE doesn't.

However there is something else they can do.. they could code the game to use a VPN wrapper and tunnel it into their own datacenter. This would change the type of traffic and give it a higher scheduling priority by ISP's. Just a thought.. seems easier than attempting to contacting every ISP on the planet and beg them to fix it.

Also this doesn't mean that other problems don't exist as well, naturally they do.

[Cervani]

your hotspot may have a DHCP with a 1h lease time

I wonder how I could fix that then, if that is indeed the case. I use the Straight Talk Unimax u240c, and sadly it's admin panel is very lacking in regards to adaptability and options.

[Dalavon]

Hi all. I'm a super genius network engineer with 15 years experience working for some of the largest tech companies in the business. /wave Hi.

Your pings mean nothing, forget about them. Don't post them. They just don't give any meaningful info. Why? Because the traffic is ICMP. Why more? Because ICMP isn't UDP and what is going on is ISP's are incorrectly applying a low priority protocol scheduling to the UDP game traffic, NOT ICMP traffic. They do this because they mistake the traffic as P2P traffic, specifically torrent traffic.. and yes, it's similar so it's easy to see how this occurred. ICMP usually has high scheduling.

SE's ability to control the protocol scheduling priority of your ISP is very limited. They basically contact them and ask for a whitelisting. This is difficult and tedious for them. I mean you've called your ISP right? Their customer service is shit and you actually pay them, SE doesn't.

However there is something else they can do.. they could code the game to use a VPN wrapper and tunnel it into their own datacenter. This would change the type of traffic and give it a higher scheduling priority by ISP's. Just a thought.. seems easier than attempting to contacting every ISP on the planet and beg them to fix it.

Also this doesn't mean that other problems don't exist as well, naturally they do.

So basically... We can't do anything

[Jadi]

So basically... We can't do anything

yeah.. not really. There are some programs out there but they cost. that or you can get yourself a vpn.. it seems to work.

[Kindread]

And your ISP does not have just one layer of support... if you've called the help desk and a technician came on site, you just went through a couple layers of support getting that accomplished. The point is you need to ask specifically to be escalated to Tier3 support--those are the guys that can actually look at the larger sections of the network--the tier1 and tier2 people are basically just there to handle the simple problems like downed modems, cabling issues, etc. that affect localized connectivity issues. Tier3 and above deals with more advanced issues like routing and such, which is where the bulk of our problems reside.

Not to split hairs, but my ISP really does only have 1 level of support, living in the mountains in Idaho there are 3 guys that answer phones and drive the trucks that respond to all tech support issues. I know them all by first name.

[Raist]

Hi all. I'm a super genius network engineer with 15 years experience working for some of the largest tech companies in the business. /wave Hi.

Your pings mean nothing, forget about them. Don't post them. They just don't give any meaningful info. Why? Because the traffic is ICMP. Why more? Because ICMP isn't UDP and what is going on is ISP's are incorrectly applying a low priority protocol scheduling to the UDP game traffic, NOT ICMP traffic. They do this because they mistake the traffic as P2P traffic, specifically torrent traffic.. and yes, it's similar so it's easy to see how this occurred. ICMP usually has high scheduling.

SE's ability to control the protocol scheduling priority of your ISP is very limited. They basically contact them and ask for a whitelisting. This is difficult and tedious for them. I mean you've called your ISP right? Their customer service is shit and you actually pay them, SE doesn't.

However there is something else they can do.. they could code the game to use a VPN wrapper and tunnel it into their own datacenter. This would change the type of traffic and give it a higher scheduling priority by ISP's. Just a thought.. seems easier than attempting to contacting every ISP on the planet and beg them to fix it.

Also this doesn't mean that other problems don't exist as well, naturally they do.

XIV actually relies on TCP... which is why we see so much rubber-banding and such. If it were relying on UDP, we'd see more FFXI like behavior. If you'd run a netstat or look at it in resmon, you'll see that all the connections are TCP and not UDP. So, there is the built in potential for throttling within the protocol itself that can be triggered by congestion. The P2P vs game flag in the headers was brought to SE's attention in beta, and hashed out for quite a while with some UK ISP's. The thread about gathering data has hopefully helped to resolve it further with more ISP's--but that is old news and can be reasonably assumed those throttling rules are being addressed on an ongoing basis if not already handled at this point. Hard to say for certain, since they have made no official statement--but it's an old problem that has been addressed between SE and some ISP's already, so there is reason to consider taking that out of the discussion I guess. That leaves us with addressing issues native to the protocol and routing analytics that have been identified already as a problem, and testing has been done to support that theory (VPN/proxy, as well as some professionals that have presented their own analysis of what they've picked up on with sniffers and looking glasses the past 3 or 4 months). The trick now is in getting the ISP's to own up to these as valid concerns worth investigating, which is why there is this ongoing push to get people to file proper support tickets with their ISP's.

And we aren't providing simple pings... we are using tracert and pathping reports. This DOES have it's uses. It shows the actual route we are taking, as well as gives hints at where congestive failure may be on the horizon (if not actually happening). Even though it is using ICMP, it still works to show signs of congestion issues. If you will actually look at what we've been pointing out, there are recurring subnets operated by a select group of 3rd party ISP's (and often isolated to a specific region) that shows very erratic ping responses. It frequently gets to the point that you will see intermittent timeouts at those same points in the route. And that's not a *** event in a single report that may indicate a rule to flat out not reply to ICMP--but where a hop actually responds and then drops it in the same run. We've had pathpings showing varying degrees of rejection at these points--7% one time, 66% another, and so and so forth. This is a pretty clear indicator something is going on that needs to be investigated with that route.

If you combine enough of this data, you will begin to see a pattern. For instance, the PAIX and SIX exchanges have been hosed for some time now (and it's not uncommon by the way.. this is a returning problem that goes back to the early days of WoW, and even the original Diablo release). Flip through the pages and review the threads complaining about issues in the California corridor. This isn't a coincidence---it's a pretty clear indicator that some exchanges may be getting overloaded during primetime. This <should> give the ISP's reason to review their analytics to see if they need to switch to different peering/transit/exchange partners to try to relieve some of the congestion.

The trick is providing them with sufficient data to spark an interest into investigating along such lines. What if they got flooded with pathping and tracert reports from a few thousand subscribers... maybe that would get their attention? And it's not unreasonable to say thousands either. SE has already published a stat of hitting 5k+ simultaneous connections per world. I think we could gain some traction with the ISP's involved if enough people would get proactive with the reporting.

Remember, it's a market out there where your ISP purchases access to these routes from someone else. Just like you have the means to change who you purchase many services from, so does your ISP have the means to purchase access to alternate routes to get us to Ormuco's segments. And yes, that is a big part of the solution. Just look at how many people have resolved their problems with VPN or Proxy services to change how they are being routed.

So, no... we are not helpless to affect change on this matter. For a long term, more permanent fix, it just takes a few moments of everybody's time to generate a proper support ticket. I sent a total of 3 emails, but only really needed to send one. I saw changes after the first email was sent, but I voluntarily sent 2 followups with tracerts because twice they made a change that made things worse than the previous route change. For a short term fix, there are the VPN/Proxy services--some of which are actually free (maybe not as robust as the premium ones, but they can still improve things).

[RaineMagus]

Your pings mean nothing, forget about them. Don't post them. They just don't give any meaningful info. Why? Because the traffic is ICMP. Why more? Because ICMP isn't UDP and what is going on is ISP's are incorrectly applying a low priority protocol scheduling to the UDP game traffic, NOT ICMP traffic. They do this because they mistake the traffic as P2P traffic, specifically torrent traffic.. and yes, it's similar so it's easy to see how this occurred. ICMP usually has high scheduling.

It's true that many ISP's categorize UDP traffic as "torrent" with a blanket catch-all rule .. although this game uses TCP and not UDP.
(still this doesn't mean TCP catch-all rules don't exist, or that layer 7 inspection can't mistake traits of one protocol for another .. pretty common)


You're absolutely right in that ISP's have been to known to do stupid things like the above -- *catch-all rules* -- even on something as simple as uncommon 'ports' for traffic (eg, if it's not port 22, 80, 443, 1723, 500, 4500 .. the common messenger ports [MSN, ICQ, AIM], etc etc .. consider it low priority data).. and that many gateways also shape ICMP. Yet, I wouldn't go so far as to call ICMP 'worthless' as a tool for diagnostics.

For instance: If someone sees high latency (via ICMP or UDP ping) to a destination, and that high latency spike, or loss, corresponds to a behavior that they're also consistently seeing in a game (or internet behavior in general from their end) .. it really doesn't matter the 'cause', as those pings are still helpful in determining at which gateway the problem originates [even if the absolute cause is not revealed]. A traceroute isn't useless as long as you interpret it with a grain of salt, and understand that the problems shown may or may not actually be issues at all, or effect other protocols.


I'll elaborate (especially for everyone else who's submitting traces). Below is an example ran quickly from one of my servers in LA, to Japan. While this may look like an imperfect route ... there's absolutely nothing wrong with it / no packet loss to the destination whatsoever.

traceroute to 124.150.157.52 (124.150.157.52), 30 hops max, 60 byte packets
1 * * * << isn't a problem .. the hop is simply not responding to messages addressed to itself
2 * * *
3 * * *
4 gi8-0-0.cr1.nrt1.asianetcom.net (202.147.0.121) 135.188 ms 115.191 ms 107.194 ms
5 gi1-0-0.gw1.nrt5.asianetcom.net (202.147.0.178) 107.570 ms 107.572 ms 119.566 ms
6 squareco.asianetcom.net (203.192.149.210) 98.913 ms 98.935 ms 98.809 ms
7 61.195.56.129 (61.195.56.129) 140.494 ms * * << this hop is shaping pings addressed "to itself"
8 219.117.144.66 (219.117.144.66) 107.804 ms 107.758 ms 107.728 ms << see above and final comment
9 * * * << same as hop #1
10 * * *
11 * * *
12 * * *
13 124.150.157.52 (124.150.157.52) 99.209 ms 99.103 ms 99.403 ms << 100% of the pings to here are 'clean' at ~99ms

PING 124.150.157.52 (124.150.157.52) 56(84) bytes of data.
64 bytes from 124.150.157.52: icmp_seq=1 ttl=244 time=99.1 ms
64 bytes from 124.150.157.52: icmp_seq=2 ttl=244 time=99.1 ms
64 bytes from 124.150.157.52: icmp_seq=3 ttl=244 time=99.2 ms
...
--- 124.150.157.52 ping statistics ---
2000 packets transmitted, 2000 received, 0% packet loss, time 2000019ms
rtt min/avg/max/mdev = 99.131/99.180/99.234/0.417 ms


Seeing as the pings to the destination are 'clean' over a long term, we can conclude that all prior drops seen in the traceroute (concurrently) have no bearing on the destination's responses. We could additionally run the same trace over and over repeatedly, or run a pathping, and we'd see 100% timeouts for TTL 1, 2, 3, 9, 10, 11, & 12.

To everyone else looking at the above and thinking "Why the heck would this be done in the first place?". The general line of thought behind it is that a gateway's 'primary' purpose is to forward data (*not to respond to data addressed to itself*). ICMP bombing is also a very common form of DoS attack, and the protection against it is usually rate limiting, prioritization / queuing, or just not responding at all.



Now to make up a bit of data to illustrate an obvious problem...

Let's say that the above tracert instead looked like the following, randomly, during periods of high delay that we're seeing ingame.

traceroute to 124.150.157.52 (124.150.157.52), 30 hops max, 60 byte packets
1 core-router (xxx.xxx.xxx.xxx) 0 0 0 (near zero latency)
2 xxx.coresite.com (xxx.xxx.xxx.xxx) 0 0 0 (near zero latency)
3 xxx.coresite.com (xxx.xxx.xxx.xxx) * * ms 400.xxx ms [connection to asianet ... in California]
4 gi8-0-0.cr1.nrt1.asianetcom.net (202.147.0.121) 357.188 ms * 387.194 ms
...
dst 124.150.157.52 (124.150.157.52) 400.209 ms * 390.403 ms

If all hops after a certain point are showing the problem at some time of the day, and we're seeing the above where we're getting several hundred ms added (& loss) to each ping (*before the international backbone*) ... then we could conclude that gateway is probably overloaded, and this is the cause of the lag we're feeling ingame right-now.

Useless? I'd think not. [it shows the problem is likely in California, and not Japan]

Will someone care (the owner of that gateway) and fix / do something about the issue? That's a whole other can of worms...

SE's ability to control the protocol scheduling priority of your ISP is very limited. They basically contact them and ask for a whitelisting. This is difficult and tedious for them. I mean you've called your ISP right? Their customer service is shit and you actually pay them, SE doesn't.

However there is something else they can do.. they could code the game to use a VPN wrapper and tunnel it into their own datacenter. This would change the type of traffic and give it a higher scheduling priority by ISP's. Just a thought.. seems easier than attempting to contacting every ISP on the planet and beg them to fix it.

That all assumes that layer 7 inspection is even the root of the problem to begin with. If it was, concealing XIV's traffic through a tunnel would indeed solve the problem ... It'd be unwise to embed a protocol such as SSH into XIV's netcode simply on a whim though. Compression and encryption of all traffic isn't exactly trivial in terms of server overhead, nor is the price of double encapsulation.

-Rather than adopting a "tunneling protocol", perhaps running all of the game's traffic through a simple shift-cipher would be sufficient to throw off whatever rule(s) are mistakenly identifying it. Unless of course there's a catch-all for everything other than known protocols by something like port. In that case, using a port such as 80 or 443 would be an immediate fix.


EDIT: Should go into a bit of "why" this usually isn't done. IP addresses are not an unlimited resource, and thousands of servers can technically be operated on the same address. To use only "common" protocol ports (eg, HTTP, HTTPS, SSH, FTP, etc) puts a huge constraint on how many servers can be run on a single address.

As IPv4 addresses are a limited resource.. Whenever requesting an address block, usually a company has to provide some 'rationale' (why they "need" it) in order to receive them.

[Raist]

Not to split hairs, but my ISP really does only have 1 level of support, living in the mountains in Idaho there are 3 guys that answer phones and drive the trucks that respond to all tech support issues. I know them all by first name.

They have a higher tier of support they can escalate you to... you just may have to push for it. Sounds like you are dealing just with your local office, which may just be tier1 or 2 levels. They are basically just going to work in the capacity of making sure your local lines are good, and that you are connecting to the next circuit up the chain. Their T3 or higher is likely in a remote location. For the longest time, I had to go to people in either Columbia or Myrtle Beach (up to an hours drive), and that was only during "normal" hours. Sometimes I had to go as far away as Herndon, VA or Milwaukee, WI for support with issues on my line here in South Carolina. If you are using Qwest or CenturyLink out of the Boise area, you could ultimately have your issue investigated by someone in Colorado or Louisiana... it all depends on where they send your support request.

It's a multi-tiered structure by design, just like the networking stack and the internet--it pretty much has to be to function properly.

Simply put, your local guys are not the end of the support chain on this issue... unless you settle for that.