I never have this problem before 2.1 patch. Now I am haunted by constant lag and rubber-banding plus the 90000 errors that follow right after. I can't believe Square Enix have not acknowledge this problem yet. My ISP came out to check my lines and everything was working as intended so it's not my end.
Is Square Enix planning to just stay quiet and let the problem fix itself??? If you experience the same issue please reply here. Hopefully we can get SE attention and they will fix the problem for us.
"Game Lag is a Crime"
Please reply with the template below so we can let our voices be heard!
ISP: Charter Communication
Speed: 30 down/5 up
Location: California
Server: Behemoth
Server Group: Primal
Connectivity Issue: Lag, Rubberband, 90000 Error, Freeze
FIX THE LAG and 90000 Errors petition!
Printable View
-
01-01-2014, 06:15 AMBroCodeFIX THE LAG and 90000 Errors petition!
-
01-01-2014, 07:34 AMKindread
Very similar issue here, the game seems to update every half a second.
This causes everyone and everything (players and monsters alike) to stop for .2 to .5 seconds, then either speed up, or teleport to their "actual" location.
This also seems to effect all the skills in the game. When I press a button to activate a skill it takes too long to actually activate! It is beyond frustrating when running dungeons, and AoEs are nearly impossible to get out of unless it is an extremely slow cast speed.
This is NOT an occasional happening, this is constant! FFXIV is a VERY enjoyable game, but with this level of constant latency lag I will not be subscribing when my free trial runs out next week. -
01-01-2014, 11:24 AMmisterdapi
Cool empty threats. Call your ISP and get transferred to tier 3 support and get an email address to send trace routes to. Cool your modem works. Those guys have no control over what is happening 2 states over. Square doesn't control the internet either. If you subscribe with TWC their tier 3 support created a public email address asking for trace routes.
There is a thread sticked to the top already asking you to fill out this information. You are not a special snow flake. -
01-01-2014, 12:53 PMGuts_Taichou
I was having some insane lag spikes (like 10 second spike every couple of minutes) on my NA Server (Balmung), so i decided to transfer over to a japanese server. On the japanese sever I get no lag at all and everything is extremely smooth, it's very odd. Unfortunately noone will let me in their groups because I don't speak japanese, but still it seems to only be a problem with the NA servers
-
01-01-2014, 04:16 PMKindreadWow seriously?Quote:
Originally Posted by misterdapi
We are here for help, I am not a technically savvy person and would love to be able to play the game if I could. Your negativity is not helping the situation.
While reading through your haze of insults, I want to thank you for at least letting me know about the sticky thread. Unfortunately my ISP only has a single tier of support, and it is rather poor. -
01-01-2014, 07:33 PMBroCodeRegardless what you say...I don't see why I have to go through all that trouble when it was working fine pre 2.1Quote:
Originally Posted by misterdapi
Do you go buy a car and manually push it to get somewhere? Same thing, I didn't buy a game just to troubleshoot it myself so that I can play without lag/disconnection.
And the fact that I contacted my ISP first before posting here already show that I did my part. If you don't have anything useful to say don't say it at all. Instead you wasted a post space with your nonsense about trace routes and tier 3 support when I'm not too tech savvy. I just want to be able to tank for my coil group without disconnecting mid-fight. Is that too much to ask for? -
01-02-2014, 11:51 AMsquidbunny
This issue is not limited to Time Warner Cable, and in fact the OP lists his ISP (Charter). This is also not limited to SoCal. I'm in Wisconsin (though also using Charter) and have had latency issues from launch, but they seem to have gotten even worse with the 2.1 patch. I cannot clear Titan HM for a relic because I don't see Landslides before I'm being slid off the map and I've basically quit any kind of lvl 50 progression. This is probably my last month paying for the game.
To reiterate for the billionth time: my ISP is fine. There is no throttling apparent on actual torrenting on my end, and every other application is taking advantage of my consistent 25ish Mbps down. My traceroutes show ridiculously long hops well beyond my ISP:
ISP: Charter CommunicationQuote:
Tracing route to 199.91.189.28 over a maximum of 30 hops
1 6 ms 1 ms 1 ms 192.168.1.1
2 14 ms 25 ms 9 ms 10.130.64.1
3 22 ms 12 ms 10 ms dtr03jnvlwi-tge-0-6-0-6.jnvl.wi.charter.com [96.34.16.130]
4 27 ms 23 ms 25 ms crr01ftbgwi-tge-0-4-0-3.ftbg.wi.charter.com [96.34.16.177]
5 27 ms 23 ms 19 ms crr01euclwi-bue-1.eucl.wi.charter.com [96.34.16.77]
6 25 ms 23 ms 23 ms bbr01euclwi-bue-4.eucl.wi.charter.com [96.34.2.4]
7 39 ms 25 ms 31 ms 4.34.69.9
8 53 ms 48 ms 64 ms vl-3502-ve-116.ebr1.Chicago2.Level3.net [4.69.158.6]
9 52 ms 45 ms 48 ms ae-6-6.ebr1.Chicago1.Level3.net [4.69.140.189]
10 * 349 ms 110 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
11 46 ms 49 ms 55 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
12 * * * Request timed out.
13 90 ms 46 ms 47 ms 192.34.76.2
14 56 ms 47 ms 53 ms 199.91.189.234
15 50 ms 47 ms 47 ms 199.91.189.28
Trace complete.
Speed: 30 down/5 up
Location: Wisconsin
Server: Sargatanas
Connectivity Issue: Lag, Rubberbanding, Freezing, Latency spikes up to 5000ms (yes, 5 secs) -
01-03-2014, 09:58 AMRaistThough the attitude may not have been the greatest, the advice is sound. This is an issue with your ISP and their partners (will be illustrated in a quote further down), and so that is who you need to be working with to address the issue, and not the players (in case you missed the sticky welcoming you to the forums, this is user to user support and not for requesting assistance from SE).Quote:
Originally Posted by Kindread
And your ISP does not have just one layer of support... if you've called the help desk and a technician came on site, you just went through a couple layers of support getting that accomplished. The point is you need to ask specifically to be escalated to Tier3 support--those are the guys that can actually look at the larger sections of the network--the tier1 and tier2 people are basically just there to handle the simple problems like downed modems, cabling issues, etc. that affect localized connectivity issues. Tier3 and above deals with more advanced issues like routing and such, which is where the bulk of our problems reside.
Your issues are not resulting specifically from the patch (at least not the main issue, there may be some issues contributing to it, but it's not the core issue at fault here). It's more a side effect of the patch, and various other things that are resulting in increased congestion along your route. It's a combination of various things that have resulted in increased internet traffic across specific corridors--more players in the game, more people cutting off cable/sattelite TV or for whatever reason are streaming content more and more these days, all the buzz about the healthcare.gov site, etc.---it all adds up to heavier traffic on the routers, resulting in systemic failures trying to manage the congestion. You see this all the time with things like traffic jams and Black Friday sales--it's a very similar situation here. More people, more traffic through a fixed pipeline, and things slow down... sometimes to the point of a screeching halt.Quote:
Originally Posted by BroCode
As for the car analogy... when you have to take that car in to a service center to troubleshoot a problem, do you not get asked very specific questions to try to gather a clear picture of what is going on with your car? Do you not at the very least expect to have to give some sort of description of what you've noticed about your problem? Same concept applies here. Try just dropping the car off, say "It's broke, fix it." and walk away with out saying a word, or leaving a phone number for them to contact you---just get in your friends car and ride off. That's kind of what you've done by just calling Tier 1, and having them check your local connectivity. Like an issue with your ignition or emissions systems in your car, this problem goes well beyond the scope of your local Jiffy Lube that can offer quick fixes for things like a burned out bulb/fuse or changing a filter. It requires a more advanced approach with specific tools to provide specific information. They will need at the VERY least the proper IP address that is having connection problems. Providing a tracert/pathping result will help them even further, as it provides a much more detailed account of just what may be happening--and where to look for the symptoms/problems you are concerned about.
And no, Misterdapi was NOT telling you to diagnose the problem yourself. What was suggested was for you to generate a simple report that will help them diagnose your specific problems. It's really not that much more than what nearly every support ticket system will ask of you when you generate a ticket. Ever use that handy tool SE provided to generate your system info to submit a ticket through their support portal? It's the same concept, and nearly just as simple. The process has been outlined dozens of times on these forums since it was fingered as a routing issue back during beta4 (if not sooner, wasn't around for the earlier betas). Some have been very simple outlines, some have been very detailed. Here is one such post I made a while back that references a simplified post, followed by some additional details on how you can go about doing it:
http://forum.square-enix.com/ffxiv/t...=1#post1644618
Misterdapi and several others (myself included) have started their own threads to try to get more people to do this, but way too many simply won't even bother...but they will sure enough create multiple threads daily to complain about the issue, even though there is often a couple threads about it already on the first page or two. So, I can understand why that post came off a bit on the rough side... it gets exhausting rehashing this over and over again.Quote:
A lot of how-to's have been posted over the months. Here's one post with a simplifed version:
Basically, you need to be able to alt-tab out of XIV and get to your desktop to open a command prompt (DOS Window) by running CMD from your search or run box. From there you run the commands by simply typing them and pressing the <ENTER> key. You can copy all the text from that window into the clipboard for pasting into notepad, email, forums, etc. by right-clicking in the DOS Window and clicking "Select All" to highlight all the text in that window and then pressing the <ENTER> key. Then you can paste it all into something else with the usual paste operations (CTRL-V or right-click and select "Paste", etc.). If you want to clear the screen to get a fresh window again, you can just type the command "CLS" and press <ENTER>. This may help avoid excessive editing. When done, just type the word "exit" and press the <ENTER> key to close the window.Quote:
Quote Originally Posted by ninesunz View Post
system specs?
while logged into the game:
start > run > cmd
type "netstat"
look for listing that starts with 199.
in the same cmd window type
tracert 199.xxx.xxx.xxx
do this 4 or 5 times
post results
this will show if anywhere in your connection there is a provider having an issue keeping your data alive
Tracert is a very simple report, but if you are having a lot of trouble it should capture either a * event (time out) at a hop, or at least show very erratic variances in the 3 ping times taken at a hop. A slight variance is normal (like 85-100 range), but if you see a really big jump (like a 50% increase or more on the same hop) then there may be something they need to look at even though it didn't time out with a * on that hop. If you want to provide an even more detailed report, you can use Pathping in place of tracert (pathping 199.91.189.74). This will run 100 pings per hop instead of just 3 in tracert, and also tracks additional data on the quality of traffic when forwarding between hops as well. It can be a bit impractical for just taking a quick look because it takes a while to run (up to 10 minutes depending on the system and the route), but can be very helpful to a technician if they actually commit to opening an investigation.
Quote:
Originally Posted by squidbunny
And, finally... we come to a post that clearly shows where the problem lies, and ultimately hints at the proper course of action.
Note where things break down in the route---switching to Charter's routing partner (Level3) to get across the US and Canada and then finally handed off to SE's ISP (Ormuco):
Notice two things happening there--ping responses ramp up suddenly, there is a timeout. Hop 10 did not respond in time. This ultimately results in a retransmit and potentially a forced downward throttle in transfer speed if the quality is bad enough. This is something that is built into the protocol itself, and NOT just something that an ISP enforces (although, they do have rules that throttle to try to reduce congestion). So, even though they may not be doing P2P throttling---your connection is indeed getting throttled---it's just not because it is detecting P2P traffic, but because congestion has been detected. So, for the helpdesk to tell you that you are not getting throttled is grossly incorrect--it happens 24/7 on every connection, even Dial-Up... it's built into the transfer protocols, so it happens even if they don't set a specific routing rule to throttle specific types of traffic.Code:7 39 ms 25 ms 31 ms 4.34.69.9
8 53 ms 48 ms 64 ms vl-3502-ve-116.ebr1.Chicago2.Level3.net [4.69.158.6]
9 52 ms 45 ms 48 ms ae-6-6.ebr1.Chicago1.Level3.net [4.69.140.189]
10 * 349 ms 110 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
11 46 ms 49 ms 55 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
Those segments are not Charters.. but they are managed by their routing partner, Level3. Your ISP has agreements with third party ISP's to carry your data to Ormuco--NOT Square-Enix, but YOUR ISP has these agreements. Level3 has a responsibility to Charter to provide an acceptable level of service to Charter's customers, which falls under the umbrella of Charter's responsibility to provide their customers with an acceptable level of service.
Because of this structure, your ISP has the most clout with getting this issue addressed. So... you need to lean on them to lean on their routing partners (Level3, TATA, Cogentco, i-web, or whoever their routing partner is that may be showing signs of congestive failure along their segments). Pure and simple. This does NOT fall under the responsibility of SE to address--but rather your ISP. SE can come into play here if the problem is with Ormuco's segments because that is their ISP, or if there is a problem with the game's packet headers (which was discovered a while back by O2, but cannot verify if SE changed the headers to correct that issue yet). But what we are seeing here is an issue with routing and flow control that needs to be addressed by those who manage the troubled segments (typically your ISP's routing partners).
The BEST way to demonstrate who is dropping the ball and ultimately responsible for addressing such issues is to provide a simple tracert or pathping report. At the VERY least, you should take the extra 30 seconds (or less) it takes to identify the IP's your client is using so you can provide that to your ISP and demand that it be escalated to Tier3 support. They can terminal into your modem (or at least the nearest hop) to generate their own reports--if they have the proper IP address to diagnose. But... providing your own report can not only provide proof of a routing issue, but can also provide incentive for them to escalate it to a higher tier if necessary for further investigation. That is why it is so frequently suggested you submit a tracert---it is a VERY valuable report to have on hand to get someone to look into just where the problems actually are instead of just stamping your feet and screaming at the players (again, this is a user to user forum) about how SE needs to fix something they may very well be powerless to address.
Edit: Oops.. set me off into Dennis Miller mode again. Sorry about that. /rant -
01-03-2014, 12:40 PMRavnis
Honestly, I have had zero lag until patch 2.1 and their latency fix. prior to that I made 3 hops to their server, 3 , with a total time of 15ms. Now after the patch I make 15 hops and time out or throttle in between. Reality is they changed it to allow people who were getting bad transmissions to have an easier time and rolled the others over to the bad.
I live in NJ, on FioS, 50down 35up. Ping any server on any game in at most 50ms. I ping Korean servers in 73ms. Montreal, which is maybe 5 hours away is 315ms now.
I picked up a few latency fix programs and now it seems my BEST route is to Los Angeles and then over to fix my packet movement. It's obviously not their fault that suddenly ping routes completely changed for most players after a patch to fix latency.
Reality is the change has caused serious problems for quite a number of players and needs to be addressed. Whether you see it as our ISP or theirs; something is breaking down our packet routes AFTER a patch to fix these issues.
If there is something that changed it it's pretty obvious when all the problems started simultaneously after one change. I understand how the internet works, but there is something more than a little suspicious about these events, when nothing has changed for any other game I have played. -
01-03-2014, 01:24 PMRaistThat's most likely because either your ISP or their partner is routing you differently. You may have changed routing partners, or you may be coming into a different entry point or something else that is causing you to route differently along either your ISP's segments or their partner's (or maybe even both).Quote:
Originally Posted by Ravnis
Before my last routing change, I was using Cogent to get into Canada and it was 16 hops. They changed me to Level3 back around the middle of the month, and since then I have been hitting 19 hops. All those changes came BEFORE the Ormuco segments. Those hops have remained the same, at 3 hops once I got there: 192.34.76.10, then 192.91.189.242, then the final IP for the game server that I was using at the time (.74, .31, and lately .25).
The only thing SE has been changing is that final IP--everything else that has changed has been either in my TWC/RR segments, or their routing partner. This has been changing every 2-3 weeks since near the end of September when I first fingered Cogentco as the culprit and contacted my ISP's Network Operations Center with tracerts to get them to investigate it. Since then, TWC has been bouncing me all over different corridors from the DC/VA area, to the Great Lakes, to LA, up through Seatle, down to Atlanta---been all over the map with my routing, and used TATA, Cogent, and now Level3. All this time, it's been consistent at the Ormuco/SE end, and their latency has been pretty consistent at their hops (within about 20-30 ms variance). But the latency along the way has been gradually improving. My worst hops hover around the 130ms during primetime now--it was hitting the 300 mark frequently when we first started. Now, if we could just shave off some more of the packet loss it would be good to go... still likes to flirt with the 20% line periodically. A lot better than the 40+ we were tracking, but still a lot of room for improvement.
The point is... all the changes in routing aren't coming from SE, per se. SE may have changed your endpoint IP (mine changed from .31 to .25), and that may have resulted in some changes in your route--but SE didn't make that change. The ISP's that are carrying your data have negotiated those route changes based on specific traffic characteristics and what they've determined is best for getting you from point A to point B. -
01-03-2014, 06:00 PMCervani
This is only happening to me when I use my hotspot at work to connect to the game. I never drop connection when I'm at home. It's odd too, because I get error 90000 exactly once an hour and once it happens once it will happen at the exact same time every hour. So if it happened at 5:10, I will guaranteed be kicked off the game at 6:10, 7:10 etc etc. I did run a tracert, and it looks exactly like squids:
Tracing route to 199.91.189.28 over a maximum of 30 hops
1 * * * Request timed out.
2 60 ms 58 ms 59 ms 224.sub-66-174-162.myvzw.com [66.174.162.224]
3 44 ms 68 ms 64 ms 17.sub-66-174-63.myvzw.com [66.174.63.17]
4 63 ms 65 ms 46 ms 194.sub-69-83-70.myvzw.com [69.83.70.194]
5 65 ms 64 ms 68 ms 113.sub-69-83-83.myvzw.com [69.83.83.113]
6 65 ms 66 ms 66 ms 241.sub-69-83-70.myvzw.com [69.83.70.241]
7 71 ms 65 ms 65 ms 210.sub-69-83-70.myvzw.com [69.83.70.210]
8 72 ms 71 ms 72 ms 0.sub-69-83-83.myvzw.com [69.83.83.0]
9 75 ms 74 ms 67 ms 0.sub-69-83-83.myvzw.com [69.83.83.0]
10 67 ms 64 ms 67 ms 209.sub-69-83-66.myvzw.com [69.83.66.209]
11 72 ms 70 ms 75 ms ae57.bar2.Philadelphia1.Level3.net [4.30.46.5]
12 88 ms 85 ms 85 ms ae-6-6.ebr1.NewYork1.Level3.net [4.69.133.170]
13 84 ms 79 ms 78 ms ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]
14 85 ms 84 ms 79 ms ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]
15 84 ms 79 ms 80 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
16 85 ms 86 ms 85 ms 192.34.76.10
17 85 ms 84 ms 81 ms 199.91.189.242
18 92 ms 91 ms 86 ms 199.91.189.28
Edit: It just happened again, this time while playing League of Legends so I think I can pretty much confirm it's my hotspot doing it. -
01-03-2014, 09:09 PMZohnoReecho
your hotspot may have a DHCP with a 1h lease time
-
01-03-2014, 10:17 PMJadi
Hi all. I'm a super genius network engineer with 15 years experience working for some of the largest tech companies in the business. /wave Hi.
Your pings mean nothing, forget about them. Don't post them. They just don't give any meaningful info. Why? Because the traffic is ICMP. Why more? Because ICMP isn't UDP and what is going on is ISP's are incorrectly applying a low priority protocol scheduling to the UDP game traffic, NOT ICMP traffic. They do this because they mistake the traffic as P2P traffic, specifically torrent traffic.. and yes, it's similar so it's easy to see how this occurred. ICMP usually has high scheduling.
SE's ability to control the protocol scheduling priority of your ISP is very limited. They basically contact them and ask for a whitelisting. This is difficult and tedious for them. I mean you've called your ISP right? Their customer service is shit and you actually pay them, SE doesn't.
However there is something else they can do.. they could code the game to use a VPN wrapper and tunnel it into their own datacenter. This would change the type of traffic and give it a higher scheduling priority by ISP's. Just a thought.. seems easier than attempting to contacting every ISP on the planet and beg them to fix it.
Also this doesn't mean that other problems don't exist as well, naturally they do. -
01-03-2014, 10:27 PMCervaniI wonder how I could fix that then, if that is indeed the case. I use the Straight Talk Unimax u240c, and sadly it's admin panel is very lacking in regards to adaptability and options.Quote:
Originally Posted by ZohnoReecho
-
01-03-2014, 10:44 PMDalavonSo basically... We can't do anything ;)Quote:
Originally Posted by Jadi
-
01-03-2014, 10:47 PMJadiyeah.. not really. There are some programs out there but they cost. that or you can get yourself a vpn.. it seems to work.Quote:
Originally Posted by Dalavon
-
01-04-2014, 07:51 AMKindreadNot to split hairs, but my ISP really does only have 1 level of support, living in the mountains in Idaho there are 3 guys that answer phones and drive the trucks that respond to all tech support issues. I know them all by first name.Quote:
Originally Posted by Raist
-
01-04-2014, 07:56 AMRaistXIV actually relies on TCP... which is why we see so much rubber-banding and such. If it were relying on UDP, we'd see more FFXI like behavior. If you'd run a netstat or look at it in resmon, you'll see that all the connections are TCP and not UDP. So, there is the built in potential for throttling within the protocol itself that can be triggered by congestion. The P2P vs game flag in the headers was brought to SE's attention in beta, and hashed out for quite a while with some UK ISP's. The thread about gathering data has hopefully helped to resolve it further with more ISP's--but that is old news and can be reasonably assumed those throttling rules are being addressed on an ongoing basis if not already handled at this point. Hard to say for certain, since they have made no official statement--but it's an old problem that has been addressed between SE and some ISP's already, so there is reason to consider taking that out of the discussion I guess. That leaves us with addressing issues native to the protocol and routing analytics that have been identified already as a problem, and testing has been done to support that theory (VPN/proxy, as well as some professionals that have presented their own analysis of what they've picked up on with sniffers and looking glasses the past 3 or 4 months). The trick now is in getting the ISP's to own up to these as valid concerns worth investigating, which is why there is this ongoing push to get people to file proper support tickets with their ISP's.Quote:
Originally Posted by Jadi
And we aren't providing simple pings... we are using tracert and pathping reports. This DOES have it's uses. It shows the actual route we are taking, as well as gives hints at where congestive failure may be on the horizon (if not actually happening). Even though it is using ICMP, it still works to show signs of congestion issues. If you will actually look at what we've been pointing out, there are recurring subnets operated by a select group of 3rd party ISP's (and often isolated to a specific region) that shows very erratic ping responses. It frequently gets to the point that you will see intermittent timeouts at those same points in the route. And that's not a *** event in a single report that may indicate a rule to flat out not reply to ICMP--but where a hop actually responds and then drops it in the same run. We've had pathpings showing varying degrees of rejection at these points--7% one time, 66% another, and so and so forth. This is a pretty clear indicator something is going on that needs to be investigated with that route.
If you combine enough of this data, you will begin to see a pattern. For instance, the PAIX and SIX exchanges have been hosed for some time now (and it's not uncommon by the way.. this is a returning problem that goes back to the early days of WoW, and even the original Diablo release). Flip through the pages and review the threads complaining about issues in the California corridor. This isn't a coincidence---it's a pretty clear indicator that some exchanges may be getting overloaded during primetime. This <should> give the ISP's reason to review their analytics to see if they need to switch to different peering/transit/exchange partners to try to relieve some of the congestion.
The trick is providing them with sufficient data to spark an interest into investigating along such lines. What if they got flooded with pathping and tracert reports from a few thousand subscribers... maybe that would get their attention? And it's not unreasonable to say thousands either. SE has already published a stat of hitting 5k+ simultaneous connections per world. I think we could gain some traction with the ISP's involved if enough people would get proactive with the reporting.
Remember, it's a market out there where your ISP purchases access to these routes from someone else. Just like you have the means to change who you purchase many services from, so does your ISP have the means to purchase access to alternate routes to get us to Ormuco's segments. And yes, that is a big part of the solution. Just look at how many people have resolved their problems with VPN or Proxy services to change how they are being routed.
So, no... we are not helpless to affect change on this matter. For a long term, more permanent fix, it just takes a few moments of everybody's time to generate a proper support ticket. I sent a total of 3 emails, but only really needed to send one. I saw changes after the first email was sent, but I voluntarily sent 2 followups with tracerts because twice they made a change that made things worse than the previous route change. For a short term fix, there are the VPN/Proxy services--some of which are actually free (maybe not as robust as the premium ones, but they can still improve things). -
01-04-2014, 09:22 AMRaineMagusIt's true that many ISP's categorize UDP traffic as "torrent" with a blanket catch-all rule .. although this game uses TCP and not UDP.Quote:
Originally Posted by Jadi
(still this doesn't mean TCP catch-all rules don't exist, or that layer 7 inspection can't mistake traits of one protocol for another .. pretty common)
You're absolutely right in that ISP's have been to known to do stupid things like the above -- *catch-all rules* -- even on something as simple as uncommon 'ports' for traffic (eg, if it's not port 22, 80, 443, 1723, 500, 4500 .. the common messenger ports [MSN, ICQ, AIM], etc etc .. consider it low priority data).. and that many gateways also shape ICMP. Yet, I wouldn't go so far as to call ICMP 'worthless' as a tool for diagnostics.
For instance: If someone sees high latency (via ICMP or UDP ping) to a destination, and that high latency spike, or loss, corresponds to a behavior that they're also consistently seeing in a game (or internet behavior in general from their end) .. it really doesn't matter the 'cause', as those pings are still helpful in determining at which gateway the problem originates [even if the absolute cause is not revealed]. A traceroute isn't useless as long as you interpret it with a grain of salt, and understand that the problems shown may or may not actually be issues at all, or effect other protocols.
I'll elaborate (especially for everyone else who's submitting traces). Below is an example ran quickly from one of my servers in LA, to Japan. While this may look like an imperfect route ... there's absolutely nothing wrong with it / no packet loss to the destination whatsoever.
traceroute to 124.150.157.52 (124.150.157.52), 30 hops max, 60 byte packets
1 * * * << isn't a problem .. the hop is simply not responding to messages addressed to itself
2 * * *
3 * * *
4 gi8-0-0.cr1.nrt1.asianetcom.net (202.147.0.121) 135.188 ms 115.191 ms 107.194 ms
5 gi1-0-0.gw1.nrt5.asianetcom.net (202.147.0.178) 107.570 ms 107.572 ms 119.566 ms
6 squareco.asianetcom.net (203.192.149.210) 98.913 ms 98.935 ms 98.809 ms
7 61.195.56.129 (61.195.56.129) 140.494 ms * * << this hop is shaping pings addressed "to itself"
8 219.117.144.66 (219.117.144.66) 107.804 ms 107.758 ms 107.728 ms << see above and final comment
9 * * * << same as hop #1
10 * * *
11 * * *
12 * * *
13 124.150.157.52 (124.150.157.52) 99.209 ms 99.103 ms 99.403 ms << 100% of the pings to here are 'clean' at ~99ms
PING 124.150.157.52 (124.150.157.52) 56(84) bytes of data.
64 bytes from 124.150.157.52: icmp_seq=1 ttl=244 time=99.1 ms
64 bytes from 124.150.157.52: icmp_seq=2 ttl=244 time=99.1 ms
64 bytes from 124.150.157.52: icmp_seq=3 ttl=244 time=99.2 ms
...
--- 124.150.157.52 ping statistics ---
2000 packets transmitted, 2000 received, 0% packet loss, time 2000019ms
rtt min/avg/max/mdev = 99.131/99.180/99.234/0.417 ms
Seeing as the pings to the destination are 'clean' over a long term, we can conclude that all prior drops seen in the traceroute (concurrently) have no bearing on the destination's responses. We could additionally run the same trace over and over repeatedly, or run a pathping, and we'd see 100% timeouts for TTL 1, 2, 3, 9, 10, 11, & 12.
To everyone else looking at the above and thinking "Why the heck would this be done in the first place?". The general line of thought behind it is that a gateway's 'primary' purpose is to forward data (*not to respond to data addressed to itself*). ICMP bombing is also a very common form of DoS attack, and the protection against it is usually rate limiting, prioritization / queuing, or just not responding at all.
Now to make up a bit of data to illustrate an obvious problem...
Let's say that the above tracert instead looked like the following, randomly, during periods of high delay that we're seeing ingame.
traceroute to 124.150.157.52 (124.150.157.52), 30 hops max, 60 byte packets
1 core-router (xxx.xxx.xxx.xxx) 0 0 0 (near zero latency)
2 xxx.coresite.com (xxx.xxx.xxx.xxx) 0 0 0 (near zero latency)
3 xxx.coresite.com (xxx.xxx.xxx.xxx) * * ms 400.xxx ms [connection to asianet ... in California]
4 gi8-0-0.cr1.nrt1.asianetcom.net (202.147.0.121) 357.188 ms * 387.194 ms
...
dst 124.150.157.52 (124.150.157.52) 400.209 ms * 390.403 ms
If all hops after a certain point are showing the problem at some time of the day, and we're seeing the above where we're getting several hundred ms added (& loss) to each ping (*before the international backbone*) ... then we could conclude that gateway is probably overloaded, and this is the cause of the lag we're feeling ingame right-now.
Useless? I'd think not. [it shows the problem is likely in California, and not Japan]
Will someone care (the owner of that gateway) and fix / do something about the issue? That's a whole other can of worms...
Quote:
Originally Posted by Jadi
That all assumes that layer 7 inspection is even the root of the problem to begin with. If it was, concealing XIV's traffic through a tunnel would indeed solve the problem ... It'd be unwise to embed a protocol such as SSH into XIV's netcode simply on a whim though. Compression and encryption of all traffic isn't exactly trivial in terms of server overhead, nor is the price of double encapsulation.
-Rather than adopting a "tunneling protocol", perhaps running all of the game's traffic through a simple shift-cipher would be sufficient to throw off whatever rule(s) are mistakenly identifying it. Unless of course there's a catch-all for everything other than known protocols by something like port. In that case, using a port such as 80 or 443 would be an immediate fix.
EDIT: Should go into a bit of "why" this usually isn't done. IP addresses are not an unlimited resource, and thousands of servers can technically be operated on the same address. To use only "common" protocol ports (eg, HTTP, HTTPS, SSH, FTP, etc) puts a huge constraint on how many servers can be run on a single address.
As IPv4 addresses are a limited resource.. Whenever requesting an address block, usually a company has to provide some 'rationale' (why they "need" it) in order to receive them. -
01-04-2014, 01:25 PMRaistThey have a higher tier of support they can escalate you to... you just may have to push for it. Sounds like you are dealing just with your local office, which may just be tier1 or 2 levels. They are basically just going to work in the capacity of making sure your local lines are good, and that you are connecting to the next circuit up the chain. Their T3 or higher is likely in a remote location. For the longest time, I had to go to people in either Columbia or Myrtle Beach (up to an hours drive), and that was only during "normal" hours. Sometimes I had to go as far away as Herndon, VA or Milwaukee, WI for support with issues on my line here in South Carolina. If you are using Qwest or CenturyLink out of the Boise area, you could ultimately have your issue investigated by someone in Colorado or Louisiana... it all depends on where they send your support request.Quote:
Originally Posted by Kindread
It's a multi-tiered structure by design, just like the networking stack and the internet--it pretty much has to be to function properly.
Simply put, your local guys are not the end of the support chain on this issue... unless you settle for that. -
01-04-2014, 04:27 PMJadi
Here's the thing. I don't know if it's layer 7 or not. You can't assume such things but I would take a guess that is NOT the case because of the intensity required for application filtering. More likely is just QOS by protocol and destination port set in ranges. Now the highest priority would be TCP ACK flags and destination ports of 53 and 80. We can agree on that so we know we are not top priority.
I haven't done a protocol analysis on this but on cursory inspection it looks like you are correct its TCP, that surprises me but it's still upper port ranges, meaning it is a potential candidate for a policy to apply to it. (I also am curious if there is a control and data element to it meaning if thats true it would be both.. unfortunately I don't know enough about it to tell you exactly how it works it takes time to investigate this stuff and it's also not really important to this conversation.)
Also I don't see how the protocol would cause "rubber-banding" .. thats application side, the server moves your position to where it believes you are. The protocol facalitating that message is indifferent to the actual message.Quote:
here are my states currently, this is at idle keep in mind.
pfctl -ss | grep 199.91.189.26
rl0 tcp 199.91.189.26:55028 <- nat.ip:59145 ESTABLISHED:ESTABLISHED
fxp0 tcp nat.ip:59145 -> fw.ip:11066 -> 199.91.189.26:55028 ESTABLISHED:ESTABLISHED
rl0 tcp 199.91.189.26:55028 <- nat.ip:59146 ESTABLISHED:ESTABLISHED
fxp0 tcp nat.ip:59146 -> fw.ip:33285 -> 199.91.189.26:55028 ESTABLISHED:ESTABLISHED
IT is weird ya know.. there are some people out there that always want to appear to be experts in the field.. so the'll try to out-do one another. A good tip for laymen to know if someone knows what they are doing.. if someone makes something sound easy they probably have a firm understanding of things.. if they make something sound complex they probably view it as complex to them. When people talk about the availability of IPv4 address.. I'm.. kinda.. speechless for a response as I fail to see what that has to do with the conversation at hand. Regardless of whatever the minute details of this issue are, the truth of the matter is that A) It's not really in SE's hands. B) We really can't have much of an effect on the policies of these ISP's. We don't have a large enough user base and those policies are in place because corporate bastards are cheep. -
01-04-2014, 04:47 PMJadi
I had to look up what pathping actually was. lol I never heard of it before.. its mtr. (saying traceRT was a clue I guess) I didn't know windows machienes did networking. ;) heh
-
01-04-2014, 06:49 PMRaineMagusI think the main incentive, and why virtually all MMO developers opt for TCP over using UDP is purely to avoid having to deal with shotty routers. Even if UDP is a better choice, and they're aware that it's a better choice for the gameplay style.Quote:
I haven't done a protocol analysis on this but on cursory inspection it looks like you are correct its TCP, that surprises me
TCP has very clear defined behavior on all routers in that it's session oriented. While your typical consumer will make assumptions with UDP and somewhat simulate session oriented behavior ... all bets are pretty much off as I've learned first hand over the years.
It's a pretty daunting prospect to have to step users through port triggering, forwarding, shutting off security settings, etc. Whereas with TCP, it just-works pretty much always.
If IPv6 is ever fully adopted, god knows how many years from now, then I'd assume this will be much less an issue and more developers will choose UDP.
TCP doesn't directly cause rubber-banding, yet exacerbates it by forcing ordered delivery of packets (at the app layer).Quote:
Also I don't see how the protocol would cause "rubber-banding" .. thats application side, the server moves your position to where it believes you are. The protocol facalitating that message is indifferent to the actual message.
-If one single packet of an update fails to reach the other end, then even if subsequent position updates were sent, the other end is going to have to wait after receiving them (for retransmission of the missed).
If there's no loss or congestion, then TCP is quite A-OK for gaming.
-It has very little to do with the conversation as the end of my response was purely thinking outloud of what SE could do about the problem.Quote:
When people talk about the availability of IPv4 address.. I'm.. kinda.. speechless for a response as I fail to see what that has to do with the conversation at hand.
Aka, taking a blind stab at fixing the issue without contacting any ISP by changing around port assignments (try to force elevated priority with problem ISP's). I can't comment on how many servers that SE has per world ... or of their physical distribution (as I have no clue, like anyone else who's not an SE employee). Yet, if each zone is listening on a different port (which they are) and you're forced to use only say .. one or two ports per address of those that you expect would have high-priority.. That'd be whole lot of IP's needed per world, not even considering the Duty Finder / instances.
Earth isn't going to run out of addresses any time soon (there's no IPv4 apocalypse -- although it'd be cool if there was.....). Though it's alot harder to get them assigned today than it was in the past. It's pretty tough to purchase a large address block without legit justification, which the above wouldn't be.
Of course if SE has a sufficiently large IP block laying around, and they have nothing better to do with it.. All the more power to them.
Pretty sure most people have never heard of pathping. I didn't know of it until close to the end of my using Windows 7, when randomly in the last MMO I was playing, support asked me to run pathpings instead of prolonged ping plots (of each hop).Quote:
I had to look up what pathping actually was. lol I never heard of it before.. its mtr. (saying traceRT was a clue I guess) I didn't know windows machienes did networking. heh
Windows Networking is actually pretty good, yet there's still one thing that Linux / BSD can do that Windows' TCP stack cannot. Something very important for servers... On Windows there's no way (even in 8.1 server), to direct new-sessions programatically out a specific interface (only via the routing table).
Actually I think we're all just really super-bored and jump at the chance to talk about things in our field. Can't speak for others, yet that surely applies to me on forums.Quote:
IT is weird ya know.. there are some people out there that always want to appear to be experts in the field.. so the'll try to out-do one another.
-
01-20-2014, 06:45 AMFurlamseere
It seems that there have not been any further posts about this concern being an ongoing issue. Because of this, we are considering it to be resolved. If this does continue to be a concern, please create another thread to further discuss resolutions or contact the Support Center at support.na.square-enix.com for assistance.
- Furlamseere