Details about Server issues

[Raist]

In addition to Enepttastic's post, have to understand that this (as usual) loosk like a routing problem occurring upstream from our localized networks. This is something managed by your ISP, and is affected by your point of origin and the requested destination. Rebooting your network can cause new leases to be assigned, effectively sometimes changing something in your auto-assigned IP config or otherwise correcting an issue with your geolocation---which has been known to resolve issues in the past. It is also sometimes required by your ISP in order for you to download an updated config file from the ISP's TFTP server to address potential issues with your modem and/or their network.

Basically... the old Microsoft mantra of rebooting your devices can and does have it's place because it is a documented step in the troubleshooting process that has been shown to sometimes work and at times is actually necessary to address issues.

Back to the OP though, this is something that needs to be addressed by your ISP. you ned to get in touch with no less then their Tier3 people---not the tier1 and tier2's that you will get on the phone when you call in or that come out and check the cables and possibly swap out your modem. That's not to say that you don't have an issue with your network---you may find signs of issues because of out of spec Signal to Noise Ratio (SNR) or other values listed on your modem's signal pages that those guys will need to address. But you have specifically identified the very common issue we've all been dealing with for some time now---a routing issue with our ISP's routing partners. Our ISP's have the enforceable agreements with these third (and sometimes fourth) parties and not SE, so it actually falls to our ISP's to go lean on them to address their problems on our behalf.

And yes, such an issue can affect a LOT of players at the same time. When PSN goes AWOL like it has been doing more and more often since the launch of the PS4, it potentially affects EVERY console player of this game. OTher games may run just fine because they don't continually check in with PSN like XIV does. Likewise, comparing how LoL, GW2, or WoW is performing against XIV is not an accurate comparison. XIV is hosted in Canada for the NA/EU accounts--they aren't run on state-side servers in New York, LA, Chicago, or Seattle and so may not be affected by routing issues getting into and/or across Canada.

Important point of fact there... a very large part of the problems stem from getting into and/or across Canada. There are very limited entry points into Canada, as well as a limited number of trunks for getting across Canada. If AT&T, Verizon, Comcast, TWC, etc. all use the same 3 companies for this process and any number of those three get overloaded it can severely knacker up access for a LOT of people at once. This in fact is what is happening---Level3's lines are crapping out, and everyone using their routes through that same corridor are getting hosed.

[Edit:]
Just picked up signs of congestion within TWC's own network around Scarborough, Maine as well. Fluctuations from 20's up to around 400ms, with the occasional spikes up over 1 second. Things are just getting out of hand in general up near the border I guess... Northeast is showing it's usual signs for TWC, as well as the other usual areas of Southern California, Texas, Georgia, the Carolinas. Nothing new here... just more of the same oversold markets and under maintained markets we've dealt with for years:



You can find more maps like this for other services (including PSN and XBL, among others) and a few other ISP's at the downdetector's company pages:
http://downdetector.com/companies

[Mykll]

Yeah. You're right. Good call. Guess we should just ignore the 7 threads from the last 3 hours about the instance servers being totally borked. Or the (multiple) server wide and multi-person connection issues. Glad we had your insight for this problem.

Important point of fact there... a very large part of the problems stem from getting into and/or across Canada. There are very limited entry points into Canada, as well as a limited number of trunks for getting across Canada. If AT&T, Verizon, Comcast, TWC, etc. all use the same 3 companies for this process and any number of those three get overloaded it can severely knacker up access for a LOT of people at once. This in fact is what is happening---Level3's lines are crapping out, and everyone using their routes through that same corridor are getting hosed.

SE game servers are now controlling internet traffic, before your connection even gets to their servers? Huh, didn't know that. Thanks Burrmanchu.

[Raist]

Edit:
Misread previous post, and responded in error. Amended the post slightly and left it up, as it was mostly to clarify the points about restrictive access to Canada, and how a seemingly minor hiccup along a route for one person can indicate a big problem for many in the scope of FFXIV.

There are a small handful of ISP's providing access to Canada, and they basically "lease" that access out to our ISP's so that we can connect to servers in Canada. This happens all over the place. If not, we would not be able to load webpages or send email to anyone not on our ISP's own network. You may not even be able to contact people with the same ISP if they were in a disparate region--you would only be able to connect to the people on the localized network.

What this means is that if Level3's routers are overloaded/stalled/taken offline, everyone who is stuck with them as their routing partner to the region managed by those routers---their access to that region becomes compromised. This is what has been happening for YEARS. It isn't a phenomenon isolated to just XIV.

Perhaps some visuals will help demonstrate the issue better:

Cogent Communications Network Map:


Take a very close look at how their trunks access Montreal from the US (can right-click the picture and "View Image", then you can zoom in and such, as it is a higher res picture than displayed in the forum).

They basically have 3 paths... let me restate that: THREE paths that feed into Canada for getting to Montreal: Two from the East, and one from the West. That means for the US, you are coming through the New York/Massachusetts, or Washington state corridors. Unfortunately, California is routinely congested on it's own, and then you can get bogged down again coming around the WA/BC corridor even further. It doesn't matter where you are coming from in the US if your ISP has partnered with Cogent to carry your data to Montreal--you basically have those 3 paths to take to get into Canada. If there are issues along one of those limited paths, everyone using that path gets screwed at once. SE has absolutely NO control over that---that is between Cogent Communication because it is their network hardware in play, and your ISP because THEY have chosen to route you across Cogent's hardware.

Now, here is a link to Level3's map, the ISP in question from the OP:
http://maps.level3.com/default/#.U5UPHig2lW8

Just take a moment to zoom in on Montreal and look at how things get routed there via Level3's network. Guess what? It's once again NEW YORK AND MASSACHUSETTS for the US entry points if you are on their network. They don't show an entry point for the Vancouver area. It's only via the North East.

And now, here's TATA's map:


That image is hard to discern, but they basically have the same access as Cogent--one corridor for the west, two for the east. There is a link to a PDF you can load if you want to see things more clearly.

These are three of the big players up there, and who almost all of us ultimately wind up dealing with in order to get to Ormuco, who is SE's ISP in Montreal. Now, compound this with how everyone is partnered with everyone else to get us over there. For example, i-web is partnered with all three of them:

http://iweb.com/reliable-hosting
(Network Partners logos, down near bottom of the page)

Likewise, our ISP's partner with multiple other routing partners to build routes to get us to where ever we want to go on the internet. You may find you hop off and on to multiple ISP's segments to finally reach your final destination. These routes are purchased by your ISP in some fashion or another (often times it is a quid-pro-quo arrangement and not actually paid for, but an agreement IS made with your ISP for access to the route). So, when there is a problem along such routes... it falls to your ISP to investigate and resolve the issue on your behalf. Either they get their partner to fix the problem, or they switch us to an alternate route (sometimes this requires a reboot of our modem as they may need to change our modem's config). Square-Enix has no responsibility, not to mention little to no influence, to address such issues unless it is shown to be an issue with their ISP (Ormuco) or their own network.

[Burrmanchu]

SE game servers are now controlling internet traffic, before your connection even gets to their servers? Huh, didn't know that. Thanks Burrmanchu.

I suppose the 5 hour emergency maintenance tonight is to fix the internet traffic in Canada then? Let's wait and see if SE can or cannot fix our problem.... Servers or not, your original point was that SE couldn't do anything about it.

[Brine_Gildchaff]

I suppose the 5 hour emergency maintenance tonight is to fix the internet traffic in Canada then? Let's wait and see if SE can or cannot fix our problem.... Servers or not, your original point was that SE couldn't do anything about it.

Because it's totally impossible that the maintenance could be for anything besides the already-clearing lag problems, like the laundry list of known issues they might now have fixes ready to go for. I mean that would be just /absurd/, right?

[Burrmanchu]

Because it's totally impossible that the maintenance could be for anything besides the already-clearing lag problems, like the laundry list of known issues they might now have fixes ready to go for. I mean that would be just /absurd/, right?

Another excellent point. Why would I ever assume that a long emergency maintenance after 3 days of an unplayable game, would in any way be related to the game being unplayable? Silly me. I'm so /absurd/.

On another note, the game's been fine all morning. No d/c's yet. I have no clue what other "already-clearing" (wut?) lag problems you're talking about, and neither does anyone else I know. Maybe you should restart your modem.

edit: YEP! It fixed it! Good call.

[Raist]

Actually Burrmanchu, you have been the one launching the attacks... I've been more focused on trying to demonstrate the flaws with the logic behind them. Even after trying to address such inflammatory and off-topic posts, I still try to bring my post back to the actual topic brought forth in the OP. But it's hard sometimes to remain completely focused and civil when one is posting such insulting/inflammatory or otherwise detracting statements such as yours. If you are going to continue making such posts, you should be prepared for a little push back now and then.

The issue at hand is the routing issues we have been investigating, documenting, and getting our ISP's to address. The act of resetting the modem has been shown to be part of the standard troubleshooting process and at times even a critical part of the remedy to address ROUTING ISSUES THAT ARE DICTATED BY OUR ISP. SE has absolutely nothing to do with those policies, and little to no influence over what our ISP's routing partners are doing with their networks. Our ISP's are the ones with enforceable agreements to affect changes, changes that at times (again) require a modem reset.

No way around it... the process suggested was valid, relative, and depending on exactly what has been going on behind the scenes already may have actually resolved the issue for the OP, as there actually is a chance that it could have led him/her to switching to an alternate route assignment and bypassed the failing segment.

You have aggressively gone after people that gave valid suggestions on how to remedy the situation, and have attempted to take the thread off-topic. The issue at hand has nothing to do with SE's server stability issues---that is a completely separate issue. This thread has not been about server stability and connectivity, it has been about extreme latency that is interrupting communication in route---it is crashing before the server can even be reached. Even if after the maintenance and potentially reboot of their hardware and people are able to connect consistently again until they become overloaded once more, the type of latency issue brought forth by the OP will still be cropping up until it is properly addressed, and the advice put forth within this thread will still apply to addressing such issues.

Instead of reporting your actions (which I am beginnning to regret not doing), I chose to see this as a teachable moment and took the opportunity to try to demonstrate how and why this problem persists, how impactful the issue is, how one may go about addressing the issue properly, and why previous suggestions that had been challenged were correct. Perhaps I made an error in judgement in hoping to enlighten some forum members that might not have a good understanding of the situation and how to address it. The Dennis Miller bit was an attempt at injecting some levity into the aggressive tone that had sprung forth in the thread... a throw back to his early days of starting his witty diatribes with "I hate to get off on a rant here, but...". His criticisms always had a hint of truth buried in there, and if one paid close enough attention to them, some precious pearls of wisdom could be gathered.

Guess I just dated myself... oh well, it's a shame it has fallen on some deaf ears. But, hopefully it gets through to some, so it's worth the jabs I am now getting from you if it gets people more proactive in pushing their ISP's to get involved and improve things for more people.

Oh, and just an FYI... I in fact have recently rebooted my modem, got new channel alignments, and my local segments have been much more responsive since (thanks for asking). I will likely have to bounce it again this weekend though, and it's about time for them to open a new route as TATA has started to flake out on us now (will get to that in a second). It's been an ongoing process for about 2 months now while we've been trying to tackle a troublesome CMTS that at first was only bonding 4 channels (and poorly at that), eventually got up to 6, but is now finally running at full capacity with all 8 channels bonding. We finally tracked down the ingress that was knocking them offline, but they are concerned it may come back because of aging hardware and lines... TWC's infrastructure is in really bad shape down here. New channel lineups are on the way, and the hopes are that we may be able to get new assignments for network use and get away from the nearby transmissions at channels 31, 35, and 38 that have been giving us problems. Until then, we will have to continue to reboot our modems to lock on to cleaner primary channels whenever we see our headend's latency start spiking high--when things go south, it will start breaching a full second or more in delayed response at the start of our run, which just gets uglier as we go upstream towards Canada. Three guesses on the quickest way we resolve this issue...

A modem reset.

It should also be noted that latency has already begun to creep up on TATA's lines tonight (who my ISP has most recently paired me with after my last reset)---already picked up spikes over 700ms getting into Montreal via one of their routes. That's approaching a 3/4 second lag for a single transmission cycle between client and server (in case some aren't aware of the impact), and it isn't even prime time yet. Others may soon find similar issues with their routing as things get busier in the days/weeks to come. So, we may unfortunately find ourselves right back here again having the same debate, even after some may have gotten the impression that SE fixed all their latency woes with last night's maintenance.....

[Burrmanchu]

TOO MANY WORDS

I only read one sentence of that ridiculous wall of bs.. but once again, this issue had nothing to do with restarting modems. It's fantastic that the first thing inet providers tell you to do is just that, but exactly as I said, this wasn't related.. And if you think my comments are just so insanely offensive and "inflammatory", you're in for a HUGE surprise when you surf literally any website on the internet. Good job posting another great wall of text that has no point inside it whatsoever.

It's absolutely AMAZING that the server woes (i mean, CANADIAN INTERNET TRAFFIC ZOMG) were totally fixed after the maintenance.. Must be a complete coincidence.

Maintenance fixed the issue. Just like I said. As far as me harassing YOU, I originally replied to someone else, and you typed a 1500 character dissertation on how I need to think and behave... Move along please.

[Mykll]

FIX YOUR SHIT SE I HAVE COIL TONIGHT.

Let's wait and see if SE can or cannot fix our problem....

I agree, we should wait and not immediately assume it is a SE server problem. We do not know what the problem is or what the maintenance is for. I can guess and assume all I want but until I hear SE say it...

Servers or not, your original point was that SE couldn't do anything about it.

My original point was that if there was an internet interruption somewhere, from an ISP that SE does not own, then it is not their shit to fix. They can call and put pressure on the ISP or whoever owns the broken link or router, if that is what the problem is, but they themselves cannot fix it. Yeah I am frustrated. I have been getting a lot of DCs since Saturday night. But I also understand that I am not plugged in directly to their servers, so there could be something completely out of their control to fix. They have done some boneheaded moves, absolutely, but I am also not going to come here and rant and rave and cuss them out for every little hiccup. I hope despite their silence they know exactly what the problem is, either on their end, or on some ISP end, and that it gets fixed tonight. After all, they have done maintenance at the same time as the ISP they use when they (the ISP) needed to fix something.