Details about Server issues

[Raist]

In addition to Enepttastic's post, have to understand that this (as usual) loosk like a routing problem occurring upstream from our localized networks. This is something managed by your ISP, and is affected by your point of origin and the requested destination. Rebooting your network can cause new leases to be assigned, effectively sometimes changing something in your auto-assigned IP config or otherwise correcting an issue with your geolocation---which has been known to resolve issues in the past. It is also sometimes required by your ISP in order for you to download an updated config file from the ISP's TFTP server to address potential issues with your modem and/or their network.

Basically... the old Microsoft mantra of rebooting your devices can and does have it's place because it is a documented step in the troubleshooting process that has been shown to sometimes work and at times is actually necessary to address issues.

Back to the OP though, this is something that needs to be addressed by your ISP. you ned to get in touch with no less then their Tier3 people---not the tier1 and tier2's that you will get on the phone when you call in or that come out and check the cables and possibly swap out your modem. That's not to say that you don't have an issue with your network---you may find signs of issues because of out of spec Signal to Noise Ratio (SNR) or other values listed on your modem's signal pages that those guys will need to address. But you have specifically identified the very common issue we've all been dealing with for some time now---a routing issue with our ISP's routing partners. Our ISP's have the enforceable agreements with these third (and sometimes fourth) parties and not SE, so it actually falls to our ISP's to go lean on them to address their problems on our behalf.

And yes, such an issue can affect a LOT of players at the same time. When PSN goes AWOL like it has been doing more and more often since the launch of the PS4, it potentially affects EVERY console player of this game. OTher games may run just fine because they don't continually check in with PSN like XIV does. Likewise, comparing how LoL, GW2, or WoW is performing against XIV is not an accurate comparison. XIV is hosted in Canada for the NA/EU accounts--they aren't run on state-side servers in New York, LA, Chicago, or Seattle and so may not be affected by routing issues getting into and/or across Canada.

Important point of fact there... a very large part of the problems stem from getting into and/or across Canada. There are very limited entry points into Canada, as well as a limited number of trunks for getting across Canada. If AT&T, Verizon, Comcast, TWC, etc. all use the same 3 companies for this process and any number of those three get overloaded it can severely knacker up access for a LOT of people at once. This in fact is what is happening---Level3's lines are crapping out, and everyone using their routes through that same corridor are getting hosed.

[Edit:]
Just picked up signs of congestion within TWC's own network around Scarborough, Maine as well. Fluctuations from 20's up to around 400ms, with the occasional spikes up over 1 second. Things are just getting out of hand in general up near the border I guess... Northeast is showing it's usual signs for TWC, as well as the other usual areas of Southern California, Texas, Georgia, the Carolinas. Nothing new here... just more of the same oversold markets and under maintained markets we've dealt with for years:



You can find more maps like this for other services (including PSN and XBL, among others) and a few other ISP's at the downdetector's company pages:
http://downdetector.com/companies

[Mykll]

Yeah. You're right. Good call. Guess we should just ignore the 7 threads from the last 3 hours about the instance servers being totally borked. Or the (multiple) server wide and multi-person connection issues. Glad we had your insight for this problem.

Important point of fact there... a very large part of the problems stem from getting into and/or across Canada. There are very limited entry points into Canada, as well as a limited number of trunks for getting across Canada. If AT&T, Verizon, Comcast, TWC, etc. all use the same 3 companies for this process and any number of those three get overloaded it can severely knacker up access for a LOT of people at once. This in fact is what is happening---Level3's lines are crapping out, and everyone using their routes through that same corridor are getting hosed.

SE game servers are now controlling internet traffic, before your connection even gets to their servers? Huh, didn't know that. Thanks Burrmanchu.

[Raist]

Edit:
Misread previous post, and responded in error. Amended the post slightly and left it up, as it was mostly to clarify the points about restrictive access to Canada, and how a seemingly minor hiccup along a route for one person can indicate a big problem for many in the scope of FFXIV.

There are a small handful of ISP's providing access to Canada, and they basically "lease" that access out to our ISP's so that we can connect to servers in Canada. This happens all over the place. If not, we would not be able to load webpages or send email to anyone not on our ISP's own network. You may not even be able to contact people with the same ISP if they were in a disparate region--you would only be able to connect to the people on the localized network.

What this means is that if Level3's routers are overloaded/stalled/taken offline, everyone who is stuck with them as their routing partner to the region managed by those routers---their access to that region becomes compromised. This is what has been happening for YEARS. It isn't a phenomenon isolated to just XIV.

Perhaps some visuals will help demonstrate the issue better:

Cogent Communications Network Map:


Take a very close look at how their trunks access Montreal from the US (can right-click the picture and "View Image", then you can zoom in and such, as it is a higher res picture than displayed in the forum).

They basically have 3 paths... let me restate that: THREE paths that feed into Canada for getting to Montreal: Two from the East, and one from the West. That means for the US, you are coming through the New York/Massachusetts, or Washington state corridors. Unfortunately, California is routinely congested on it's own, and then you can get bogged down again coming around the WA/BC corridor even further. It doesn't matter where you are coming from in the US if your ISP has partnered with Cogent to carry your data to Montreal--you basically have those 3 paths to take to get into Canada. If there are issues along one of those limited paths, everyone using that path gets screwed at once. SE has absolutely NO control over that---that is between Cogent Communication because it is their network hardware in play, and your ISP because THEY have chosen to route you across Cogent's hardware.

Now, here is a link to Level3's map, the ISP in question from the OP:
http://maps.level3.com/default/#.U5UPHig2lW8

Just take a moment to zoom in on Montreal and look at how things get routed there via Level3's network. Guess what? It's once again NEW YORK AND MASSACHUSETTS for the US entry points if you are on their network. They don't show an entry point for the Vancouver area. It's only via the North East.

And now, here's TATA's map:


That image is hard to discern, but they basically have the same access as Cogent--one corridor for the west, two for the east. There is a link to a PDF you can load if you want to see things more clearly.

These are three of the big players up there, and who almost all of us ultimately wind up dealing with in order to get to Ormuco, who is SE's ISP in Montreal. Now, compound this with how everyone is partnered with everyone else to get us over there. For example, i-web is partnered with all three of them:

http://iweb.com/reliable-hosting
(Network Partners logos, down near bottom of the page)

Likewise, our ISP's partner with multiple other routing partners to build routes to get us to where ever we want to go on the internet. You may find you hop off and on to multiple ISP's segments to finally reach your final destination. These routes are purchased by your ISP in some fashion or another (often times it is a quid-pro-quo arrangement and not actually paid for, but an agreement IS made with your ISP for access to the route). So, when there is a problem along such routes... it falls to your ISP to investigate and resolve the issue on your behalf. Either they get their partner to fix the problem, or they switch us to an alternate route (sometimes this requires a reboot of our modem as they may need to change our modem's config). Square-Enix has no responsibility, not to mention little to no influence, to address such issues unless it is shown to be an issue with their ISP (Ormuco) or their own network.

[Zaero]

Now there's some good info here (I work in this field for a living and Raist has it nailed).

Basically, some of the devices borked out, and (I'd bet) someone's taken a "FIX DAT THANG" call by now.

[Burrmanchu]

You do realize there's actually at least two good reasons tech support asks you those "stupid" questions?

Do you realize that anyone with a basic level of human intelligence would see that the OP was -far- beyond unplugging a modem? This was not a "ITZ NOT WERKING" post. Just read it. Also your job at a ISP has nothing to do with a random forum person. They aren't required to do anything. Thanks tho.

[Burrmanchu]

SE game servers are now controlling internet traffic, before your connection even gets to their servers? Huh, didn't know that. Thanks Burrmanchu.

I suppose the 5 hour emergency maintenance tonight is to fix the internet traffic in Canada then? Let's wait and see if SE can or cannot fix our problem.... Servers or not, your original point was that SE couldn't do anything about it.

[Brine_Gildchaff]

I suppose the 5 hour emergency maintenance tonight is to fix the internet traffic in Canada then? Let's wait and see if SE can or cannot fix our problem.... Servers or not, your original point was that SE couldn't do anything about it.

Because it's totally impossible that the maintenance could be for anything besides the already-clearing lag problems, like the laundry list of known issues they might now have fixes ready to go for. I mean that would be just /absurd/, right?

[Firesped]

My guess is that the maintenance is for a hot fix. It may or may not have anything to do with this lag issue. I was lagging a little bit ago, but I just ran a pathping on 199.91.189.32 and it shows no packet loss.

[Mykll]

FIX YOUR SHIT SE I HAVE COIL TONIGHT.

Let's wait and see if SE can or cannot fix our problem....

I agree, we should wait and not immediately assume it is a SE server problem. We do not know what the problem is or what the maintenance is for. I can guess and assume all I want but until I hear SE say it...

Servers or not, your original point was that SE couldn't do anything about it.

My original point was that if there was an internet interruption somewhere, from an ISP that SE does not own, then it is not their shit to fix. They can call and put pressure on the ISP or whoever owns the broken link or router, if that is what the problem is, but they themselves cannot fix it. Yeah I am frustrated. I have been getting a lot of DCs since Saturday night. But I also understand that I am not plugged in directly to their servers, so there could be something completely out of their control to fix. They have done some boneheaded moves, absolutely, but I am also not going to come here and rant and rave and cuss them out for every little hiccup. I hope despite their silence they know exactly what the problem is, either on their end, or on some ISP end, and that it gets fixed tonight. After all, they have done maintenance at the same time as the ISP they use when they (the ISP) needed to fix something.

[Raist]

Do you realize that anyone with a basic level of human intelligence would see that the OP was -far- beyond unplugging a modem? This was not a "ITZ NOT WERKING" post. Just read it. Also your job at a ISP has nothing to do with a random forum person. They aren't required to do anything. Thanks tho.

/sigh

Fair warning... moving into Dennis Miller mode.... trying to resist it... but... can't ... hold-on....

[rant]

From page 26 of Cogent's User Guide, available here:
http://cogentco.com/files/docs/custo...user_guide.pdf

In the event that you have lost service, please have the following information available when calling Technical Support:
-
For incidents in which you have lost connectivity to the Internet or external sites:
Can you ping the gateway on your network to which the Cogent line is provisioned?
If there is no response from your gateway, please ensure that your equipment is receiving power and that all cables are properly attached. If possible, power-cycle your equipment.
If there is a response from your gateway, please specify whether or not you are able to ping sites on the Cogent Network such as your default gateway or 66.28.0.45
(res1.dns.cogentco.com).

(colorized the relevant line myself, not done in the original document)


Just saying the process is troubleshooting 101 in the IT industry, and (as noted in previous posts) the simple process of rebooting devices sometimes resolves an issue and sometimes is required to implement configuration changes that may have been made in order to address an identified problem.

In the case of an ISP, sometimes such changes are made at the servers that give out leases and configuration files without being actively pushed, but instead get handed out during the next scheduled renewals. In the case of my ISP (TWC), they have frequently made changes to config files to upgrade services on a broad scale (like bumping basic internet from 10Mb to 15Mb) on their TFTP server but do not push a reboot to everyone's modem to affect the changes immediately. They instead will insert a notice on the next bill advising people of the change and telling them to power cycle their modems to receive the updates. This allows them to exert some level of control over how the rollouts take place so they can somewhat monitor the impact on a regional basis.

So no... someone's experiences working with an ISP (or networking in general in the IT industry) actually DOES have something to do with this thread, as they have very credible knowledge and experience with the particular issues at hand. If not for my limited experiences working with TWC during their local DOCSIS 1.1 Beta rollout of Cable internet back in the 90's and then later for a small MS/IBM partnered software firm years ago when TWC was rolling out their local 2.0 Residential and Business Class internet (helped sell, plan, and implement new drops to local businesses and assisted in troubleshooting networks then just as I did during their Beta phases), I would not have the basic understanding of how all this works in order to find documentation to present to the forum to hopefully better educate the masses on how to properly address the well documented issue and to actually get it addressed by the responsible parties.

This has all happened before, and it will all happen again. It has happened time and time again for multiple games/services going back into the 90's, as well as for various content services including but not limited to Netflix, Hulu, Amazon Prime, Twitch, and JustinTV just to name a few from recent memory. It's even recently been documented as an issue for reaching content for Google/YouTube. It's amazing how much information you can find via Google if you put your mind to it.

And in nearly every case (barring instances where internal issues were found related to things with bad packet headers or actual server capacity/stability problems), it has been identified and corrected by the intermediary ISP's that were being used as routing partners--which has been identified by the OP (Level3 is in no way affiliated with XIV or SE for that matter). This particular issue that has been identified in the OP is clearly something that needs to be addressed by the ISP's at the Tier3 or higher level, and in some cases it ultimately may indeed require the power-cycling of a modem (or at least a router or client attached to it) to resolve it.

In my own past experiences with not only FFXIV, but also FFXI, Diablo (1 & 2), Total Annihilation (and other games using Microsoft's Game Zone at the time), Descent: Freespace (1 & 2), and various games on both my PS2 and PS3, and more recently my Roku's (who were having issues with throttled content because of a routing partner's shaping policy), the fix has often required me to at least reboot my network hardware or at least conduct a reset of my connection config (either by a ipconfig /renew or a full out netsh int TCP reset). In each case where such a reboot was required it was because TWC/RR was basically changing my modem's "home" of sorts as a quick and dirty fix to get me rerouted. It basically was "spoofing" my geolocation so that I would get sent along a considerably different path to avoid stability issues until a permanent fix could be put in place along my normal route from the Carolinas. After rebooting and getting assigned a new route, I would be asked to test and verify if routing was better, and if necessary we would do it again until we found a viable path.

Back during the holidays I was going through this process about every 3 weeks with TWC trying to stay ahead of the congestion issues... and guess what? It kept me playing while everyone else was continuously flaking out and unable to play, coming here to just scream and complain... blaming it all SOLELY on SE. But, you know the really interesting thing? Those who listened to the advice of those trying to explain this process and how to remedy it that actually took the advice and properly pursued it--they got results. In one such case, a small group in the UK (O2/BT Orange customers) actually fingered an issue that was impacting an entire region over there and managed to get the ISP's in touch with SE about an issue with their packet headers. It was all over their forums back then, just like TWC currently has a firestorm on their forums to address their identified internal routing problems (we've just about got things resolved for the most part with their partners). This ultimately resulted in a change that addressed a large part of the throttling issue that was impacting people worldwide last year, and because we've recently identified and affected changes addressing a big problem with our headends here in SC, we've fixed issues for a large portion of the eastern half of our state--hopefully more fixes will come with new channel alignment rollouts that are on the way (haven't been able to drag any details out of them about speed increases, but they are considering channel realignments because of identified ingress issues between 573 and 615MHz.).

All because some people familiar with the ISP's and how the system works got involved and got a group of people to lean on their ISP's to affect a change locally--and I assure you , to properly implement some of those changes, a LOT of modems had to be rebooted.

So, I respectfully advise that you be careful about belittling the advice of someone connected to the industry who actually may have pertinent knowledge of the situation and is actually providing relevant advice.

[/endrant]