Details about Server issues

[Mykll]

Yeah. You're right. Good call. Guess we should just ignore the 7 threads from the last 3 hours about the instance servers being totally borked. Or the (multiple) server wide and multi-person connection issues. Glad we had your insight for this problem.

Important point of fact there... a very large part of the problems stem from getting into and/or across Canada. There are very limited entry points into Canada, as well as a limited number of trunks for getting across Canada. If AT&T, Verizon, Comcast, TWC, etc. all use the same 3 companies for this process and any number of those three get overloaded it can severely knacker up access for a LOT of people at once. This in fact is what is happening---Level3's lines are crapping out, and everyone using their routes through that same corridor are getting hosed.

SE game servers are now controlling internet traffic, before your connection even gets to their servers? Huh, didn't know that. Thanks Burrmanchu.

[Raist]

Edit:
Misread previous post, and responded in error. Amended the post slightly and left it up, as it was mostly to clarify the points about restrictive access to Canada, and how a seemingly minor hiccup along a route for one person can indicate a big problem for many in the scope of FFXIV.

There are a small handful of ISP's providing access to Canada, and they basically "lease" that access out to our ISP's so that we can connect to servers in Canada. This happens all over the place. If not, we would not be able to load webpages or send email to anyone not on our ISP's own network. You may not even be able to contact people with the same ISP if they were in a disparate region--you would only be able to connect to the people on the localized network.

What this means is that if Level3's routers are overloaded/stalled/taken offline, everyone who is stuck with them as their routing partner to the region managed by those routers---their access to that region becomes compromised. This is what has been happening for YEARS. It isn't a phenomenon isolated to just XIV.

Perhaps some visuals will help demonstrate the issue better:

Cogent Communications Network Map:


Take a very close look at how their trunks access Montreal from the US (can right-click the picture and "View Image", then you can zoom in and such, as it is a higher res picture than displayed in the forum).

They basically have 3 paths... let me restate that: THREE paths that feed into Canada for getting to Montreal: Two from the East, and one from the West. That means for the US, you are coming through the New York/Massachusetts, or Washington state corridors. Unfortunately, California is routinely congested on it's own, and then you can get bogged down again coming around the WA/BC corridor even further. It doesn't matter where you are coming from in the US if your ISP has partnered with Cogent to carry your data to Montreal--you basically have those 3 paths to take to get into Canada. If there are issues along one of those limited paths, everyone using that path gets screwed at once. SE has absolutely NO control over that---that is between Cogent Communication because it is their network hardware in play, and your ISP because THEY have chosen to route you across Cogent's hardware.

Now, here is a link to Level3's map, the ISP in question from the OP:
http://maps.level3.com/default/#.U5UPHig2lW8

Just take a moment to zoom in on Montreal and look at how things get routed there via Level3's network. Guess what? It's once again NEW YORK AND MASSACHUSETTS for the US entry points if you are on their network. They don't show an entry point for the Vancouver area. It's only via the North East.

And now, here's TATA's map:


That image is hard to discern, but they basically have the same access as Cogent--one corridor for the west, two for the east. There is a link to a PDF you can load if you want to see things more clearly.

These are three of the big players up there, and who almost all of us ultimately wind up dealing with in order to get to Ormuco, who is SE's ISP in Montreal. Now, compound this with how everyone is partnered with everyone else to get us over there. For example, i-web is partnered with all three of them:

http://iweb.com/reliable-hosting
(Network Partners logos, down near bottom of the page)

Likewise, our ISP's partner with multiple other routing partners to build routes to get us to where ever we want to go on the internet. You may find you hop off and on to multiple ISP's segments to finally reach your final destination. These routes are purchased by your ISP in some fashion or another (often times it is a quid-pro-quo arrangement and not actually paid for, but an agreement IS made with your ISP for access to the route). So, when there is a problem along such routes... it falls to your ISP to investigate and resolve the issue on your behalf. Either they get their partner to fix the problem, or they switch us to an alternate route (sometimes this requires a reboot of our modem as they may need to change our modem's config). Square-Enix has no responsibility, not to mention little to no influence, to address such issues unless it is shown to be an issue with their ISP (Ormuco) or their own network.

[Burrmanchu]

SE game servers are now controlling internet traffic, before your connection even gets to their servers? Huh, didn't know that. Thanks Burrmanchu.

I suppose the 5 hour emergency maintenance tonight is to fix the internet traffic in Canada then? Let's wait and see if SE can or cannot fix our problem.... Servers or not, your original point was that SE couldn't do anything about it.

[Brine_Gildchaff]

I suppose the 5 hour emergency maintenance tonight is to fix the internet traffic in Canada then? Let's wait and see if SE can or cannot fix our problem.... Servers or not, your original point was that SE couldn't do anything about it.

Because it's totally impossible that the maintenance could be for anything besides the already-clearing lag problems, like the laundry list of known issues they might now have fixes ready to go for. I mean that would be just /absurd/, right?

[Burrmanchu]

Because it's totally impossible that the maintenance could be for anything besides the already-clearing lag problems, like the laundry list of known issues they might now have fixes ready to go for. I mean that would be just /absurd/, right?

Another excellent point. Why would I ever assume that a long emergency maintenance after 3 days of an unplayable game, would in any way be related to the game being unplayable? Silly me. I'm so /absurd/.

On another note, the game's been fine all morning. No d/c's yet. I have no clue what other "already-clearing" (wut?) lag problems you're talking about, and neither does anyone else I know. Maybe you should restart your modem.

edit: YEP! It fixed it! Good call.

[Raist]

Actually Burrmanchu, you have been the one launching the attacks... I've been more focused on trying to demonstrate the flaws with the logic behind them. Even after trying to address such inflammatory and off-topic posts, I still try to bring my post back to the actual topic brought forth in the OP. But it's hard sometimes to remain completely focused and civil when one is posting such insulting/inflammatory or otherwise detracting statements such as yours. If you are going to continue making such posts, you should be prepared for a little push back now and then.

The issue at hand is the routing issues we have been investigating, documenting, and getting our ISP's to address. The act of resetting the modem has been shown to be part of the standard troubleshooting process and at times even a critical part of the remedy to address ROUTING ISSUES THAT ARE DICTATED BY OUR ISP. SE has absolutely nothing to do with those policies, and little to no influence over what our ISP's routing partners are doing with their networks. Our ISP's are the ones with enforceable agreements to affect changes, changes that at times (again) require a modem reset.

No way around it... the process suggested was valid, relative, and depending on exactly what has been going on behind the scenes already may have actually resolved the issue for the OP, as there actually is a chance that it could have led him/her to switching to an alternate route assignment and bypassed the failing segment.

You have aggressively gone after people that gave valid suggestions on how to remedy the situation, and have attempted to take the thread off-topic. The issue at hand has nothing to do with SE's server stability issues---that is a completely separate issue. This thread has not been about server stability and connectivity, it has been about extreme latency that is interrupting communication in route---it is crashing before the server can even be reached. Even if after the maintenance and potentially reboot of their hardware and people are able to connect consistently again until they become overloaded once more, the type of latency issue brought forth by the OP will still be cropping up until it is properly addressed, and the advice put forth within this thread will still apply to addressing such issues.

Instead of reporting your actions (which I am beginnning to regret not doing), I chose to see this as a teachable moment and took the opportunity to try to demonstrate how and why this problem persists, how impactful the issue is, how one may go about addressing the issue properly, and why previous suggestions that had been challenged were correct. Perhaps I made an error in judgement in hoping to enlighten some forum members that might not have a good understanding of the situation and how to address it. The Dennis Miller bit was an attempt at injecting some levity into the aggressive tone that had sprung forth in the thread... a throw back to his early days of starting his witty diatribes with "I hate to get off on a rant here, but...". His criticisms always had a hint of truth buried in there, and if one paid close enough attention to them, some precious pearls of wisdom could be gathered.

Guess I just dated myself... oh well, it's a shame it has fallen on some deaf ears. But, hopefully it gets through to some, so it's worth the jabs I am now getting from you if it gets people more proactive in pushing their ISP's to get involved and improve things for more people.

Oh, and just an FYI... I in fact have recently rebooted my modem, got new channel alignments, and my local segments have been much more responsive since (thanks for asking). I will likely have to bounce it again this weekend though, and it's about time for them to open a new route as TATA has started to flake out on us now (will get to that in a second). It's been an ongoing process for about 2 months now while we've been trying to tackle a troublesome CMTS that at first was only bonding 4 channels (and poorly at that), eventually got up to 6, but is now finally running at full capacity with all 8 channels bonding. We finally tracked down the ingress that was knocking them offline, but they are concerned it may come back because of aging hardware and lines... TWC's infrastructure is in really bad shape down here. New channel lineups are on the way, and the hopes are that we may be able to get new assignments for network use and get away from the nearby transmissions at channels 31, 35, and 38 that have been giving us problems. Until then, we will have to continue to reboot our modems to lock on to cleaner primary channels whenever we see our headend's latency start spiking high--when things go south, it will start breaching a full second or more in delayed response at the start of our run, which just gets uglier as we go upstream towards Canada. Three guesses on the quickest way we resolve this issue...

A modem reset.

It should also be noted that latency has already begun to creep up on TATA's lines tonight (who my ISP has most recently paired me with after my last reset)---already picked up spikes over 700ms getting into Montreal via one of their routes. That's approaching a 3/4 second lag for a single transmission cycle between client and server (in case some aren't aware of the impact), and it isn't even prime time yet. Others may soon find similar issues with their routing as things get busier in the days/weeks to come. So, we may unfortunately find ourselves right back here again having the same debate, even after some may have gotten the impression that SE fixed all their latency woes with last night's maintenance.....

[Burrmanchu]

TOO MANY WORDS

I only read one sentence of that ridiculous wall of bs.. but once again, this issue had nothing to do with restarting modems. It's fantastic that the first thing inet providers tell you to do is just that, but exactly as I said, this wasn't related.. And if you think my comments are just so insanely offensive and "inflammatory", you're in for a HUGE surprise when you surf literally any website on the internet. Good job posting another great wall of text that has no point inside it whatsoever.

It's absolutely AMAZING that the server woes (i mean, CANADIAN INTERNET TRAFFIC ZOMG) were totally fixed after the maintenance.. Must be a complete coincidence.

Maintenance fixed the issue. Just like I said. As far as me harassing YOU, I originally replied to someone else, and you typed a 1500 character dissertation on how I need to think and behave... Move along please.

[Raist]

I only read one sentence of that ridiculous wall of bs.. but once again, this issue had nothing to do with restarting modems. It's fantastic that the first thing inet providers tell you to do is just that, but exactly as I said, this wasn't related.. And if you think my comments are just so insanely offensive and "inflammatory", you're in for a HUGE surprise when you surf literally any website on the internet. Good job posting another great wall of text that has no point inside it whatsoever.

It's absolutely AMAZING that the server woes (i mean, CANADIAN INTERNET TRAFFIC ZOMG) were totally fixed after the maintenance.. Must be a complete coincidence.

Maintenance fixed the issue. Just like I said. As far as me harassing YOU, I originally replied to someone else, and you typed a 1500 character dissertation on how I need to think and behave... Move along please.

Actually, for many the maintenance did not resolve the problem. Nor did the recent one on the 23rd... notice how the threads keep popping back up?

And by the way... in case you haven't been following such threads, I bounced my modem last week and I am now getting routed through Atlanta (instead of Charlotte/Raleigh), and I am using Cogent/PsiNet instead of TATA, and my conx is about the best it's ever been. Didn't contact TWC or anything, just my usual routine of resetting my modem when the connection starts to act up... and low and behold everything changed for me.

Just saying... sometimes it actually works, and it only takes minutes to do, so why are you so against the idea?

[Mykll]

FIX YOUR SHIT SE I HAVE COIL TONIGHT.

Let's wait and see if SE can or cannot fix our problem....

I agree, we should wait and not immediately assume it is a SE server problem. We do not know what the problem is or what the maintenance is for. I can guess and assume all I want but until I hear SE say it...

Servers or not, your original point was that SE couldn't do anything about it.

My original point was that if there was an internet interruption somewhere, from an ISP that SE does not own, then it is not their shit to fix. They can call and put pressure on the ISP or whoever owns the broken link or router, if that is what the problem is, but they themselves cannot fix it. Yeah I am frustrated. I have been getting a lot of DCs since Saturday night. But I also understand that I am not plugged in directly to their servers, so there could be something completely out of their control to fix. They have done some boneheaded moves, absolutely, but I am also not going to come here and rant and rave and cuss them out for every little hiccup. I hope despite their silence they know exactly what the problem is, either on their end, or on some ISP end, and that it gets fixed tonight. After all, they have done maintenance at the same time as the ISP they use when they (the ISP) needed to fix something.