Today there are issues with connectivity to the Montreal datacenter.
The issues are located somewhere in the routing and peering between Level3 and Ormuco Communications
The node between ORMUCO-COMM.car2.Montreal2.Level3.net[4.59.178.74] and [19.34.76.2] has been failing to respond all day.
Significant congestion can be seen upon entering the level3 network from as far west as Seattle at ae-32-52.ebr2.Seattle1.Level3.net [4.69.147.182] where connection latency goes up 300%
Details about Server issues
Printable View
-
06-09-2014, 03:17 AMBreviticDetails about Server issues
-
06-09-2014, 03:19 AMseannykun
IT'S BROKE
-
06-09-2014, 03:20 AMMega64
Try unplugging and replugging the modem.
-
06-09-2014, 03:21 AMSonoranUmm a lot of people are having this issue, its not all of our routersQuote:
Originally Posted by Mega64
-
06-09-2014, 03:22 AMseannykun
Try it anyways.
-
06-09-2014, 03:22 AMNamnesor
No I don't have any big downloads running MOM
-
06-09-2014, 03:58 AMBurrmanchu
Servers are fucked.. has nothing to do with this guys router. Jfc people. D/c'd many times in last hour. Expert roulette run ruined and lost a T2 drop 20min prior... FIX YOUR SHIT SE I HAVE COIL TONIGHT.
&@Mega64: Did you read the OP? The guy has it pinged and nailed down to a specific communication error and has a half page of techinical mumbo jumbo pinpointing latency vs areas on the globe. Did you really need to tell him to reset his freaking modem? People never cease to amaze me. You must be that guy I always have to talk to at Comcast.
"IS IT PLUGGED IN?" -
06-09-2014, 04:09 AMMykll
Probably nothing wrong with their servers. Thus probably nothing for SE to fix.
-
06-09-2014, 05:09 AMBurrmanchuYeah. You're right. Good call. Guess we should just ignore the 7 threads from the last 3 hours about the instance servers being totally borked. Or the (multiple) server wide and multi-person connection issues. Glad we had your insight for this problem.Quote:
Originally Posted by Mykll
-
06-09-2014, 05:39 AMEnepttasticYou do realize there's actually at least two good reasons tech support asks you those "stupid" questions?Quote:
Originally Posted by Burrmanchu
1. Any troubleshooter worth a damn will make sure and check any potential cause of the problem, often starting with the simplest to confirm first. Since the person at the other end of the line may not be able to check if your modem/router's plugged in, they gotta ask.
2. The sheer number of calls I had in the short time-span I did ISP tech support that dealt with a bitchy customer who rants and raves about their service being interrupted finding out that the modem was, in fact, NOT plugged in was staggering.
Another reason is that they HAVE to ask those stupid questions else the rep is going to get in trouble for potentially wasting time, not just the companies but yours as well. -
06-09-2014, 07:40 AMRaist
In addition to Enepttastic's post, have to understand that this (as usual) loosk like a routing problem occurring upstream from our localized networks. This is something managed by your ISP, and is affected by your point of origin and the requested destination. Rebooting your network can cause new leases to be assigned, effectively sometimes changing something in your auto-assigned IP config or otherwise correcting an issue with your geolocation---which has been known to resolve issues in the past. It is also sometimes required by your ISP in order for you to download an updated config file from the ISP's TFTP server to address potential issues with your modem and/or their network.
Basically... the old Microsoft mantra of rebooting your devices can and does have it's place because it is a documented step in the troubleshooting process that has been shown to sometimes work and at times is actually necessary to address issues.
Back to the OP though, this is something that needs to be addressed by your ISP. you ned to get in touch with no less then their Tier3 people---not the tier1 and tier2's that you will get on the phone when you call in or that come out and check the cables and possibly swap out your modem. That's not to say that you don't have an issue with your network---you may find signs of issues because of out of spec Signal to Noise Ratio (SNR) or other values listed on your modem's signal pages that those guys will need to address. But you have specifically identified the very common issue we've all been dealing with for some time now---a routing issue with our ISP's routing partners. Our ISP's have the enforceable agreements with these third (and sometimes fourth) parties and not SE, so it actually falls to our ISP's to go lean on them to address their problems on our behalf.
And yes, such an issue can affect a LOT of players at the same time. When PSN goes AWOL like it has been doing more and more often since the launch of the PS4, it potentially affects EVERY console player of this game. OTher games may run just fine because they don't continually check in with PSN like XIV does. Likewise, comparing how LoL, GW2, or WoW is performing against XIV is not an accurate comparison. XIV is hosted in Canada for the NA/EU accounts--they aren't run on state-side servers in New York, LA, Chicago, or Seattle and so may not be affected by routing issues getting into and/or across Canada.
Important point of fact there... a very large part of the problems stem from getting into and/or across Canada. There are very limited entry points into Canada, as well as a limited number of trunks for getting across Canada. If AT&T, Verizon, Comcast, TWC, etc. all use the same 3 companies for this process and any number of those three get overloaded it can severely knacker up access for a LOT of people at once. This in fact is what is happening---Level3's lines are crapping out, and everyone using their routes through that same corridor are getting hosed.
[Edit:]
Just picked up signs of congestion within TWC's own network around Scarborough, Maine as well. Fluctuations from 20's up to around 400ms, with the occasional spikes up over 1 second. Things are just getting out of hand in general up near the border I guess... Northeast is showing it's usual signs for TWC, as well as the other usual areas of Southern California, Texas, Georgia, the Carolinas. Nothing new here... just more of the same oversold markets and under maintained markets we've dealt with for years:
http://allestoringen.nl/maps/20096/latest.jpg
You can find more maps like this for other services (including PSN and XBL, among others) and a few other ISP's at the downdetector's company pages:
http://downdetector.com/companies -
06-09-2014, 09:50 AMMykllQuote:
Originally Posted by Burrmanchu
SE game servers are now controlling internet traffic, before your connection even gets to their servers? Huh, didn't know that. Thanks Burrmanchu.Quote:
Originally Posted by Raist
-
06-09-2014, 10:47 AMRaist
Edit:
Misread previous post, and responded in error. Amended the post slightly and left it up, as it was mostly to clarify the points about restrictive access to Canada, and how a seemingly minor hiccup along a route for one person can indicate a big problem for many in the scope of FFXIV.
There are a small handful of ISP's providing access to Canada, and they basically "lease" that access out to our ISP's so that we can connect to servers in Canada. This happens all over the place. If not, we would not be able to load webpages or send email to anyone not on our ISP's own network. You may not even be able to contact people with the same ISP if they were in a disparate region--you would only be able to connect to the people on the localized network.
What this means is that if Level3's routers are overloaded/stalled/taken offline, everyone who is stuck with them as their routing partner to the region managed by those routers---their access to that region becomes compromised. This is what has been happening for YEARS. It isn't a phenomenon isolated to just XIV.
Perhaps some visuals will help demonstrate the issue better:
Cogent Communications Network Map:
http://www.cogentco.com/files/images...obal_large.png
Take a very close look at how their trunks access Montreal from the US (can right-click the picture and "View Image", then you can zoom in and such, as it is a higher res picture than displayed in the forum).
They basically have 3 paths... let me restate that: THREE paths that feed into Canada for getting to Montreal: Two from the East, and one from the West. That means for the US, you are coming through the New York/Massachusetts, or Washington state corridors. Unfortunately, California is routinely congested on it's own, and then you can get bogged down again coming around the WA/BC corridor even further. It doesn't matter where you are coming from in the US if your ISP has partnered with Cogent to carry your data to Montreal--you basically have those 3 paths to take to get into Canada. If there are issues along one of those limited paths, everyone using that path gets screwed at once. SE has absolutely NO control over that---that is between Cogent Communication because it is their network hardware in play, and your ISP because THEY have chosen to route you across Cogent's hardware.
Now, here is a link to Level3's map, the ISP in question from the OP:
http://maps.level3.com/default/#.U5UPHig2lW8
Just take a moment to zoom in on Montreal and look at how things get routed there via Level3's network. Guess what? It's once again NEW YORK AND MASSACHUSETTS for the US entry points if you are on their network. They don't show an entry point for the Vancouver area. It's only via the North East.
And now, here's TATA's map:
http://www.tatacommunications.com/si....1_26-9-13.png
That image is hard to discern, but they basically have the same access as Cogent--one corridor for the west, two for the east. There is a link to a PDF you can load if you want to see things more clearly.
These are three of the big players up there, and who almost all of us ultimately wind up dealing with in order to get to Ormuco, who is SE's ISP in Montreal. Now, compound this with how everyone is partnered with everyone else to get us over there. For example, i-web is partnered with all three of them:
http://iweb.com/reliable-hosting
(Network Partners logos, down near bottom of the page)
Likewise, our ISP's partner with multiple other routing partners to build routes to get us to where ever we want to go on the internet. You may find you hop off and on to multiple ISP's segments to finally reach your final destination. These routes are purchased by your ISP in some fashion or another (often times it is a quid-pro-quo arrangement and not actually paid for, but an agreement IS made with your ISP for access to the route). So, when there is a problem along such routes... it falls to your ISP to investigate and resolve the issue on your behalf. Either they get their partner to fix the problem, or they switch us to an alternate route (sometimes this requires a reboot of our modem as they may need to change our modem's config). Square-Enix has no responsibility, not to mention little to no influence, to address such issues unless it is shown to be an issue with their ISP (Ormuco) or their own network. -
06-09-2014, 11:33 AMZaero
Now there's some good info here (I work in this field for a living and Raist has it nailed).
Basically, some of the devices borked out, and (I'd bet) someone's taken a "FIX DAT THANG" call by now. -
06-10-2014, 05:33 AMBurrmanchuDo you realize that anyone with a basic level of human intelligence would see that the OP was -far- beyond unplugging a modem? This was not a "ITZ NOT WERKING" post. Just read it. Also your job at a ISP has nothing to do with a random forum person. They aren't required to do anything. Thanks tho.Quote:
Originally Posted by Enepttastic
-
06-10-2014, 05:35 AMBurrmanchuI suppose the 5 hour emergency maintenance tonight is to fix the internet traffic in Canada then? Let's wait and see if SE can or cannot fix our problem.... Servers or not, your original point was that SE couldn't do anything about it.Quote:
Originally Posted by Mykll
-
06-10-2014, 05:48 AMBrine_GildchaffBecause it's totally impossible that the maintenance could be for anything besides the already-clearing lag problems, like the laundry list of known issues they might now have fixes ready to go for. I mean that would be just /absurd/, right?Quote:
Originally Posted by Burrmanchu
-
06-10-2014, 06:26 AMFiresped
My guess is that the maintenance is for a hot fix. It may or may not have anything to do with this lag issue. I was lagging a little bit ago, but I just ran a pathping on 199.91.189.32 and it shows no packet loss.
-
06-10-2014, 06:44 AMMykllQuote:
Originally Posted by Burrmanchu
I agree, we should wait and not immediately assume it is a SE server problem. We do not know what the problem is or what the maintenance is for. I can guess and assume all I want but until I hear SE say it...Quote:
Originally Posted by Burrmanchu
My original point was that if there was an internet interruption somewhere, from an ISP that SE does not own, then it is not their shit to fix. They can call and put pressure on the ISP or whoever owns the broken link or router, if that is what the problem is, but they themselves cannot fix it. Yeah I am frustrated. I have been getting a lot of DCs since Saturday night. But I also understand that I am not plugged in directly to their servers, so there could be something completely out of their control to fix. They have done some boneheaded moves, absolutely, but I am also not going to come here and rant and rave and cuss them out for every little hiccup. I hope despite their silence they know exactly what the problem is, either on their end, or on some ISP end, and that it gets fixed tonight. After all, they have done maintenance at the same time as the ISP they use when they (the ISP) needed to fix something.Quote:
Originally Posted by Burrmanchu
-
06-10-2014, 10:39 AMRaist/sighQuote:
Originally Posted by Burrmanchu
Fair warning... moving into Dennis Miller mode.... trying to resist it... but... can't ... hold-on....
[rant]
From page 26 of Cogent's User Guide, available here:
http://cogentco.com/files/docs/custo...user_guide.pdf
(colorized the relevant line myself, not done in the original document)Quote:
In the event that you have lost service, please have the following information available when calling Technical Support:
-
For incidents in which you have lost connectivity to the Internet or external sites:
Can you ping the gateway on your network to which the Cogent line is provisioned?
If there is no response from your gateway, please ensure that your equipment is receiving power and that all cables are properly attached. If possible, power-cycle your equipment.
If there is a response from your gateway, please specify whether or not you are able to ping sites on the Cogent Network such as your default gateway or 66.28.0.45
(res1.dns.cogentco.com).
Just saying the process is troubleshooting 101 in the IT industry, and (as noted in previous posts) the simple process of rebooting devices sometimes resolves an issue and sometimes is required to implement configuration changes that may have been made in order to address an identified problem.
In the case of an ISP, sometimes such changes are made at the servers that give out leases and configuration files without being actively pushed, but instead get handed out during the next scheduled renewals. In the case of my ISP (TWC), they have frequently made changes to config files to upgrade services on a broad scale (like bumping basic internet from 10Mb to 15Mb) on their TFTP server but do not push a reboot to everyone's modem to affect the changes immediately. They instead will insert a notice on the next bill advising people of the change and telling them to power cycle their modems to receive the updates. This allows them to exert some level of control over how the rollouts take place so they can somewhat monitor the impact on a regional basis.
So no... someone's experiences working with an ISP (or networking in general in the IT industry) actually DOES have something to do with this thread, as they have very credible knowledge and experience with the particular issues at hand. If not for my limited experiences working with TWC during their local DOCSIS 1.1 Beta rollout of Cable internet back in the 90's and then later for a small MS/IBM partnered software firm years ago when TWC was rolling out their local 2.0 Residential and Business Class internet (helped sell, plan, and implement new drops to local businesses and assisted in troubleshooting networks then just as I did during their Beta phases), I would not have the basic understanding of how all this works in order to find documentation to present to the forum to hopefully better educate the masses on how to properly address the well documented issue and to actually get it addressed by the responsible parties.
This has all happened before, and it will all happen again. It has happened time and time again for multiple games/services going back into the 90's, as well as for various content services including but not limited to Netflix, Hulu, Amazon Prime, Twitch, and JustinTV just to name a few from recent memory. It's even recently been documented as an issue for reaching content for Google/YouTube. It's amazing how much information you can find via Google if you put your mind to it.
And in nearly every case (barring instances where internal issues were found related to things with bad packet headers or actual server capacity/stability problems), it has been identified and corrected by the intermediary ISP's that were being used as routing partners--which has been identified by the OP (Level3 is in no way affiliated with XIV or SE for that matter). This particular issue that has been identified in the OP is clearly something that needs to be addressed by the ISP's at the Tier3 or higher level, and in some cases it ultimately may indeed require the power-cycling of a modem (or at least a router or client attached to it) to resolve it.
In my own past experiences with not only FFXIV, but also FFXI, Diablo (1 & 2), Total Annihilation (and other games using Microsoft's Game Zone at the time), Descent: Freespace (1 & 2), and various games on both my PS2 and PS3, and more recently my Roku's (who were having issues with throttled content because of a routing partner's shaping policy), the fix has often required me to at least reboot my network hardware or at least conduct a reset of my connection config (either by a ipconfig /renew or a full out netsh int TCP reset). In each case where such a reboot was required it was because TWC/RR was basically changing my modem's "home" of sorts as a quick and dirty fix to get me rerouted. It basically was "spoofing" my geolocation so that I would get sent along a considerably different path to avoid stability issues until a permanent fix could be put in place along my normal route from the Carolinas. After rebooting and getting assigned a new route, I would be asked to test and verify if routing was better, and if necessary we would do it again until we found a viable path.
Back during the holidays I was going through this process about every 3 weeks with TWC trying to stay ahead of the congestion issues... and guess what? It kept me playing while everyone else was continuously flaking out and unable to play, coming here to just scream and complain... blaming it all SOLELY on SE. But, you know the really interesting thing? Those who listened to the advice of those trying to explain this process and how to remedy it that actually took the advice and properly pursued it--they got results. In one such case, a small group in the UK (O2/BT Orange customers) actually fingered an issue that was impacting an entire region over there and managed to get the ISP's in touch with SE about an issue with their packet headers. It was all over their forums back then, just like TWC currently has a firestorm on their forums to address their identified internal routing problems (we've just about got things resolved for the most part with their partners). This ultimately resulted in a change that addressed a large part of the throttling issue that was impacting people worldwide last year, and because we've recently identified and affected changes addressing a big problem with our headends here in SC, we've fixed issues for a large portion of the eastern half of our state--hopefully more fixes will come with new channel alignment rollouts that are on the way (haven't been able to drag any details out of them about speed increases, but they are considering channel realignments because of identified ingress issues between 573 and 615MHz.).
All because some people familiar with the ISP's and how the system works got involved and got a group of people to lean on their ISP's to affect a change locally--and I assure you , to properly implement some of those changes, a LOT of modems had to be rebooted.
So, I respectfully advise that you be careful about belittling the advice of someone connected to the industry who actually may have pertinent knowledge of the situation and is actually providing relevant advice.
[/endrant] -
06-11-2014, 01:03 AMBurrmanchuOh dear it's Dennis Miller... /sigh away. The point is the OP's past the "replug your modem" stage. By defending this "infallible" policy again to me, you've sidestepped the entire point I made. Yeah, if you work at a ISP you have people do this. Yeah, in some instances it works. But by belittling ME (as you've explained people should take care to do), you're making the case that even after YOU put a full page post about the internet traffic, and routing problems within select areas, we should also tell YOU to unplug your modem and plug it back in. Right? Because that's what to do.Quote:
Originally Posted by Raist
Point made : The OP (specifically) is miles past restarting a modem, yet it was advised.
Point taken by you : Restarting the modem never works...? (i guess? you missed my point by a mile)
See any disconnect here?
As far as my comment about ISP jobs having nothing to do with forum people, it's a direct snap at the person explaining that the reason the internet troll said to reboot, is because said unrelated person (not the troll) works at an ISP. Which it isn't, and once again, is not related. My comment had nothing to do with "this thread as a whole". Stop attacking me directly because someone told the OP something they didn't need to know and I mentioned it. Srsly wtf? If someone would have told you to restart your modem after the giant post you made about connections, I would have told them the same thing. Save your sad Dennis Miller jive for someone who cares. If I'm the most /sigh-worthy person you've read on the internet, you're on a different internet than the rest of us.
If you're going to type a great wall of text telling someone how they're wrong, you should figure out what they're talking about first.
Now, stop trying to help and go restart your modem.
There, happy now? -
06-11-2014, 04:05 AMBurrmanchuAnother excellent point. Why would I ever assume that a long emergency maintenance after 3 days of an unplayable game, would in any way be related to the game being unplayable? Silly me. I'm so /absurd/.Quote:
Originally Posted by Brine_Gildchaff
On another note, the game's been fine all morning. No d/c's yet. I have no clue what other "already-clearing" (wut?) lag problems you're talking about, and neither does anyone else I know. Maybe you should restart your modem.
edit: YEP! It fixed it! Good call. -
06-11-2014, 08:16 AMRaist
Actually Burrmanchu, you have been the one launching the attacks... I've been more focused on trying to demonstrate the flaws with the logic behind them. Even after trying to address such inflammatory and off-topic posts, I still try to bring my post back to the actual topic brought forth in the OP. But it's hard sometimes to remain completely focused and civil when one is posting such insulting/inflammatory or otherwise detracting statements such as yours. If you are going to continue making such posts, you should be prepared for a little push back now and then.
The issue at hand is the routing issues we have been investigating, documenting, and getting our ISP's to address. The act of resetting the modem has been shown to be part of the standard troubleshooting process and at times even a critical part of the remedy to address ROUTING ISSUES THAT ARE DICTATED BY OUR ISP. SE has absolutely nothing to do with those policies, and little to no influence over what our ISP's routing partners are doing with their networks. Our ISP's are the ones with enforceable agreements to affect changes, changes that at times (again) require a modem reset.
No way around it... the process suggested was valid, relative, and depending on exactly what has been going on behind the scenes already may have actually resolved the issue for the OP, as there actually is a chance that it could have led him/her to switching to an alternate route assignment and bypassed the failing segment.
You have aggressively gone after people that gave valid suggestions on how to remedy the situation, and have attempted to take the thread off-topic. The issue at hand has nothing to do with SE's server stability issues---that is a completely separate issue. This thread has not been about server stability and connectivity, it has been about extreme latency that is interrupting communication in route---it is crashing before the server can even be reached. Even if after the maintenance and potentially reboot of their hardware and people are able to connect consistently again until they become overloaded once more, the type of latency issue brought forth by the OP will still be cropping up until it is properly addressed, and the advice put forth within this thread will still apply to addressing such issues.
Instead of reporting your actions (which I am beginnning to regret not doing), I chose to see this as a teachable moment and took the opportunity to try to demonstrate how and why this problem persists, how impactful the issue is, how one may go about addressing the issue properly, and why previous suggestions that had been challenged were correct. Perhaps I made an error in judgement in hoping to enlighten some forum members that might not have a good understanding of the situation and how to address it. The Dennis Miller bit was an attempt at injecting some levity into the aggressive tone that had sprung forth in the thread... a throw back to his early days of starting his witty diatribes with "I hate to get off on a rant here, but...". His criticisms always had a hint of truth buried in there, and if one paid close enough attention to them, some precious pearls of wisdom could be gathered.
Guess I just dated myself... oh well, it's a shame it has fallen on some deaf ears. But, hopefully it gets through to some, so it's worth the jabs I am now getting from you if it gets people more proactive in pushing their ISP's to get involved and improve things for more people.
Oh, and just an FYI... I in fact have recently rebooted my modem, got new channel alignments, and my local segments have been much more responsive since (thanks for asking). I will likely have to bounce it again this weekend though, and it's about time for them to open a new route as TATA has started to flake out on us now (will get to that in a second). It's been an ongoing process for about 2 months now while we've been trying to tackle a troublesome CMTS that at first was only bonding 4 channels (and poorly at that), eventually got up to 6, but is now finally running at full capacity with all 8 channels bonding. We finally tracked down the ingress that was knocking them offline, but they are concerned it may come back because of aging hardware and lines... TWC's infrastructure is in really bad shape down here. New channel lineups are on the way, and the hopes are that we may be able to get new assignments for network use and get away from the nearby transmissions at channels 31, 35, and 38 that have been giving us problems. Until then, we will have to continue to reboot our modems to lock on to cleaner primary channels whenever we see our headend's latency start spiking high--when things go south, it will start breaching a full second or more in delayed response at the start of our run, which just gets uglier as we go upstream towards Canada. Three guesses on the quickest way we resolve this issue...
A modem reset.
It should also be noted that latency has already begun to creep up on TATA's lines tonight (who my ISP has most recently paired me with after my last reset)---already picked up spikes over 700ms getting into Montreal via one of their routes. That's approaching a 3/4 second lag for a single transmission cycle between client and server (in case some aren't aware of the impact), and it isn't even prime time yet. Others may soon find similar issues with their routing as things get busier in the days/weeks to come. So, we may unfortunately find ourselves right back here again having the same debate, even after some may have gotten the impression that SE fixed all their latency woes with last night's maintenance..... -
06-26-2014, 07:13 AMBurrmanchuI only read one sentence of that ridiculous wall of bs.. but once again, this issue had nothing to do with restarting modems. It's fantastic that the first thing inet providers tell you to do is just that, but exactly as I said, this wasn't related.. And if you think my comments are just so insanely offensive and "inflammatory", you're in for a HUGE surprise when you surf literally any website on the internet. Good job posting another great wall of text that has no point inside it whatsoever.Quote:
Originally Posted by Raist
It's absolutely AMAZING that the server woes (i mean, CANADIAN INTERNET TRAFFIC ZOMG) were totally fixed after the maintenance.. Must be a complete coincidence.
Maintenance fixed the issue. Just like I said. As far as me harassing YOU, I originally replied to someone else, and you typed a 1500 character dissertation on how I need to think and behave... Move along please. -
06-26-2014, 01:00 PMRaistActually, for many the maintenance did not resolve the problem. Nor did the recent one on the 23rd... notice how the threads keep popping back up?Quote:
Originally Posted by Burrmanchu
And by the way... in case you haven't been following such threads, I bounced my modem last week and I am now getting routed through Atlanta (instead of Charlotte/Raleigh), and I am using Cogent/PsiNet instead of TATA, and my conx is about the best it's ever been. Didn't contact TWC or anything, just my usual routine of resetting my modem when the connection starts to act up... and low and behold everything changed for me.
Just saying... sometimes it actually works, and it only takes minutes to do, so why are you so against the idea?