All North American Data Center Worlds Network Technical Difficulties (Oct. 26)

[Shimada]

I'm hoping to see an update on the situation or for this to be resolved by the time I get home from work. Here's to wishful thinking~

[Katnip]

I do not believe this is an ISP problem at all. I have my main character, and 7 mules all on the same server. But ever time I try and log into my main character I get either 90002, 90006, or 10105. BUT, if I log in with any of my 7 mules from the same computer, to the same server, I can log in just fine, and I do not get kicked. So tell me; If I can make a connection on 7 out of 8 of my characters. How is that an ISP issue and NOT something on SE's side of things.

[wdeschain]

Still nothing from anyone...hmmm? I wonder. Does anyone know of other services affected by "whatever" the issue is at hand?

Certainly a failure of some technical sort or the fallout from last week's attacks (possibly) would not simply be isolated to Eorzeans. Wouldn't you think?

[Laili]

I REALLY hope I don't lose my garden as a result of this.

[HyoMinPark]

Still nothing from anyone...hmmm? I wonder. Does anyone know of other services affected by "whatever" the issue is at hand?

Certainly a failure of some technical sort or the fallout from last week's attacks (possibly) would not simply be isolated to Eorzeans. Wouldn't you think?

There's a report of technical difficulties on Level 3's side right now. So I'm sure if other services utilize Level 3 anywhere in their routing, they would probably encounter issues as well. According to Level 3's Down Detector page, there are other services having the issues--various comments mentioning phones being down, etc.

[Velisra]

I can do everything else I need to do on the Internet without any issues, including playing WoW, but still can't connect to FFXIV on any but one character on my account. That character is a level 1 that is still in the instanced starting quests. Of course, I'm not going to play that character as it was an experiment with another race anyway, so there it is.

I don't know what to think about it all. This is a bit crazy and more than a bit disappointing. Maybe I could download that other thing from WTFast.com, but why? I'm not going to bother with that because I shouldn't have to load some special program on my computer just to get to the servers to play this game.

[wdeschain]

http://downdetector.com/status/level3

Bob Wells
I am at the Empire Central Colo now. All of our services are offline, three different data circuits. The are telling me a ring is down and they are working on it now.
-----------
Sean Seago
A ring...ROFL. It's most of the globe.
------------------
Glen Grayban · University of Houston-Clear Lake
Any Update Bob?
-------------------
Bob Wells
I got tired of waiting and left at 12:45. Just called because they never emailed me a ticket number or status updates. Of course they have no ticket opened, and no updates. The level 3 techs haven't been back in the data center since I first arrived. So, standard level 3 response to the incident basically.
-------------------------
Bob Wells
Just got an email update: 10/27/2016 17:55:40 GMT - Field Services have arrived on site and have shot the fiber. A fault condition has been located 4.1 miles from the testing site. The Outside Plant team has been engaged to assist with the investigations.

10/27/2016 17:33:37 GMT - The ATS team has isolated a possible fiber cut or faulty amplifier in the Level 3 network. Level 3 Field Services are en route to investigate further. The estimated time of arrival is 17:45 GMT.

10/27/2016 16:54:04 GMT - A suspected fiber cut in Houston, TX is impacting IP and Unprotected Transport services. The Transport NOC has engaged Level 3 Field Services and the Level 3 Advanced Technical Support (ATS) team to assist with the investigations. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
-------------------------
Bob Wells
*** CASCADED EXTERNAL NOTES 27-Oct-2016 18:52:18 GMT From CASE: xxxxxxxx - Event
Field Services and the Outside Plant team continue troubleshooting to further isolate the fault condition. The estimated time to repair is not yet known

LOL! Bob Wells. Bwaaaaahahahahahaha

[Silverbranch]

Ok. Well, this explains why I get to a blank Char Select screen, then after several seconds a DC complaining about unable to log in to the "Lobby Server".

Bleh.

[JadenStarr]

I spoke to Level 3 and can shed a little light but first why is it i can connect to all websites plus league of legends and play fine. But when it comes to SE i cant do a damn thing. That sounds like a SE issue to me, now i dont know a lot about connections but it only makes sense. Now i pretended to be a comcast manager contacting Level 3 about the broken fiber in houston causeing issues and he stated that they have yet to find it and are currently trying to locate it right now. Can take up to 3-4 hours if it is a aerial cable and up to a day if it is buried. So my guess is we will be waiting till tomorrow night till we can play again. Rest up

[BurnzieThornton]

It looks to me like the problem could be resolved by Ormuco, who is a fiber provider and looks to be the cloud provider for the Aether NA server cluster in Montreal.

In theory, Ormuco could solve the issue by shifting traffic away from their Level3 transit and towards another transit peer (Cogent, Verizon, or Tata America just from a quick glance).

I would assume Eidos would either directly handle this traffic engineering exercise directly with the proper communities between them and Ormuco, or have Ormuco manually implement a simple traffic engineering solution for Eidos' 199.91.189.0/24 allocation at a minimum.

Regardless, if some people can play consistently on the Aether cluster then this is a network issue that should be resolved by Eidos and their immediate upstream Ormuco.

Putting this all on a Level3 fiber cut in Houston is disingenuous, as given adequate capacity they could cut them out of the equation entirely for the duration of the issue (and 14 hours deep into an issue like this is somewhat unforgivable from an operations standpoint of operating a content network with multiple upstreams).