All North American Data Center Worlds Network Technical Difficulties (Oct. 26)

[HyoMinPark]

I believe someone in the other thread about this (Aether Data Center thread) posted that is was a DDoS attack affecting Level 3, which is a route most ISPs use to try and connect to SE's NA servers. Others posted their trace routes while trying to connect to various NA servers, and it showed issues when trying to connect to Level 3.

You can connect to other data centers because they do not route using Level 3.

I still wish SE had been a little more specific rather than saying "certain ISPs" when it's not really the ISPs' fault, but something this is affecting Level 3--which, unfortunately, is used to connect the the NA servers. I didn't expect them to mention if it was another DDoS attack, since they didn't last week when Dyn was hit and PSN was affected.

[worldofneil]

I'll be expecting some form of compensation for not being able to play the game that cost's money to play every month.

From the User Agreement you agreed to:

5. INTERRUPTION TO THE SERVICE

From time to time, in order to provide our customers with the optimal gaming experience, it is necessary for Square Enix to conduct routine maintenance on the computers and/or servers that support the Game. During these times, access to the Game or support services may be interrupted. Access may also be temporarily suspended in whole or in part, without notice, due to emergency repairs, fire, flood, explosion, war, strike, embargo, governmental action or failure to act, the act of any civil or military authority, act of God, or by any other causes beyond Square Enix's control, or any other reasons for which Square Enix, in its sole discretion, deems that temporary suspension is necessary. You acknowledge that disruptions of service may occur and waive any causes of action against Square Enix in any way arising from or related to any such disruptions of service.

[Kosmos992k]

From the User Agreement you agreed to:

User agreement not withstanding. It is *NOT* in SE's business interests to have 25+% of the NA customer base of FFXIV locked out and mad at SE.

[gothickitty83]

can someone explain abt this "level 3"? i am sorry but kinda new to this and this never happen to me and its my first time that i cant get back in the game for awhile now

[Chadamius]

Found this information on LEVEL 3 outage map ---

http://downdetector.com/status/level3

-----------
Bob Wells
I am at the Empire Central Colo now. All of our services are offline, three different data circuits. The are telling me a ring is down and they are working on it now.
-----------
Sean Seago
A ring...ROFL. It's most of the globe.
------------------
Glen Grayban · University of Houston-Clear Lake
Any Update Bob?
-------------------
Bob Wells
I got tired of waiting and left at 12:45. Just called because they never emailed me a ticket number or status updates. Of course they have no ticket opened, and no updates. The level 3 techs haven't been back in the data center since I first arrived. So, standard level 3 response to the incident basically.
-------------------------
Bob Wells
Just got an email update: 10/27/2016 17:55:40 GMT - Field Services have arrived on site and have shot the fiber. A fault condition has been located 4.1 miles from the testing site. The Outside Plant team has been engaged to assist with the investigations.

10/27/2016 17:33:37 GMT - The ATS team has isolated a possible fiber cut or faulty amplifier in the Level 3 network. Level 3 Field Services are en route to investigate further. The estimated time of arrival is 17:45 GMT.

10/27/2016 16:54:04 GMT - A suspected fiber cut in Houston, TX is impacting IP and Unprotected Transport services. The Transport NOC has engaged Level 3 Field Services and the Level 3 Advanced Technical Support (ATS) team to assist with the investigations. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
-------------------------
Bob Wells
*** CASCADED EXTERNAL NOTES 27-Oct-2016 18:52:18 GMT From CASE: xxxxxxxx - Event
Field Services and the Outside Plant team continue troubleshooting to further isolate the fault condition. The estimated time to repair is not yet known

[ReplicaX]

Level3 isnt the only side being a problem.
When its possible to log in to some of our alts on the same server, but cant log into others, there's an issue on SE's side as well.

Unless you alts are on par with your Main as far as data. They send/receive less packet counts/size. With Level 3 having high latency and packet loss and main characters having larger amounts of data between server/client, you having a much higher chance of disconnection.

Just something to consider. SE does not share its server information. Also if you logged out of a high pop area, that is a lot of info to send/receive as well.

[Kosmos992k]

can someone explain abt this "level 3"? i am sorry but kinda new to this and this never happen to me and its my first time that i cant get back in the game for awhile now

This isn't a perfect explanation, but it should give you some idea of who and what Level 3 are.

Level 3 is one of a very few Internet 'Backbone' companies that provide the major data transmission services that are the foundation of the Internet as it stands now. These kinds of company have fiber optic networks for moving huge quantities of Internet traffic around the globe very efficiently. ISPs - Internet Service Providers - interconnect with companies like Level 3 to ensure that their customer's traffic is efficiently routed across the internet to the desired destination. Very large ISPs like AT&T, ComCast, Charter/Time Warner and others maintain their own backbones that cover part of the service, but they interface with companies like Level 3 to connect their backbone to the rest of the world. These agreements are called peering agreements because both parties agree to operate as peers, and allow traffic from their backbone to run across their peer's backbone infrastructure.

So, if you think of all the customer network links as twigs on branches. The branches are the ISPs, and the large branches and trunk of the tree are the network backbone providers like Level 3. Since more than one branch is affected by this problem, the problem is not specific ISP branches of the tree, the problem is in the trunk of the tree.

[Alahaji]

Some interesting information I found, a friend that's out of country but playing on a NA data center is currently in the game with no problems.

[worldofneil]

I don't agree at all that it's up to the many customers to somehow put pressure on their ISPs to fix this. This is a peripheral issue at most to any of the ISPs involved, most of their customer support people don't know how to reset the modems never mind know what a peering agreement is, who level 3 are and why it's a problem for people playing games on one particular game company's servers.

The customers are the ones giving the ISPs money each month. They have the biggest chance of actually making a change happen.

SE on the other hand has a lot more to gain/lose, it's a big issue for them, and it's a big issue for their partners.

Only if people actually unsubscribe over this and I don't think that many will. The problem will get fixed and brushed under the rug again for another few months. Anyone who researches it will realise it's not Square Enix at fault and it's a Level 3 issue.

I don't know whether SE has any direct contract with Level 3. If they do, then they need to contact their customer agent and kick them hard

I don't know for sure if they have any contact either, but it seems unlikely as they're not directly connected. SE therefore can't dictate policy and the only way they could influence Level 3 would be to throw money at them... where's that money coming from? Subscribers? Why should my subscription costs go to paying for your ISPs bad routing instead of improving the game that I pay for?

SE needs to kick Ormuco hard enough for them to raise the priority with Level 3.

Depends on the agreements Ormuco has with Level 3, but that might be an option. But it also might not be, the traffic is coming FROM Level 3, they're losing/messing it up before it even gets to Ormuco.

End users are essentially powerless in this kind of situation because the impact and importance of the business affected is so dilute on the customer ISP end, that they literally have a financial interest in playing dumb - it would cost them far too much to investigate and 'fix'.

End users SHOULD be the most powerful because you should be able to voice your thoughts by moving ISP if one doesn't work out, but I appreciate that in the US there are lot of monopoly situations so that isn't always viable. That doesn't make it acceptable for the company causing the issues to get away with it though. Why is it acceptable for your ISP (who chose the route) to play dumb and not fix it rather than route you differently?

The only 3 organizations that are in a place to do something about this are, SE, Ormuco, and Level 3. Level 3 as been a problem for years and years, and that's pretty well known. Ormuco - I don'tknow.

Do you have shares in your ISP or something? You seem to acknowledge that they chose your route, so why do they have a get out of jail card for fixing the problem they're causing you?

SE is the business most impacted by this and *have* to do something to protect their own business interests.

Only if their subscribers decrease significantly and chances are that (like every other times this happens), L3 will be working again within a few days and everyone will ignore it again for a few more months.

And I cannot get in on my mobile hotspot (as I feared) because it's an AT&T phone & network.

Will you be complaining to your ISP to see if they will route you differently? I think I already know the answer.



I can't get to any of them, getting to the actual data center lobby server would be an improvement, but it ain't happening. It really doesn't matter whether it's SE's fault or not, they bear the burden of being the service provider of the game, and they have to ensure that this kind of problem either doesn't happen, or get's resolved.[/QUOTE]

[gothickitty83]

This isn't a perfect explanation, but it should give you some idea of who and what Level 3 are.

Level 3 is one of a very few Internet 'Backbone' companies that provide the major data transmission services that are the foundation of the Internet as it stands now. These kinds of company have fiber optic networks for moving huge quantities of Internet traffic around the globe very efficiently. ISPs - Internet Service Providers - interconnect with companies like Level 3 to ensure that their customer's traffic is efficiently routed across the internet to the desired destination. Very large ISPs like AT&T, ComCast, Charter/Time Warner and others maintain their own backbones that cover part of the service, but they interface with companies like Level 3 to connect their backbone to the rest of the world. These agreements are called peering agreements because both parties agree to operate as peers, and allow traffic from their backbone to run across their peer's backbone infrastructure.

So, if you think of all the customer network links as twigs on branches. The branches are the ISPs, and the large branches and trunk of the tree are the network backbone providers like Level 3. Since more than one branch is affected by this problem, the problem is not specific ISP branches of the tree, the problem is in the trunk of the tree.

thanks!!! thats very helpful information so i can undy whats going on so they are going to fix it so we all can get back on?