All North American Data Center Worlds Network Technical Difficulties (Oct. 26)

Quote:

---

Originally Posted by Roth_Trailfinder

I'm certain its NOT a SE problem. Data routing to the other data centers would be, naturally enough, different from the routing to the NA data centers. There's something wrong with the path the data takes between the ISP and the NA data centers.

People have gotten in by using mobile hotspots (which, presumably being a different carrier than their home internet, use a different pathway). People have gotten in using that WTFast software, which does not help PS3/PS4 users. That software finds a different path for the data to travel on, bypassing whatever section is bottlenecking/stopping everyone else.

Its an internet infrastructure problem. Whether or not it was put in place maliciously a few days ago is not something I can speak to, but from what I'm gathering on SE's boards here, its an internet problem, not SE's problem.

---

Indeed...options are a screwed up DNS, DDoS on the DNS or infrastructure of the ISPs, or a lasped peering agreement between the ISPs involved and Level 3 (assuming SE still uses those ...people

It's a great demonstration of how broken the Internet is. Originally (with the original intent of the Internet) this should not have been a problem because the network protocols were designed to route around things like routing problems in particular nodes. Now, we lose service. As much as I hate lag, I'd prefer it if the Net could work as it's supposed to and reroute dynamically to find a new route.

I'm on Charter communications, and started having problems yesterday evening. I've been locked out since about 16 hours ago.

mine is comcast.....still cant get in over like ...more than 5 hours or so

So many of you have been unable to log in for a period of over 10 hours? Is it too early to call this The Great DC disaster of 2016?

I will have to see how my connection is when I get home. Will post if its not resolved by the time I can get online. I am from Brynhildir, my sister is in Siren.

Here is hoping it gets resolved soon!

Over 12 hours and i still can't get in. A dns etc flush didn't help either alas.

UK here:(

I want to play darnit!

Im using a VPN to connect to a canadian VPN and am able to get on, otherwise I haven't been able to. I have Charter

Well, it seems that the message on the Lodestone is around 6 hours old now, so perhaps we could get an update from SE to help us understand what is going on and whether we should all be yelling at our ISPs or not...?

I , like most others here cannot connect to the servers. I downloaded Pingzapper and I can connect. I dont know why this would make a difference, however it is working (so far) My ISP is Charter, I I was disconnected last night like 10 PM CST.

The pingzapper thing has a free 7 days, I am not usually one to advertise an app, but its working for me, hopefully things are back to normal soon and I wont need this, but in the interim its helping (I am on PC)

It's because your traffic is now being routed differently, going through their servers first, and avoiding whatever blockage is going on.

well i am on ps4 and theres no way i can change different connection and etc cuz my wifi work just fine, just that game kept saying that server couldnt connect and all =<

Quote:

---

Originally Posted by Friggan

I , like most others here cannot connect to the servers. I downloaded Pingzapper and I can connect. I dont know why this would make a difference, however it is working (so far) My ISP is Charter, I I was disconnected last night like 10 PM CST.

The pingzapper thing has a free 7 days, I am not usually one to advertise an app, but its working for me, hopefully things are back to normal soon and I wont need this, but in the interim its helping (I am on PC)

---

PS4 here, and I am about to break out the mobile hotspot.

ISP's fault, SE's fault...

It's not affecting anything but my ability to use a product that SE provides me in exchange for money up front.

It's affecting every NA data center and world.

Sounds like SE's obligation to provide us with some answers, whether it's their fault or not. I don't think anyone cares whose fault it is, just get it fixed.

I don't see this being reported yet in any of the gaming press that Google New aggregates....

It's very strange that traffic isn't being routed around this.

Quote:

---

Originally Posted by PerrineClostermann

It's very strange that traffic isn't being routed around this.

---

That's why I am suspecting a peering agreement issue since clients from ISPs affected would not be able to route around it since Level 3 is the last leg to the servers in the NA Data Center. By using a VPN you circumvent that, as you do using a hotspot as long as that mobile provide is not also affected of course.

we not going fix thing by get upset we just got be patients and check back every hour see if we connected

i use Charter my self

Got kicked off around 0030-0100 EST on the final boss on a WC run that wasn't doing well in the first place :( really hammered the nail in the coffin... still not working. Pretty frustrated about the whole situation.

My home server is Famfrit... but I can log in to another server like Leviathan... doesn't make any sense :(

I use Spectrum. Been out of commission since this started last night. Would love some answers.

Quote:

---

Originally Posted by PerrineClostermann

It's very strange that traffic isn't being routed around this.

---

The ISPs would have to do that and for whatever reason(s) they're not. Perhaps the right people at the ISPs aren't aware of the issue or perhaps they just don't care because they'll lose money by routing via someone else and it's easier for them to just wait it out until Level 3 are working again.

Quote:

---

Originally Posted by Kosmos992k

Level 3 is the last leg to the servers in the NA Data Center

---

It's not. It's *a* last leg, but it's not the only last leg. My ISP (thankfully) never routes me through Level3 and while I appreciate that doesn't help anyone with a problem and for that I apologise... but this comes up every few months and the only way it's going to be fixed long term is moving ISPs to someone who doesn't route via Level 3 or customers somehow putting enough pressure on those ISPs to change their routes.

But if you're on PC and you're tired of waiting for the problem to resolve itself, use a VPN

Go to WTFast.com and download it. It is SO simple, a monkey to do it. It does not require a credit card or any personal information besides your email for the account you make.

It is a trial, not sure how long it lasts but damnit, it logged me in for today and that's all that counts.

tracert to Behemoth.

Here's the problem;

Quote:

---

8 38 ms 30 ms 30 ms bbr01spbgsc-bue-4.spbg.sc.charter.com [96.34.2.50]
9 35 ms 36 ms 32 ms bbr01gnvlsc-bue-800.gnvl.sc.charter.com [96.34.0.134]
10 * * * Request timed out.
11 300 ms 60 ms 60 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
12 61 ms 60 ms 61 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
13 * * * Request timed out.
14 81 ms 66 ms 60 ms 192.34.76.2

---

Hop 10 is the one from Charter to Level 3. Hop 13 is level 3 to Ormuco. Hop 14 is an Ormuco server, and then hops after that are SE servers.

This tells me that Level 3 is not playing nice with anyone since ISPs and Ormuco are both seeing requests time out on the hops to/from Level 3.

Isn't it time that SE fired Level 3 for being crap?

This happened to me last night, was hoping it'd be resolved by the time I got back from work today, but apparently not. Hope I don't end up having to raid on my phone tonight.

Hopefully my character still being logged in (as I'm guessing it is, it was logged in for three hours at least) doesn't cause issues with logging in from a mobile connection.

Quote:

---

Originally Posted by worldofneil

It's not. It's *a* last leg, but it's not the only last leg. My ISP (thankfully) never routes me through Level3 and while I appreciate that doesn't help anyone with a problem and for that I apologise... but this comes up every few months and the only way it's going to be fixed long term is moving ISPs to someone who doesn't route via Level 3 or customers somehow putting enough pressure on those ISPs to change their routes.

---

That's just not an option, look at the ISPs involved. Comcast, Charter, Time Warner, AT&T. They cover what, 95% of US ISP customers? Changing ISPs isn't gonna happen, this is a Level 3 issue and they need to be fired and replaced. I understand that they might not be the last hop for some, but they are for most in NA (at least east of the Rockies).

The trial for https://www.wtfast.com/# lasts for two weeks which is more than enough time to get the issues sorted out

Oh I don't care, I'm moving to some JP server.
If that is the kind of quality NA servers are subjected to, then I have no reason to be there anymore, my subscription time is rolling and I can't play in NA servers, but JP servers work. Good riddance 'murica.

Quote:

---

Originally Posted by Kosmos992k

Isn't it time that SE fired Level 3 for being crap?

---

SE didn't choose to route via Level 3 to get to your destination (the Square Enix servers)... your ISP did. Square Enix have nothing at all to do with Level 3 so cannot fire them.

Quote:

---

Originally Posted by Kosmos992k

That's just not an option, look at the ISPs involved. Comcast, Charter, Time Warner, AT&T. They cover what, 95% of US ISP customers? Changing ISPs isn't gonna happen, this is a Level 3 issue and they need to be fired and replaced. I understand that they might not be the last hop for some, but they are for most in NA (at least east of the Rockies).

---

I do appreciate that and agree they need to be replaced, but you're leaning on the wrong company. The ISPs in question need to stop using Level 3 and use different routing, but they probably have an agreement to use Level 3 which saves them money over other routes. So long term they won't want to route elsewhere as they would lose money and I'm sure no customer wants their Internet price to rise to compensate... :(

Level3 isnt the only side being a problem.
When its possible to log in to some of our alts on the same server, but cant log into others, there's an issue on SE's side as well.

got kick off game around 12:30 am last night in middle of castrum meridianum been try all day get back in just now checked I can finally get into primal data center but I can't log into my main but can log into my alt on leviathan keep get code 10104 or 90002

All you people that are talking about just moving Data Center's are missing the point. Who the hell cares if you can't get to your static tonight, good chance at least half of your static isn't making it tonight. Also, WTFast doesn't help the console player base. This is Final Fantasy 14. Not Archeage, not TERA. This is a REQUIRED subscription game, it's not "Oh, I paid for benefits so a few days won't hurt." This is, "Server's aren't working, I'm not receiving the service I'm paying for."

Our Sub-Timers are ticking away as at least 25% of the player base can't log in, myself included. I'll be expecting some form of compensation for not being able to play the game that cost's money to play every month.

Quote:

---

Originally Posted by AchilliesFroste

Our Sub-Timers are ticking away as at least 25% of the player base can't log in, myself included. I'll be expecting some form of compensation for not being able to play the game that cost's money to play every month.

---

Good luck having Level 3 compensate your loss.

Quote:

---

Originally Posted by worldofneil

SE didn't choose to route via Level 3 to get to your destination (the Square Enix servers)... your ISP did. Square Enix have nothing at all to do with Level 3 so cannot fire them.



I do appreciate that and agree they need to be replaced, but you're leaning on the wrong company. The ISPs in question need to stop using Level 3 and use different routing, but they probably have an agreement to use Level 3 which saves them money over other routes. So long term they won't want to route elsewhere as they would lose money and I'm sure no customer wants their Internet price to rise to compensate... :(

---

Level 3 has been a problem with FFXIV since the inception of ARR and probably before. I don't agree at all that it's up to the many customers to somehow put pressure on their ISPs to fix this. This is a peripheral issue at most to any of the ISPs involved, most of their customer support people don't know how to reset the modems never mind know what a peering agreement is, who level 3 are and why it's a problem for people playing games on one particular game company's servers. SE on the other hand has a lot more to gain/lose, it's a big issue for them, and it's a big issue for their partners. I don't know whether SE has any direct contract with Level 3. If they do, then they need to contact their customer agent and kick them hard, If not, and it's indirected through Ormuco, then SE needs to kick Ormuco hard enough for them to raise the priority with Level 3. End users are essentially powerless in this kind of situation because the impact and importance of the business affected is so dilute on the customer ISP end, that they literally have a financial interest in playing dumb - it would cost them far too much to investigate and 'fix'. The only 3 organizations that are in a place to do something about this are, SE, Ormuco, and Level 3. Level 3 as been a problem for years and years, and that's pretty well known. Ormuco - I don'tknow. SE is the business most impacted by this and *have* to do something to protect their own business interests.

And I cannot get in on my mobile hotspot (as I feared) because it's an AT&T phone & network.

Quote:

---

Originally Posted by Claire_Pendragon

Level3 isnt the only side being a problem.
When its possible to log in to some of our alts on the same server, but cant log into others, there's an issue on SE's side as well.

---

I can't get to any of them, getting to the actual data center lobby server would be an improvement, but it ain't happening. It really doesn't matter whether it's SE's fault or not, they bear the burden of being the service provider of the game, and they have to ensure that this kind of problem either doesn't happen, or get's resolved.

I believe someone in the other thread about this (Aether Data Center thread) posted that is was a DDoS attack affecting Level 3, which is a route most ISPs use to try and connect to SE's NA servers. Others posted their trace routes while trying to connect to various NA servers, and it showed issues when trying to connect to Level 3.

You can connect to other data centers because they do not route using Level 3.

I still wish SE had been a little more specific rather than saying "certain ISPs" when it's not really the ISPs' fault, but something this is affecting Level 3--which, unfortunately, is used to connect the the NA servers. I didn't expect them to mention if it was another DDoS attack, since they didn't last week when Dyn was hit and PSN was affected.

Quote:

---

Originally Posted by AchilliesFroste

I'll be expecting some form of compensation for not being able to play the game that cost's money to play every month.

---

From the User Agreement you agreed to:

Quote:

---

5. INTERRUPTION TO THE SERVICE

From time to time, in order to provide our customers with the optimal gaming experience, it is necessary for Square Enix to conduct routine maintenance on the computers and/or servers that support the Game. During these times, access to the Game or support services may be interrupted. Access may also be temporarily suspended in whole or in part, without notice, due to emergency repairs, fire, flood, explosion, war, strike, embargo, governmental action or failure to act, the act of any civil or military authority, act of God, or by any other causes beyond Square Enix's control, or any other reasons for which Square Enix, in its sole discretion, deems that temporary suspension is necessary. You acknowledge that disruptions of service may occur and waive any causes of action against Square Enix in any way arising from or related to any such disruptions of service.

---

Quote:

---

Originally Posted by worldofneil

From the User Agreement you agreed to:

---

User agreement not withstanding. It is *NOT* in SE's business interests to have 25+% of the NA customer base of FFXIV locked out and mad at SE.

can someone explain abt this "level 3"? i am sorry but kinda new to this and this never happen to me and its my first time that i cant get back in the game for awhile now

Found this information on LEVEL 3 outage map ---

http://downdetector.com/status/level3

-----------
Bob Wells
I am at the Empire Central Colo now. All of our services are offline, three different data circuits. The are telling me a ring is down and they are working on it now.
-----------
Sean Seago
A ring...ROFL. It's most of the globe.
------------------
Glen Grayban · University of Houston-Clear Lake
Any Update Bob?
-------------------
Bob Wells
I got tired of waiting and left at 12:45. Just called because they never emailed me a ticket number or status updates. Of course they have no ticket opened, and no updates. The level 3 techs haven't been back in the data center since I first arrived. So, standard level 3 response to the incident basically.
-------------------------
Bob Wells
Just got an email update: 10/27/2016 17:55:40 GMT - Field Services have arrived on site and have shot the fiber. A fault condition has been located 4.1 miles from the testing site. The Outside Plant team has been engaged to assist with the investigations.

10/27/2016 17:33:37 GMT - The ATS team has isolated a possible fiber cut or faulty amplifier in the Level 3 network. Level 3 Field Services are en route to investigate further. The estimated time of arrival is 17:45 GMT.

10/27/2016 16:54:04 GMT - A suspected fiber cut in Houston, TX is impacting IP and Unprotected Transport services. The Transport NOC has engaged Level 3 Field Services and the Level 3 Advanced Technical Support (ATS) team to assist with the investigations. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
-------------------------
Bob Wells
*** CASCADED EXTERNAL NOTES 27-Oct-2016 18:52:18 GMT From CASE: xxxxxxxx - Event
Field Services and the Outside Plant team continue troubleshooting to further isolate the fault condition. The estimated time to repair is not yet known

Quote:

---

Originally Posted by Claire_Pendragon

Level3 isnt the only side being a problem.
When its possible to log in to some of our alts on the same server, but cant log into others, there's an issue on SE's side as well.

---

Unless you alts are on par with your Main as far as data. They send/receive less packet counts/size. With Level 3 having high latency and packet loss and main characters having larger amounts of data between server/client, you having a much higher chance of disconnection.

Just something to consider. SE does not share its server information. Also if you logged out of a high pop area, that is a lot of info to send/receive as well.

Quote:

---

Originally Posted by gothickitty83

can someone explain abt this "level 3"? i am sorry but kinda new to this and this never happen to me and its my first time that i cant get back in the game for awhile now

---

This isn't a perfect explanation, but it should give you some idea of who and what Level 3 are.

Level 3 is one of a very few Internet 'Backbone' companies that provide the major data transmission services that are the foundation of the Internet as it stands now. These kinds of company have fiber optic networks for moving huge quantities of Internet traffic around the globe very efficiently. ISPs - Internet Service Providers - interconnect with companies like Level 3 to ensure that their customer's traffic is efficiently routed across the internet to the desired destination. Very large ISPs like AT&T, ComCast, Charter/Time Warner and others maintain their own backbones that cover part of the service, but they interface with companies like Level 3 to connect their backbone to the rest of the world. These agreements are called peering agreements because both parties agree to operate as peers, and allow traffic from their backbone to run across their peer's backbone infrastructure.

So, if you think of all the customer network links as twigs on branches. The branches are the ISPs, and the large branches and trunk of the tree are the network backbone providers like Level 3. Since more than one branch is affected by this problem, the problem is not specific ISP branches of the tree, the problem is in the trunk of the tree.

Some interesting information I found, a friend that's out of country but playing on a NA data center is currently in the game with no problems.

Quote:

---

Originally Posted by Kosmos992k

I don't agree at all that it's up to the many customers to somehow put pressure on their ISPs to fix this. This is a peripheral issue at most to any of the ISPs involved, most of their customer support people don't know how to reset the modems never mind know what a peering agreement is, who level 3 are and why it's a problem for people playing games on one particular game company's servers.

---

The customers are the ones giving the ISPs money each month. They have the biggest chance of actually making a change happen.

Quote:

---

Originally Posted by Kosmos992k

SE on the other hand has a lot more to gain/lose, it's a big issue for them, and it's a big issue for their partners.

---

Only if people actually unsubscribe over this and I don't think that many will. The problem will get fixed and brushed under the rug again for another few months. Anyone who researches it will realise it's not Square Enix at fault and it's a Level 3 issue.

Quote:

---

Originally Posted by Kosmos992k

I don't know whether SE has any direct contract with Level 3. If they do, then they need to contact their customer agent and kick them hard

---

I don't know for sure if they have any contact either, but it seems unlikely as they're not directly connected. SE therefore can't dictate policy and the only way they could influence Level 3 would be to throw money at them... where's that money coming from? Subscribers? Why should my subscription costs go to paying for your ISPs bad routing instead of improving the game that I pay for?

Quote:

---

Originally Posted by Kosmos992k

SE needs to kick Ormuco hard enough for them to raise the priority with Level 3.

---

Depends on the agreements Ormuco has with Level 3, but that might be an option. But it also might not be, the traffic is coming FROM Level 3, they're losing/messing it up before it even gets to Ormuco.

Quote:

---

Originally Posted by Kosmos992k

End users are essentially powerless in this kind of situation because the impact and importance of the business affected is so dilute on the customer ISP end, that they literally have a financial interest in playing dumb - it would cost them far too much to investigate and 'fix'.

---

End users SHOULD be the most powerful because you should be able to voice your thoughts by moving ISP if one doesn't work out, but I appreciate that in the US there are lot of monopoly situations so that isn't always viable. That doesn't make it acceptable for the company causing the issues to get away with it though. Why is it acceptable for your ISP (who chose the route) to play dumb and not fix it rather than route you differently?

Quote:

---

Originally Posted by Kosmos992k

The only 3 organizations that are in a place to do something about this are, SE, Ormuco, and Level 3. Level 3 as been a problem for years and years, and that's pretty well known. Ormuco - I don'tknow.

---

Do you have shares in your ISP or something? You seem to acknowledge that they chose your route, so why do they have a get out of jail card for fixing the problem they're causing you?

Quote:

---

Originally Posted by Kosmos992k

SE is the business most impacted by this and *have* to do something to protect their own business interests.

---

Only if their subscribers decrease significantly and chances are that (like every other times this happens), L3 will be working again within a few days and everyone will ignore it again for a few more months.

Quote:

---

Originally Posted by Kosmos992k

And I cannot get in on my mobile hotspot (as I feared) because it's an AT&T phone & network.

---

Will you be complaining to your ISP to see if they will route you differently? I think I already know the answer.



I can't get to any of them, getting to the actual data center lobby server would be an improvement, but it ain't happening. It really doesn't matter whether it's SE's fault or not, they bear the burden of being the service provider of the game, and they have to ensure that this kind of problem either doesn't happen, or get's resolved.[/QUOTE]

Quote:

---

Originally Posted by Kosmos992k

This isn't a perfect explanation, but it should give you some idea of who and what Level 3 are.

Level 3 is one of a very few Internet 'Backbone' companies that provide the major data transmission services that are the foundation of the Internet as it stands now. These kinds of company have fiber optic networks for moving huge quantities of Internet traffic around the globe very efficiently. ISPs - Internet Service Providers - interconnect with companies like Level 3 to ensure that their customer's traffic is efficiently routed across the internet to the desired destination. Very large ISPs like AT&T, ComCast, Charter/Time Warner and others maintain their own backbones that cover part of the service, but they interface with companies like Level 3 to connect their backbone to the rest of the world. These agreements are called peering agreements because both parties agree to operate as peers, and allow traffic from their backbone to run across their peer's backbone infrastructure.

So, if you think of all the customer network links as twigs on branches. The branches are the ISPs, and the large branches and trunk of the tree are the network backbone providers like Level 3. Since more than one branch is affected by this problem, the problem is not specific ISP branches of the tree, the problem is in the trunk of the tree.

---

thanks!!! thats very helpful information so i can undy whats going on so they are going to fix it so we all can get back on?