HUGE CONNECTION PROBLEM

[Raist]

That is why you don't use ping as your only diagnostic tool. Run traces to the actual IP your client uses (can get that from resmon or netstat while logged fully into the game). Forward those traces to your ISP's Tier3 techs so that they can investigate the specific route you are assigned for just this game's connection. When you call in and they look at your modem and such, they are just looking at your local segment and confirming connectivity to your localized nodes. You can have anywhere from 2 to 63 potential points of failure between your system and SE's servers. They need to see your entire path to pinpoint exactly which hops are having trouble. Typically it is either ingress or congestion past your local tap or issues with peering exchanges in your route.

In other words, it's more often a problem in another state or country between your local connection and SE's servers that only your ISP or SE's ISP has the resources and enforceable contracts to get it addressed. But, in order to get them off their duffs and actually do something, you need to push for escalation to Tier3 (not the teir1/2 departments you deal with closer to home) and get them useful details demonstrating the particular characteristics of this game's specific connection to the Canadian IP's that start with 199. The easiest way to do this is to submit a simple report like a tracert.

C:\Windows\System32>tracert 199.91.189.25

Tracing route to 199.91.189.25 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    91 ms    19 ms    19 ms  cpe-066-026-112-001.sc.res.rr.com [66.26.112.1]
  3    33 ms    31 ms    18 ms  cpe-024-031-198-009.sc.res.rr.com [24.31.198.9]
  4    23 ms    23 ms    24 ms  24.93.64.134
  5    29 ms    27 ms    27 ms  bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
  6    26 ms    25 ms    24 ms  107.14.19.99
  7    28 ms    24 ms    36 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
  8    26 ms    28 ms    28 ms  be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165]
  9    36 ms    37 ms    38 ms  be2168.ccr21.dca01.atlas.cogentco.com [154.54.31.94]
 10    43 ms    44 ms    44 ms  be2148.ccr21.jfk02.atlas.cogentco.com [154.54.31.118]
 11    48 ms    47 ms    47 ms  be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50]
 12    69 ms    70 ms    70 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 13    62 ms    64 ms    63 ms  38.122.42.34
 14    58 ms    47 ms    47 ms  10.2.2.1
 15    48 ms    47 ms    47 ms  192.34.76.2
 16    48 ms    48 ms    47 ms  199.91.189.234
 17    49 ms    47 ms    48 ms  199.91.189.25

Trace complete.

Notice that 2nd hop in my current trace. This is the last piece of the puzzle that TWC has been trying to address for us in our area. We've been working with them about thier peering issues for quite some time now, and have been switching up our routes about every 2-3 weeks. I've constantly bounce back and forth amongst Level3, TATA, and Congent for the routing partner that gets us into and across Canada. It will be good for a couple weeks, then get congested and they switch us to another partner. That is being managed by Time Warner Cable, not SE nor their ISP Ormuco... it's my TWC Teir3/4 people doing this for me.

The lag has gone from a typical ~120ms down to staying under ~70ms as a result of it, and for the most part I have no troublesome lag spikes until that gateway at that second hop goes haywire. That is a hop to Hilton Head, SC. We are in Florence... well over 100 miles north of there. This is a temporary routing change while they are trying to address issues with the gateway that serves Florence residential service. Our local gateway was severely overloaded and frequently breaking the 200ms response time. During primetime it could break 3 seconds and time out on us. This was wreaking havoc on gaming.

Again... this is the result of bringing in my ISP's Tier3 techs and providing them relevant data about my specific connection to the SE server I play one (more specifically, tracerts). They sent techs out to inspect my amp, tap, cables, modem... took signal and log dumps, made sure everything was good here. Sent bucket trucks up and down the street to address ingress they found upstream. Fixed problems preventing the CMTS from bonding more than 4 channels--all in an effort to thin out the noise and congestion that was found locally. But the problem STILL persisted, so they sent it further up the chain to try to address it with their peering partners. They keep switching us to different Tier1 ISP's (Level3, TATA, Cogent) trying to find the optimal exchange for us. This last one appears to be an attempt at a new peering agreement with close ties to Ormuco (SE's ISP). It is a route managed in part by PSInet, who has peering agreements with Ormuco. Since this path was established, I have started seeing response time stabilize under 70ms for me (provided my gateway isn't getting overloaded, but even then it is under 100ms typically).

It's the result of identifying the local and regional issues cropping up in route, and getting the right people to investigate/address the problems that were found. And it wasn't SE... but the ISP's involved in getting me to the servers in Montreal. To further illustrate how this is a regional issue between us, I will paste in some connectivity maps that refresh periodically throughout the day for several services that can be found at http://downdetector.com/companies. They track reporting feeds for various services and even some ISP's, and plot the reporting on these maps. They also log the stats and posts for some of them on their site as well. These may seem familiar to some who have been following this, as they have been posted in other threads on the issue.

These snapshots update periodically so may change as the thread is refreshed. They are based on source data from here:
http://downdetector.com/companies

The service specific source pages are linked below each image.

TWC map:


http://downdetector.com/status/time-warner-cable


AT&T map:


http://downdetector.com/status/att


Netflix map:


http://downdetector.com/status/netflix


PSN map:


http://downdetector.com/status/playstation-network


XBL map:


http://downdetector.com/status/xbox-live

Notice how the problems seem to clump up in the same regions across different services? That is because there are issues with the exchanges for the Tier1 providers. Notice specifically the clusterf*ck that is the north-east--up around NY and MA are the primary entry points to Canada for the majority of the US. The big 3 providers we use to get in there have a combined total of 7 exchanges up there for getting into Ontario and Quebec, and only 2 above Seattle, Washington for getting into British Columbia. So, when things are a mess in the north-east of the US, a LOT of us suffer getting into/across Canada. The same can happen for the EU as well, because a lot of them have to come in through Nova Scotia. Then our poor Aussie friends are often stuck coming around Southern California and have to either shoot up around Seattle/Vancouver and come across Canada through Toronto or get shifted across the US to come through the North East corridor with the rest of us.

The NA infrastructure is just a flat out mess in general. Both Level3 and TATA have been in a major pissing contest with our last mile providers (AT&T, Verizon, Comcast, TWC, etc.) to update their peering agreements to compensate for the massive increase in usage. They've even filed formal complaints with the commission to try to force them to operate under fair practices, but have not been able to make any reasonable progress. Everyone is streaming more and more (and cutting the cable/satellite cords), among other things like shopping and social media, putting greater and greater strain on the backbones that connect all the separate networks. But our ISP's have been reluctant to pony up the resources to widen the pipes to thin out the massive port congestion at the exchanges between their networks and those backbones.

If you take some time to track this stuff in general, you can see the ebb and flow of this problem all over the place. This is a handy app for looking at various industries and regions in real time. They have tools for mapping latency, threat levels, and usage by industry... all consolidated in one app with drop boxes and tabs to switch between the different reports.
http://www.akamai.com/html/technolog...try/index.html

Basically, at specific times of day, you may see the same areas line up with problems across multiple services because it is due to regional problems amongst the ISP's... it isn't JUST SE in play here. If so, then it would be affecting the vast majority of players all at once--you would see entire servers/clusters go down and not just groups of users from specific servers. Not to say the SE doesn't have internal stability issues that they need to address, but it is more often people getting routed along problematic paths to the server that are getting affected together (ie, we've seen everyone using level3 get hammered at once, while Cogent users were fine). It is more because we have a systemic problem with our infrastructure that affects critical regions all at once. When the transit is down, it affects everyone that is getting pushed through that corridor. That is why sometimes you see multiple games/services affected in kind for some and not for others---there is a bigger regional transit problem with our providers and their partners.

Edit:
To further support this, here is an example of the issue in action with my connection. During the final cut scene for my 50 BRD fight tonight, I got booted and suffered the same issues trying to log back in (connection lost, character already logged in, connection to lobby lost, failed to connect to lobby, etc.), so I pulled up a DOS window and started running traces to my lobby and game server. The portion of my path where I slingshot DC and Chicago (NSA scan and such) was stalling out. As soon as it cleared, I logged straight into the game...the stalling and disconnect was happening between TWC and Cogentco--hadn't even gotten to Ormuco, much less SE. There was nothing SE could do to remedy this until Cogent fixed it or TWC routed me away from the bad segments. Unfortunately I had to do that annoying fight again, but was able to get right back into the game as soon as the problem corrected itself in my route. Note that I've been logged in all day non-stop since shortly after initially posting here... roughly 12 hours straight (took breaks, queuing for duties and such--but short of this 10 minute hiccup, my character stayed logged into the game.)

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Windows\System32>tracert 199.91.189.74

Tracing route to 199.91.189.74 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    49 ms    29 ms    40 ms  cpe-066-026-112-001.sc.res.rr.com [66.26.112.1]
  3    19 ms    31 ms    18 ms  cpe-024-031-198-009.sc.res.rr.com [24.31.198.9]
  4    22 ms    23 ms    23 ms  24.93.64.134
  5    44 ms    27 ms    43 ms  bu-ether44.atlngamq46w-bcr00.tbone.rr.com [107.14.19.46]
  6    29 ms    25 ms    25 ms  ae-1-0.pr0.atl20.tbone.rr.com [66.109.6.177]
  7    27 ms    41 ms    25 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
  8    27 ms    25 ms    45 ms  be2051.ccr42.atl01.atlas.cogentco.com [154.54.0.161]
  9    36 ms    51 ms    59 ms  be2171.mpd22.dca01.atlas.cogentco.com [154.54.31.110]
 10    43 ms    54 ms    44 ms  be2151.mpd22.jfk02.atlas.cogentco.com [154.54.40.74]
 11    48 ms    48 ms    48 ms  be2109.ccr21.alb02.atlas.cogentco.com [154.54.3.138]
 12    70 ms    73 ms    66 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 13    88 ms    70 ms    77 ms  38.122.42.34
 14    50 ms    47 ms    63 ms  10.2.2.1
 15    48 ms    57 ms    83 ms  192.34.76.2
 16    50 ms    47 ms    48 ms  199.91.189.234
 17    73 ms    50 ms    47 ms  199.91.189.74

Trace complete.

C:\Windows\System32>tracert -d 199.91.189.25

Tracing route to 199.91.189.25 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  10.10.100.1
  2    44 ms    36 ms    38 ms  66.26.112.1
  3    17 ms    18 ms    15 ms  24.31.198.9
  4    37 ms    23 ms    23 ms  24.93.64.134
  5    27 ms    30 ms    28 ms  107.14.19.48
  6    25 ms    24 ms    25 ms  107.14.19.99
  7    25 ms    28 ms    25 ms  154.54.12.109
  8    28 ms    32 ms    27 ms  154.54.0.165
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *     ^C
C:\Windows\System32>tracertd 199.91.189.25
'tracertd' is not recognized as an internal or external command,
operable program or batch file.

C:\Windows\System32>tracert 199.91.189.25

Tracing route to 199.91.189.25 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    56 ms    34 ms    38 ms  cpe-066-026-112-001.sc.res.rr.com [66.26.112.1]
  3    18 ms    19 ms    18 ms  cpe-024-031-198-009.sc.res.rr.com [24.31.198.9]
  4    25 ms    27 ms    24 ms  24.93.64.134
  5    25 ms    27 ms    31 ms  bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
  6    30 ms    25 ms    25 ms  107.14.19.99
  7    25 ms    25 ms    25 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
  8    42 ms    32 ms    28 ms  be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165]
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13    68 ms    84 ms    70 ms  38.122.42.34
 14    48 ms    49 ms    62 ms  10.2.2.1
 15    51 ms    47 ms    48 ms  192.34.76.2
 16    47 ms    47 ms    47 ms  199.91.189.234
 17    49 ms    50 ms    47 ms  199.91.189.25

Trace complete.

C:\Windows\System32>tracert 199.91.189.25

Tracing route to 199.91.189.25 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  LPTSRV [10.10.100.1]
  2   870 ms    35 ms    68 ms  cpe-066-026-112-001.sc.res.rr.com [66.26.112.1]
  3    16 ms    18 ms    18 ms  cpe-024-031-198-009.sc.res.rr.com [24.31.198.9]
  4    24 ms    23 ms    23 ms  24.93.64.134
  5    29 ms    27 ms    27 ms  bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
  6    27 ms    34 ms    25 ms  107.14.19.99
  7    27 ms    25 ms    25 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
  8    27 ms    28 ms    28 ms  be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165]
  9    37 ms    47 ms    38 ms  be2168.ccr21.dca01.atlas.cogentco.com [154.54.31.94]
 10    43 ms    44 ms    44 ms  be2148.ccr21.jfk02.atlas.cogentco.com [154.54.31.118]
 11    49 ms    46 ms    47 ms  be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50]
 12    69 ms    70 ms    70 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 13    73 ms    75 ms    75 ms  38.122.42.34
 14    50 ms    48 ms    50 ms  10.2.2.1
 15    51 ms    47 ms    47 ms  192.34.76.2
 16    49 ms    47 ms    47 ms  199.91.189.234
 17    48 ms    47 ms    47 ms  199.91.189.25

Trace complete.

C:\Windows\System32>

[Esham]

Im having this issue as well. Lucky for me i work for my ISP so i know that as a single customer with very little pull (and even as an employee) i will not be able to force my ISP or Square Enix to get their routes in order.

What raist says is great and all but its weeks to months to actually get this stuff changed for a very small minority of paying customers on an ISP. Even if you call in expect to jump through many hoops, and the biggest will be getting that front line person to actually care about what you are saying. You might have to just ask directly for tier 2 or a supervisor as front line will run a trace route, say "its offnet, not our problem" and send you away.

Guess who has the weight to get this fixed though? Square enix does. They shell out 1000x what i pay my ISP so they should be able to get something sorted a hell of a lot faster than a regular paying customer can.

Note, i have a 50mbps connection, no saturation, perfect levels on my cable modem (i know this as i work for my ISP). I have seen getting routes changed take 1 month to ongoing for years. At the end of the day as a customer you are not paying for a quality offnet route to a server, square enix is paying for that......

[Raist]

Im having this issue as well. Lucky for me i work for my ISP so i know that as a single customer with very little pull (and even as an employee) i will not be able to force my ISP or Square Enix to get their routes in order.

What raist says is great and all but its weeks to months to actually get this stuff changed for a very small minority of paying customers on an ISP. Even if you call in expect to jump through many hoops, and the biggest will be getting that front line person to actually care about what you are saying. You might have to just ask directly for tier 2 or a supervisor as front line will run a trace route, say "its offnet, not our problem" and send you away.

Guess who has the weight to get this fixed though? Square enix does. They shell out 1000x what i pay my ISP so they should be able to get something sorted a hell of a lot faster than a regular paying customer can.

Note, i have a 50mbps connection, no saturation, perfect levels on my cable modem (i know this as i work for my ISP). I have seen getting routes changed take 1 month to ongoing for years. At the end of the day as a customer you are not paying for a quality offnet route to a server, square enix is paying for that......

And all the people thinking like this is why it is so hard to make progress on the matter. SE can do little more then us to directly affect the peering agreements with the Tier1 providers (Tata/Level3/Cogent). Those guys have agreements with the last mile ISP's---our ISP's. We the clients (that includes SE) need to lean on our ISP's to address the problems with their peering agreements to these intermediate parties, as it is our ISP's that have the contracts in play that are affecting the peering problem at hand. Here is the basic structure:

We contract with our ISP (TWC, Verizon, AT&T) for service, Our ISP sets up peering agreements with Tier 1 ISP's (like Tata/Level3/Cogent) for access to everyone else's networks.

SE contracts with their ISP's for service (Ormuco for the game, i-web for some website servers, JP government in Tokyo), and those ISP's set up peering agreements with those same Tier1's for access to everyone else's networks.

So the flow is basically:

Us -> Our Last Mile ISP -> Tier1 Providers <- SE's Last Mile ISP <- SE

The bulk of the problems are being detected between the two last mile ISP's--mostly on our side of the equation, which falls more squarely on our ISP's to address.

Couple this with the formal complaints lodged with regulators about how those last mile ISP's are neglecting their obligations to their customers to operate under fair practices concerning their peering arrangements that is resulting in overly congested ports at the exchanges, and one gains a better understanding of why these breakdowns are occuring in the middle of the route at the routing partners' segments. Will edit a link or two about this once I can find them again.. been a while since I read up on them. Found these articles because another user put them up in other threads...forget who it was and where though. I would like to give credit where it is due as these were good finds... will do so if I figure it out.

http://www.internap.com/2010/12/02/p...vel-3-and-you/
http://blog.level3.com/global-connec...-isps-internet
(there's a lot more like this out there, but the links in these articles will lead you to more about it if interested in learning more)

If the people negotiating these exchanges for the upstream are unable to convince either side to pony up the necessary resources to improve their networks, who else has a means to convince them?

Their paying clients.... as in us. Ultimately, if the last miles don't have enough paying customers to sustain operations then things may start to fall apart.

We need the strength in numbers though. True, just one person may have a hard time affecting change... but it is not to say that it is impossible. If one is well informed and persistent enough to get the needed escalation, one can get something done about it. Myself and others have in fact managed to do this in the past, and are doing so now. My route changes at least once a month on average. It changed a few weeks ago, and after things went south this weekend (as noted earlier with my disconnects and the route going south on me) and complaints were filed--the route changed. As of Sunday morning, I am now using Level3... but it is up to it's old tricks again, so I opened a ticket with them and we are in the process of gathering data so we can get it escalated once again.

This has been the "modus operandi" for our area for several months now as we have been working to improve service. As noted previously, our CMTS was fubar and only bonding half the channels. After we got that addressed, we moved on to looking into our routing issues But in order to get the ball rolling, SOMEONE had to take the initiative and press them to take action instead of taking the patent line about the connection to our modem being "fine". The problem typically is not our local connections at this point--it is upstream, and thus needs to be escalated to higher tiers for investigation and correction. All I had to do was bascally not take no as an answer and push for higher level support until I found someone willing to actually look at the data I was sending them.

Yes, it can take time to affect a change... but it doesn't have to take forever and a day to do it. My route has changed in as little as 3 business days on occasion (sucks when it goes to crap on the weekends though). Some of these ISP's have been continuously made aware of their issues (just take a look at the TWC forums for an example). When they start getting flooded with complaints containing specific information demonstrating issues upstream from enough customers demanding escalation to address the issue, they eventually get motivated and do take action.

The question is how many will it take. If everyone remains complacent, there is no incentive for them to take action. But if everyone becomes proactive and goes after them, they may be left no choice but to take action or risk loosing dissatisfied customers (and possibly suffer bad press from it via Twitter/Facebook/blogs). Vendors are already bleeding customers by the thousands as is (TWC has recently lost like 250k or something). Eventually the light bulb has to go off somewhere.

TLDR:
Strength in numbers can and does make a difference on this. But it only works if enough people get involved... complacency begets mediocrity. A means to affect change has been demonstrated countless times on these forums, and those that have been persistent have in fact made progress. If people want to sit idly by and wait for someone else to do the legwork for them, that is their right... but please, do not shoot down the efforts of those that are not only trying but are in fact making a difference by going after their ISP's to get them to address this particular issue.

[Gummy]

Again the problem lies in putting the stupid servers in CANADA... So dumb I can't even comprehend. The routing has been bad since ... the counter-strike days (and probably well before that). I blame SE for placing servers in F'n CANADA!

[Serret]

I've been having issues with Tata Communications for 2 months now.

I feel like I've been playing the ping-pong game except with 3 services,
Square Enix, Tata Communications and my ISP.


Square Enix says they can't do anything about it -> redirects me to tata or my isp
Tata Communications doesn't say anything at all
My ISP says they can't do anything about it -> redirects me to tata or square

I've been going over this for over a month now sending Traceroutes to all of them.
None seems to want to do anything about it.

My lowest ping when playing FF14 atm is 145ms ... with some packet loss in the middle ...prior to Tata Communications my conneciton was much more stable and the ping was as low as 120ms.
My average ping however, ranges from 165-180+ms which is just barely playable.. not to mention the DCs.

I did NOT have these issues when the connection was going through Level3 lines or Cogent lines.


This situation is despairing honestly, and I don't see a solution for this happening anytime soon -.-
I don't know what else to do at this point.
I'm honestly starting to consider whether I should just drop the subscription.


edit: I agree with Raist where he mentions that we should ACT, and not just "complain" then do nothing about it. Acting means reporting the situation and complaining in the proper channels (square enix support, tata communications' support, your ISP's support). The first and most important step I believe is to make Square Enix well aware that THIS is an issue. For Europeans it's particularly annoying because the servers aren't even located in Europe, it's basically adding insult to the injury.
I don't mean to say "my problem is bigger than yours", but it's particularly painful and because apparently we all share it US and EU we should try and make Square aware of it. For me its big enough an issue that I've considered quitting my sub several times over the past 2 months.


and no: http://forum.square-enix.com/ffxiv/t...ues-for-Europe
this issue is NOT resolved despite the post being closed and marked as resolved -.-

[ErickGaray-9]

I have also been encountering all these error codes as well this past week and I haven't been able to log into my character at all, and when I am able to log into it, it kicks me out of the world after like 5mins or so, I have tried calling square enix technical support for this whole past week as well and I all I get is to be put on hold for almost an hour before I hang up because the wait IA too long, I also have tried emailing square enix to notify them of my problem and still the company has yet to reach back to me at all, I just don't understand why I'm getting these errors codes when I wouldn't get them before, The error codes are (2002) and (90000) mostly are the ones I receive, I just want to simply play the game If anyone can be of any help I would greatly appreciate it on how to fix this, I have a very stable Internet connection and also a fast speed as well.

[Mankilla]

I have been playing for about 2 months now never any problems until this latest patch.. You should not roll out a new patch unless you know its not going to screw up your game???

[Turtle56k]

Up until now (in less of an hour), three times the message 90000 today.

And several episodes of lag (up to 5 seconds).

[SufhiSaito]

I'm getting constant lag spikes and 90000 as well. The internet connection is fine I've tested and troubleshooted everything Comcast a constant 50mbps tested router ping as well and never over 1ms. The only issue is on XIV no other games or streaming services.

[Towerbox12]

Honestly I was worried that my PS4's wifi connection was crapping out on me, and almost bought a new one. We all know this is a major problem, and even when you go afk you can get disconnected by 90000 error, it needs to be addressed sooner rather than later. All worlds have this issue, and every patch in the last two weeks has done nothing to repair it long term. All work on new patches for additional content should be stopped and the major problem fixed if people are losing money on this major concern. Honestly if SE doesn't act now, it will lose a LOT of money and support, which is completely their problem, not ours. All we can do now is wait for the problem to be fixed. If it goes on for long enough, maybe SE will give out free month subscriptions for the wasted time, however unlikely that seems.