Extreme lag with FFXIV Excalibur(NA)

[szalkerous]

It's going on two days with extreme lag to and from the Excalibur server.

I have contacted Comcast (my local provider) and my internet connection is performing to expectations. I am able to use other internet services without error or lag.

However, trying to play FFXIV I encounter almost unplayable lag, so bad that I failed a dungeon 4 times because I'd go from full health to dead during lag spikes.

I have heard a lot of other players with Comcast have very bad lag. Is there anything going on with the data center and routing packets to the Comcast backbone?

[szalkerous]

There is also a Reddit thread with hundreds of players from Comcast complaining:

https://www.reddit.com/r/ffxiv/comme...excessive_lag/

and a game forum post:

http://forum.square-enix.com/ffxiv/t...ual-Lag-Spikes

This needs to be looked at as a high priority.

[Raist]

They were likely only looking at the localized nodes and not potential issues from their peering partner or their exchange to that partner. It is important that you are dealing specifically with Tier3 support because of this. Those Tier1 and Tier2 guys most people deal with when they call in cannot look into the deeper issues usually at fault. Why people usually have better luck with the online contact options--it provides them a means to send things like tracert data demonstrating the problems crop up in relation to their peering arrangements with groups like Level3, Telia, Verizon Business/UUnet, etc.

Here is a sample trace I ran from a looking glass and posted a few days ago. It is from a Hurricane Electric router in LA to the Aether lobby in Canada:

  1    65 ms   48 ms   59 ms 213.248.67.141
  2    81 ms  144 ms   15 ms 4.68.70.129
  3    80 ms  273 ms  117 ms 4.69.153.86
  4    81 ms  218 ms  175 ms 4.69.153.86
  5   111 ms   99 ms   99 ms 4.59.178.74
  6   181 ms   82 ms  162 ms 192.34.76.10
  7    88 ms  178 ms  124 ms 199.91.189.242
  8   121 ms  153 ms   96 ms 199.91.189.74

The connection is showing signs of high congestion coming into and through their peering partner (Level3). This is an issue happening on the other side of their customers localized networks--beyond the point where those Tier 1/2 guys are usually looking. It would be along the line of the 5th hop or further down the route in most cases--those help desk guys are rarely looking past the third hop in such cases.

For comparison, here is a sample trace from my laptop to the same server here in South Carolina:

Tracing route to neolobby02.ffxiv.com [199.91.189.74]
over a maximum of 30 hops:

  1     1 ms     1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    36 ms    25 ms    26 ms  cpe-75-176-160-1.sc.res.rr.com [75.176.160.1]
  3    31 ms    30 ms    25 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    13 ms    15 ms    17 ms  clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
  5    26 ms    25 ms    24 ms  be33.drhmncev01r.southeast.rr.com [24.93.64.180]
  6    33 ms    33 ms    32 ms  bu-ether15.asbnva1611w-bcr00.tbone.rr.com [66.109.6.80]
  7    30 ms    29 ms    31 ms  0.ae2.pr1.dca10.tbone.rr.com [107.14.17.204]
  8    38 ms    35 ms    32 ms  ix-17-0.tcore2.AEQ-Ashburn.as6453.net [216.6.87.149]
  9    65 ms    63 ms    60 ms  if-11-3.thar2.NJY-Newark.as6453.net [216.6.87.242]
 10    56 ms    56 ms    58 ms  if-6-4.tcore2.N0V-New-York.as6453.net [66.198.111.98]
 11    69 ms    65 ms    61 ms  if-1-2.tcore2.NYY-New-York.as6453.net [216.6.99.6]
 12    73 ms    62 ms    60 ms  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
 13    59 ms    66 ms    61 ms  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
 14    59 ms    55 ms    57 ms  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
 15    63 ms    64 ms    60 ms  66.198.96.50
 16    60 ms    66 ms    71 ms  192.34.76.2
 17    63 ms    60 ms    60 ms  199.91.189.234
 18    64 ms    66 ms    68 ms  199.91.189.74

Trace complete.

Notice how much tighter the response times are across the entire route? That is because Time Warner is using TATA for our peering at the moment. When they switch us to Cogent, it gets a little worse...when it is Level3 it doesn't take long for it to go to pot on us and we have to contact them to get it switched again.

These decisions are governed by our ISP's policies not SE, and managed by the higher level teams in their infrastructure like Tier 3 Support (sometimes referred to as Engineering).

[szalkerous]

They were likely only looking at the localized nodes and not potential issues from their peering partner or their exchange to that partner. It is important that you are dealing specifically with Tier3 support because of this. Those Tier1 and Tier2 guys most people deal with when they call in cannot look into the deeper issues usually at fault. Why people usually have better luck with the online contact options--it provides them a means to send things like tracert data demonstrating the problems crop up in relation to their peering arrangements with groups like Level3, Telia, Verizon Business/UUnet, etc.
{snip}

Completely agree, but I've found as a "lowly residential customer" it's incredibly difficult to try and make any progress with that level of tech support, however if the DC or the fiber provider complains in reverse, they can usually get stuff done. I recall a similar problem with Rift a few years back, and Trion had their fiber provider yell at Comcast and replaced a malfunctioning router.

Although, I also point out that things are functioning "to spec" for other sites to eliminate my local routing hardware or modem/ISP-to-the-home service from the list of culprits.

[Raist]

Completely agree, but I've found as a "lowly residential customer" it's incredibly difficult to try and make any progress with that level of tech support, however if the DC or the fiber provider complains in reverse, they can usually get stuff done. I recall a similar problem with Rift a few years back, and Trion had their fiber provider yell at Comcast and replaced a malfunctioning router.

Although, I also point out that things are functioning "to spec" for other sites to eliminate my local routing hardware or modem/ISP-to-the-home service from the list of culprits.

And that is the point. The localized network is all they are looking at. The problems have routinely been upstream from the localized gateways. Typically at the exchanges or further into the routes.

You can usually get a more direct path to the higher tiers through online channels. Formal ticket systems, support email addresses, social media contacts---tend to get a lot further simply because you typically aren't dealing with the "scripted" process of the helpdesk. Might still windup going through the motions of the tech verifying signal on site and such, as they have to cover their assets on that front before escalation--but it can and has been done by many over the years.

[szalkerous]

And that is the point. The localized network is all they are looking at. The problems have routinely been upstream from the localized gateways. Typically at the exchanges or further into the routes.

You can usually get a more direct path to the higher tiers through online channels. Formal ticket systems, support email addresses, social media contacts---tend to get a lot further simply because you typically aren't dealing with the "scripted" process of the helpdesk. Might still windup going through the motions of the tech verifying signal on site and such, as they have to cover their assets on that front before escalation--but it can and has been done by many over the years.

I actually have a contact over at Comcast Business Tech Support who could probably loop that whole system, but for me to pester him I'd need to have all my ducks in a row--like you said, routing logs, proof that bypassing hop X eliminates the issue, etc.--and it's my busy season in software developer land. Hopefully others have the time to pursue this in detail like I would like to.

If I had time to trace and package it all I'd probably have that new RouterBoard cloud router set up too :P

[Kuleid]

I've also been having these exact issues for weeks now. I finally had enough and decided to contact Comcast this past weekend. Of course, being on the weekend, I knew from the start that I wouldn't get a representative who could actually help me, but none the less, after dealing with one of the desk employees, I managed to get them to elevate the ticket.

I received a call yesterday from someone in their routing department and he told me that he would keep the ticket on standby, but that if the packet loss occurred after the fourth hop (outside of Comcast's domain), they could not do anything about it. I told him that the packet loss mostly occurs at the fifth hop, which is the first Level3 server I get hopped to, and asked who I should contact to push this issue further along. He told me that I would have to talk to the game developer (SE) or the "website", which I'm assuming he meant Level3.

What I'm guessing is that I did not get quite high enough on up their chain. I'm going to try to re-open the ticket tomorrow and see if I can get it elevated higher.

Unfortunately, the last couple of times I've tried WTFast, it hasn't actually helped. In fact, in some cases, it's made it worse, making me think that the default route they set up for me isn't actually bypassing the problems.

It's a real shame this is happening, and that the "solution" has to be so complex, if even possible.

[Raist]

Comcast should be peering with other top tier ISP's besides Level3. Ormuco peers directly with Level3, TATA, Cogent, Verizon's UUNet Business class division, and TiNet. Depending on your region, they SHOULD be able to route you to at least one of the other 4 in that list.

They've been monkeying with their announcements for a few months now, so seeing a lot of outdated data on specific ASN's, but they do have peering set up with each one of those in their global peering list. Hard to say for certain exactly which of them would be available from your region without a specific IP to use to lookup the ASN for that particular network.

But... it should be possible to switch you to another peer. Rather, it IS possible for them to switch it...the questions are whether they have a peering arrangement with one of those other providers in your area, and whether someone will approve for them to tweak the policies to better optimize the routing for your gateway(s) there.

Edit:
yeah... found a list of addresses for some of their route servers:

68.86.80.0 - Ashburn, VA
68.86.80.2 - New York, NY
68.86.80.5 - Chicago, IL
68.86.80.6 - Chicago, IL 2
68.86.80.7 - Denver, CO
68.86.80.10 - San Jose, CA
68.86.80.11 - Los Angeles, CA
68.86.80.12 - Atlanta, GA
68.86.80.13 - Dallas, TX

Looking at the ID's assigned to them...they SHOULD have options available besides JUST Level3 if they aren't willing to lean on Level3 or otherwise improve QoS at the exchange point into Level3. (I pickup TATA in Ashburn, and when they put me on Cogent I go to Atlanta for the exchange--Comcast can do the same thing if you are close to those areas).

The trick is getting in touch with someone who can and will actually do something to help improve the routing in your area.

I know.. some of that data may make your head swim if you don't follow this stuff. The point is, they DO have options...but they are going to try to shank the responsibility and blame someone else. You may have to stick to your guns and push for supervisors or higher tiered support or something until you get someone who says yes. Sometimes they will do eveything they can to discourage people from pursuing it further...sometimes you have to go into bulldog mode (or raging chihuahua, whatever suits your style) and not take "no" for an answer.

Remember... you are paying them for internet service, and what you are getting is shoddy internet service within the scope of your usage. They don't advertise a quality service to just specific services, but internet service--that is a very broad network of networks, and the one they are assigning you as the go between to this particular section of the internet is not performing to a reasonable level of usage and THEY need to be held accountable to remedy it, as it is THEIR policy that is putting you on that troubled network. Not SE, not Microsoft, not Netflix... but COMCAST is deciding you should use Level3.