Serious Performance Issues (latency/server response time)

[Synovius]

Thanks for the replies.

As to it not being server-side and it being with our connection, I'm afraid to report that this isn't an issue isolated to me or even 2 or 3 or 5 people. This is an issue spread across literally everyone I've ever raided with. This is through 100mbps connections, through using Battleping or WTFast or even through VPNing to locations close to the data center. This is an issue with FFXIV and not with our machines or our connections. For example, here are my stats:

i5 2500K CPU
GTX980 4GB GPU
8GB RipJaws DDR3 RAM
Intel SSD
100mbps down / ~20mbps up

This is an issue with the game and with the servers or the way the game handles collision detection/positioning. I can even make a video if you want of me running directly through the balls on AS4 to have them not explode. I've even done this on AS3 where I've accidentally side-stepped through the little ball that comes out in the last phase only to have it not explode (thankfully in this case).

All I can say is that were I to fire up a game like WoW (I don't play WoW anymore because I find it boring and old), the responsiveness of the game is extremely crisp and exact. Things are far more fluid and synchronized there. I have no idea why this is but it is.

As to E'P's point, that makes sense with what limited understanding I have of networking concepts. I wonder if WoW uses UDP? At the very least I know the positioning of their data centers makes for substantially better routing. Although I could have sworn my latency to Gilgamesh was around 100ms which isn't amazing but it's all that bad (for an FPS it would be noticeable but shouldn't really be noticeable for an MMO). By comparison, the latency I feel in FFXIV, were I to compare it to my days of competitive UT2K4, would be like I was playing Bombing Run at 300-400ms.
Also HI E'P'!

[Raist]

Thanks for the replies.

As to it not being server-side and it being with our connection, I'm afraid to report that this isn't an issue isolated to me or even 2 or 3 or 5 people. This is an issue spread across literally everyone I've ever raided with. This is through 100mbps connections, through using Battleping or WTFast or even through VPNing to locations close to the data center. This is an issue with FFXIV and not with our machines or our connections. For example, here are my stats:

i5 2500K CPU
GTX980 4GB GPU
8GB RipJaws DDR3 RAM
Intel SSD
100mbps down / ~20mbps up

This is an issue with the game and with the servers or the way the game handles collision detection/positioning. I can even make a video if you want of me running directly through the balls on AS4 to have them not explode. I've even done this on AS3 where I've accidentally side-stepped through the little ball that comes out in the last phase only to have it not explode (thankfully in this case).

All I can say is that were I to fire up a game like WoW (I don't play WoW anymore because I find it boring and old), the responsiveness of the game is extremely crisp and exact. Things are far more fluid and synchronized there. I have no idea why this is but it is.

Also HI E'P'!

VPN'ing to the tunnel nearest the endpoint has been shown to not always improve things...the reason being you may still be hitting the congested networks in the same area you hit without the VPN. I myself have found more stable connections going through the midwest then back over (I'm on the East Coast). I've even seen one VPN that reported it's geolocation as Hungary respond more consistently than one to Montreal (though it was notably slower, but it didn't spike all over the place). I recently demonstrated this with my TunnelBear account on these forums.

It all depends on where your particular problems stem from. If there are issues with local segments--you can still have the same issues with a VPN, because even though you are now encrypted you are still physically on the same medium. If those problems are because of too many people physically on the line--bypassing the shaping filters with encryption is going to have a minimal impact. If the problem is in/around Level3's over-utilized exchange points for Montreal out of the North East, and the VPN is still going through those same exchanges---you will still have the same problems. BUT...if you are able to switch it up and get on to Cogent, TATA, TiNet/GTT...then things might improve. (Those URL's are links to their network maps...notice how they service the same regions? It means ISP's often have options on who they peer with.)

And that is why you see it happen to large groups at once...they are getting shunted through the same congested corridors. Ormuco currently lists peering with 5 major ISP's:


AS701 Verizon Business/UUnet
AS3356 Level 3 Communications, Inc.
AS3257 Tinet SpA
AS6453 TATA COMMUNICATIONS (AMERICA) INC
AS174 Cogent Communications
(source: Eidos/Ormuco ASN report for the Canadian subnet 199.91.189.0/24 from Hurricane Electric's BGP tool kit)

Notice those percentages in the pie chart. If any one of those is having issues, it is affecting potentially at least 15-28% of all traffic. If it is TWO ISP's having issues, you could be in the realm of 31-50% of all traffic being affected.

That is why your WoW experience doesn't quite cut it. Last I recall, they didn't have servers in Montreal...the addresses listed there for the Americas come back with geolocation data for New Jersey and California.

[Synovius]

While this information is all very interesting and informative, it doesn't change the fact that the poor performance of this game related to latency/lag/response time/whatever-you-want-to-refer-to-it-as affects literally everyone I've spoken too. Just an hour spent on ANY WoW server - even one across the United States from you (I live in EST and often played on PST servers) - yield crispness and seemingly immediate response times where moving out of an AoE a the very last split second is perfectly fine, moving into an AoE buff buffs you literally the split second you arrive in it and a Warrior's Charge ability tracks you every point of the way from your source to destination - so much so that you can be stunned mid-charge at any point and you will freeze immediately at that location.

So whether it's better data center locations (or more of them), or working with ISPs for better routing or reduction in any throttling (if that is even occurring), Square Enix needs to do something and do something relatively fast or their best days for this game are going to now be behind them. I've been unsuccessful at bringing a single friend over from GW2 or WoW strictly due to the poor performance of this game with respect to latency and response times. As an aside, they also often site the 2.5s GCD in this game compared to the significantly faster GCDs of games like GW2, Rift, and WoW (and most MMOs for that matter). And while I personally don't mind the 2.5s GCD and I feel that folks can easily get used to it, when you combine this slower pace of play with poor server conditions, you have a recipe for a very, VERY negative perception about how this game plays.

[Raist]

While this information is all very interesting and informative, it doesn't change the fact that the poor performance of this game related to latency/lag/response time/whatever-you-want-to-refer-to-it-as affects literally everyone I've spoken too. Just an hour spent on ANY WoW server - even one across the United States from you (I live in EST and often played on PST servers) - yield crispness and seemingly immediate response times where moving out of an AoE a the very last split second is perfectly fine, moving into an AoE buff buffs you literally the split second you arrive in it and a Warrior's Charge ability tracks you every point of the way from your source to destination - so much so that you can be stunned mid-charge at any point and you will freeze immediately at that location.

So whether it's better data center locations (or more of them), or working with ISPs for better routing or reduction in any throttling (if that is even occurring), Square Enix needs to do something and do something relatively fast or their best days for this game are going to now be behind them. I've been unsuccessful at bringing a single friend over from GW2 or WoW strictly due to the poor performance of this game with respect to latency and response times. As an aside, they also often site the 2.5s GCD in this game compared to the significantly faster GCDs of games like GW2, Rift, and WoW (and most MMOs for that matter). And while I personally don't mind the 2.5s GCD and I feel that folks can easily get used to it, when you combine this slower pace of play with poor server conditions, you have a recipe for a very, VERY negative perception about how this game plays.

It doesn't apply to everyone you've spoken to all the time. Guess you missed that I have no such issues...and there are many others like me. You apparently have issues with how your data is traversing the internet. There are MANY others here who also don't. The reason is simply that they are routed differently. Some get it from their ISP's efforts while others are doing it via a VPN.

And no, I don't rely on a VPN for this game. I used one off and on during the first months of the 2.0 relaunch, mostly as a tool for demonstration to my ISP's tier2 and tier3 techs when things would go south. Eventually they stopped going through the checklist every time I sent them a dump of traces, pings, and modem logs and just went to work tracking down the problem and fixing it for me.

Ultimately we narrowed down some localized issues to failing hardware at the head end--was actually a device that had been in play since Clinton was in office and was on it's last leg. Still haven't been put back on a dedicated gateway for Florence yet...but the Conway one isn't too bad. Most days.

They still have some upstream issues to address in the Carolinas...but we have seen much improvement in the last year. But we have to stay on them though...it is Time Warner after all. Their forums are constantly lit up with complaints...a large part of it is upstream issues(though there are still a lot of local channel alignment issues---still a lot of outdated hardware in play). The biggest change though is likely bound to us pushing them to keep us away from Level3 as much as possible. Seems to be one of the more common weak links in the routing.

But...if people aren't willing to look at the facts or listen to reason....guess we may not see any lasting dramatic improvements for them anytime soon.

EDIT:
Out of curiosity...I booted the laptop to take some snapshots. Here is some quick data on the route I take to my lobby server for Midgard (trying to avoid spamming the game servers themselves for these thing), followed by the routing to the two WoW servers in North America for a comparison. Note the GeoData for these IP's are Montreal, New Jersey, and California from a laptop on Wi-Fi in South Carolina:

C:\Windows\System32>tracert neolobby02.ffxiv.com

Tracing route to neolobby02.ffxiv.com [199.91.189.74]
over a maximum of 30 hops:

  1     1 ms     2 ms     1 ms  LPTSRV [10.10.100.1]
  2    26 ms    30 ms    27 ms  cpe-75-176-160-1.sc.res.rr.com [75.176.160.1]
  3    26 ms    23 ms    20 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    13 ms    11 ms    15 ms  clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
  5    23 ms    29 ms    27 ms  be33.drhmncev01r.southeast.rr.com [24.93.64.180]
  6    28 ms    29 ms    30 ms  ge-0-3-0.chrlncpop-rtr1.southeast.rr.com [24.93.64.173]
  7    31 ms    33 ms    34 ms  bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
  8    31 ms    30 ms    29 ms  0.ae1.pr0.atl20.tbone.rr.com [66.109.6.177]
  9    30 ms    28 ms    31 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
 10    31 ms    30 ms    30 ms  be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165]
 11    43 ms    43 ms    44 ms  be2112.ccr41.dca01.atlas.cogentco.com [154.54.7.157]
 12    47 ms    47 ms    50 ms  be2148.ccr41.jfk02.atlas.cogentco.com [154.54.31.118]
 13    52 ms    53 ms    51 ms  be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50]
 14    56 ms    57 ms    56 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 15    61 ms    60 ms    61 ms  38.122.42.34
 16    58 ms    57 ms    58 ms  192.34.76.10
 17    63 ms    65 ms    58 ms  199.91.189.242
 18    59 ms    58 ms    59 ms  199.91.189.74

Trace complete.

Ping statistics for 199.91.189.74:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 54ms, Maximum = 58ms, Average = 56ms

C:\Windows\System32>tracert 12.129.222.10

Tracing route to 12.129.222.10 over a maximum of 30 hops

  1     1 ms     1 ms     1 ms  LPTSRV [10.10.100.1]
  2    33 ms    29 ms    29 ms  cpe-75-176-160-1.sc.res.rr.com [75.176.160.1]
  3    25 ms    27 ms    31 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    22 ms    23 ms    35 ms  clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
  5    27 ms    25 ms    28 ms  be33.drhmncev01r.southeast.rr.com [24.93.64.180]
  6    29 ms    28 ms    29 ms  ae14.chrlncpop-rtr1.southeast.rr.com [24.93.64.25]
  7    32 ms    33 ms    33 ms  bu-ether24.atlngamq46w-bcr00.tbone.rr.com [107.14.19.18]
  8    38 ms    37 ms    37 ms  66.109.1.255
  9    32 ms    31 ms    39 ms  0.ae2.pr1.atl20.tbone.rr.com [107.14.17.188]
 10    29 ms    31 ms    30 ms  ix-14-0.tcore1.A56-Atlanta.as6453.net [64.86.113.37]
 11    33 ms    33 ms    33 ms  192.205.36.217
 12    47 ms    49 ms    49 ms  12.122.29.54
 13    47 ms    49 ms    48 ms  cr2.wswdc.ip.att.net [12.122.1.174]
 14    47 ms    47 ms    49 ms  12.123.250.117
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18  ^C

<I stopped the trace, as I hit their firewall as detailed in their guide. Also unable to ping it>

C:\Windows\System32>tracert 199.108.62.157

Tracing route to 199.108.62.157 over a maximum of 30 hops

  1     1 ms     1 ms     1 ms  LPTSRV [10.10.100.1]
  2    29 ms    24 ms    28 ms  cpe-75-176-160-1.sc.res.rr.com [75.176.160.1]
  3    33 ms    27 ms    29 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    12 ms    13 ms    14 ms  clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
  5    23 ms    24 ms    24 ms  be33.drhmncev01r.southeast.rr.com [24.93.64.180]
  6    28 ms    29 ms    30 ms  ge-0-3-0.chrlncpop-rtr1.southeast.rr.com [24.93.64.173]
  7    31 ms    32 ms    33 ms  bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
  8    37 ms    29 ms    38 ms  66.109.1.255
  9    31 ms    32 ms    32 ms  0.ae3.pr1.atl20.tbone.rr.com [107.14.17.190]
 10    30 ms    28 ms    33 ms  ix-14-0.tcore1.A56-Atlanta.as6453.net [64.86.113.37]
 11    48 ms    45 ms    45 ms  if-13-2.tcore1.AEQ-Ashburn.as6453.net [64.86.113.10]
 12    43 ms    44 ms    46 ms  if-2-2.tcore2.AEQ-Ashburn.as6453.net [216.6.87.1]
 13    47 ms    48 ms    47 ms  192.205.34.245
 14    47 ms    46 ms    49 ms  cr1.wswdc.ip.att.net [12.122.113.42]
 15    46 ms    44 ms    45 ms  12.123.250.37
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19  ^C
<again, hitting their security...so can't ping this one either>

Notice a few very important things about this data:

1.) No more XO.net and no Level3 hops at all in any of those routes...these were frequently fingered as major offenders in our testing.
2.) Fairly tight, and consistently low response times throughout the entire route.
3.) No circular/redundancies in the routing (we frequently see traces return the SAME level3 hops back to back in a single trace)
4.) Response times are mostly in keeping with distance--no major spikes disproportionate to the distance
(note that I go to Conway/Myrtle Beach and then to Columbia first--problems in Florence has resulted in strange band-aids. At one point, I went to Hilton Head first. Believe it or not...in both cases this sling-shotting proved much more stable)

Basically...these are "clean" and consistent routes. As a result, my gameplay is also stable and responsive...even when people are streaming to the Roku's.

Oh yeah... and barring ONE incident when TWC thought they caught an exception between Ormuco and SE---we haven't been in contact with them at all.

This was managed through Time Warner.


Edit 2:
0.o just after posting that data, they rebooted my modem....and my route has now changed. Must be some maintenance going on or something---I'm peering with Cogent now instead of TATA:

Tracing route to neolobby02.ffxiv.com [199.91.189.74]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  LPTSRV [10.10.100.1]
  2    33 ms    29 ms    27 ms  cpe-75-176-160-1.sc.res.rr.com [75.176.160.1]
  3    24 ms    32 ms    30 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    12 ms    12 ms    25 ms  clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
  5    26 ms    22 ms    27 ms  be33.drhmncev01r.southeast.rr.com [24.93.64.180]
  6    26 ms    29 ms    30 ms  ge-0-3-0.chrlncpop-rtr1.southeast.rr.com [24.93.64.173]
  7    31 ms    32 ms    31 ms  bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
  8    33 ms    52 ms    73 ms  0.ae1.pr0.atl20.tbone.rr.com [66.109.6.177]
  9    31 ms    29 ms    30 ms  te0-0-0-10.ccr21.atl02.atlas.cogentco.com [154.54.12.109]
 10    30 ms    30 ms    31 ms  be2050.ccr41.atl01.atlas.cogentco.com [154.54.0.165]
 11    43 ms    44 ms    43 ms  be2112.ccr41.dca01.atlas.cogentco.com [154.54.7.157]
 12    49 ms    47 ms    48 ms  be2148.ccr41.jfk02.atlas.cogentco.com [154.54.31.118]
 13    52 ms    52 ms    52 ms  be2106.ccr21.alb02.atlas.cogentco.com [154.54.3.50]
 14    56 ms    57 ms    58 ms  be2088.ccr21.ymq02.atlas.cogentco.com [154.54.43.17]
 15    70 ms    72 ms    72 ms  38.122.42.34
 16    55 ms    59 ms    57 ms  192.34.76.10
 17    56 ms    57 ms    59 ms  199.91.189.242
 18    56 ms    57 ms    57 ms  199.91.189.74

Trace complete.

Cogent isn't typically as solid as it has been with TATA (will often get some 150-200ms hiccups once in a while with Cogent)---but it is NOTHING like the massive spikes we see with Level3 in play...talking full on 1000+ starred events. As in full-on one second pauses (or longer) with Level3 when they misbehave (and that's not tracerts mind you--straight pings).

Again... Time Warner just did this. Not Square-Enix.

[Synovius]

Out of curiosity, Raist, have you played WoW at all? I think if you had it would be immediately apparent how much more responsive and crisp the gameplay is. And, again, I don't even like the game now because I found WoD to be boring. However, there is absolutely no denying how responsive the game is compared to FFXIV - and this is even while playing on a server located across North America from me.

Either way, I will do some pinging and tracert when I get home here shortly to Gilgamesh and report the results here.