Reporting Latency Related Issue (NA Data Center)

[Droleth]

Name of your Internet Service Provider (ISP): Shaw Communications
Ticket or reference number from your ISP: Ticket#39026184
State/Country: Alberta, Canada
Date/Time: Within the last week it has started
Frequency: Multiple times in a 10-30 min stretch.
Content: A10S
Which in-game World were you playing on when the issue occurred? Midgardsormr
Traceroute records (not sure where to get the IP)

[Hoshiko]

Name of your Internet Service Provider (ISP)
Brighthouse/Spectrum
Ticket or reference number from your ISP.
None
State/Country
Florida, United States
Date/Time (When did the issue start happening, at what specific times does this issue arise, if any.)
3/14/2017, 9pm EST through current
Frequency (How often does the issue occur?)
Constantly
Content (Which content were you playing in-game when the issue occurred?)
Latency occurs when zoning between all areas or just opening my inventory. There has been no change in internet speed for other programs.
Other characters are slower than normal to load
Which in-game World were you playing on when the issue occurred?
Balmung

[StrongMachine]

Name of your Internet Service Provider (ISP)
Charter Spectrum (was Time Warner Cable)
Ticket or reference number from your ISP.
Reported. Ping and tracert ran over chat with good results. No ticket or reference number given.
State/Country
Alabama/USA
Date/Time
Severe since Nov 2016 to current May 2017
Frequency
Consistant Packet Loss, 2-7 sec delay, everyday after 5pm. Ran WTFast and averaged 87 lost packets per hour.
Content
All
Which in-game World were you playing on when the issue occurred?
Balmung
Traceroute records

Tracing route to www.agwww.timewarnercable.com [*****] over a maximum of 30 hops: 1 <1 ms <1 ms 1 ms ***** 2 35 ms 25 ms 27 ms cpe-108-167-112-1.natsoe.res.rr.com [*****] 3 22 ms 21 ms 21 ms ***** 4 38 ms 39 ms 39 ms be29.clmkohpe01r.midwest.rr.com [*****] 5 44 ms 62 ms 48 ms bu-ether15.chctilwc00w-bcr00.tbone.rr.com [*****] 6 66 ms 71 ms 71 ms bu-ether13.vinnva0510w-bcr00.tbone.rr.com [*****] 7 86 ms 67 ms 68 ms bu-ether7.chrcnctr00w-bar00.tbone.rr.com [*****] 8 63 ms 75 ms 63 ms ***** 9 64 ms 62 ms 74 ms ***** 10 64 ms 65 ms 64 ms *****

[RedPandaWasTaken]

• Name of your Internet Service Provider (ISP): VIVO (Telefônica Brasil, what was previously GVT)
• Ticket or reference number from your ISP:N/A, they reported that everything was fine on their end (friends on a different ISP -NET Virtua- have reported that their route is better, but I cannot confirm especially since other people in the forums with said ISP are having the same problem)
• State/Country: ES/Brazil
• Date/Time: Started May 17, 2017 after the server move.
• Frequency: Latency issues all the time. Latency used to be in the mid 100s (150-180) and now is consistently 300+, I get better pings to the EU servers which makes me believe that we're being routed through EU before going to the NA servers. Other friends and players reported the same issues.
• Content: Impacts all activities, Dungeons, casually walking around, Trials, Raids, everything.
• Which in-game World were you playing on when the issue occurred?: Ultros
• Traceroute records: See below.

====================== PING ======================

Pinging 204.2.229.107 with 32 bytes of data:
Reply from 204.2.229.107: bytes=32 time=302ms TTL=51
Reply from 204.2.229.107: bytes=32 time=304ms TTL=50
Reply from 204.2.229.107: bytes=32 time=299ms TTL=51
Reply from 204.2.229.107: bytes=32 time=299ms TTL=51

Ping statistics for 204.2.229.107:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 299ms, Maximum = 304ms, Average = 301ms

====================== TRACE ======================

Tracing route to 204.2.229.107 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.1
2 6 ms 6 ms 6 ms gvt-b-se01.vta.gvt.net.br [189.59.241.114]
3 7 ms 6 ms 7 ms 177.205.8.205.static.adsl.gvt.net.br [177.205.8.205]
4 21 ms 18 ms 19 ms gvt-te-0-6-0-4.rt02.rjo.gvt.net.br [187.115.218.133]
5 49 ms 50 ms 47 ms 201.22.76.202.dynamic.dialup.gvt.net.br [201.22.76.202]
6 52 ms 52 ms 52 ms grafortw2-7-3-0.net.telefonicaglobalsolutions.com [216.184.112.214]
7 120 ms 120 ms 223 ms 94.142.127.50
8 155 ms 154 ms 154 ms 94.142.126.205
9 253 ms 232 ms 234 ms 213.140.35.205
10 247 ms 258 ms 247 ms 5.53.3.18
11 254 ms 253 ms 255 ms ae1-0-ntt-grtamstc1.net.telefonicaglobalsolutions.com [213.140.53.97]
12 241 ms 242 ms 240 ms ae-4.r25.amstnl02.nl.bb.gin.ntt.net [129.250.2.146]
13 253 ms 254 ms 253 ms ae-5.r23.asbnva02.us.bb.gin.ntt.net [129.250.6.162]
14 295 ms 296 ms 295 ms ae-10.r22.snjsca04.us.bb.gin.ntt.net [129.250.6.237]
15 300 ms 300 ms 300 ms ae-40.r02.snjsca04.us.bb.gin.ntt.net [129.250.3.121]
16 304 ms 304 ms 304 ms ae-2.r00.scrmca02.us.bb.gin.ntt.net [129.250.7.33]
17 300 ms 300 ms 299 ms xe-0-1-0-1-1.r00.scrmca02.us.ce.gin.ntt.net [129.250.195.46]
18 300 ms 299 ms 299 ms 204.2.229.234
19 302 ms 302 ms 304 ms 204.2.229.107

Trace complete.

==================== EU TRACE ======================

Tracing route to 195.82.50.9 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.1
2 6 ms 6 ms 6 ms gvt-b-se01.vta.gvt.net.br [189.59.241.114]
3 7 ms 7 ms 6 ms 177.205.8.205.static.adsl.gvt.net.br [177.205.8.205]
4 20 ms 18 ms 19 ms gvt-te-0-3-0-2.rt02.rjo.gvt.net.br [179.184.75.100]
5 47 ms 50 ms 47 ms 201.22.76.200.dynamic.dialup.gvt.net.br [201.22.76.200]
6 52 ms 51 ms 52 ms grafortw2--5-2-1.net.telefonicaglobalsolutions.com [216.184.112.72]
7 125 ms 126 ms 127 ms 94.142.98.153
8 * * * Request timed out.
9 * * * Request timed out.
10 241 ms 241 ms 241 ms 195.16.161.178
11 230 ms 231 ms 230 ms 195.82.61.14
12 234 ms 234 ms 235 ms 195.82.50.234
13 244 ms 243 ms 243 ms 195.82.50.9

Trace complete.

[Virtguy]

Name of your Internet Service Provider (ISP): Time Warner Cable
Ticket or reference number from your ISP: Ticket opened on DSLReports TWC Direct Forum
State/Country: NC, USA
Date/Time (When did the issue start happening, at what specific times does this issue arise, if any.): At least 10 months, every night
Frequency (How often does the issue occur?): Nightly, typically between 5pm and 11pm
Content (Which content were you playing in-game when the issue occurred?): Any
Which in-game World were you playing on when the issue occurred?: Ultros
Traceroute records (optional) (Learn how to generate traceroute record):

TL;DR: Nightly packet loss lasting for several hours has been ongoing for months, and generally makes the game unplayable. Using a VPN (WTFast or Pingzapper) bypasses the affected routes and results in perfectly smooth gameplay. Multiple cases opened with TWC, but no issues acknowledge on their end.

---

I've been monitoring Ultros for over a year with PRTG. Here's a current 2 day view of the issue, which depicts packet loss and the corresponding increase in latency:


This is a weekly view:


And a monthly view:


The issue has been ongoing for months. I'm no longer running PingPlotter, but this is are some screenshots from when I was. Notice the packet loss after the traffic leaves the TWC network (first hops redacted for personally identifiable info):




It's important to note that the issue is not local to my house, neighborhood, or region. There is no corresponding increase in packet loss or latency when monitoring one of the last hops out of the TWC network. As mentioned in the TL;DR, I'm able to use a VPN service and play the game without any issues, further ruling out any local or regional problems. Here's a monthly view of one of those hops, notice the average response time is very flat for the entire month, without the nightly spikes:


WTFast provides a bit of data, but does themselves a disservice the way they calculate ping spikes. The important data is the packet loss over the internet, and no packet loss over the tunnel, and the resulting latency difference between the two:


I'm starting up PingPlotter again to capture more current data. For now, here's pathping results showing the current route, but too short of a sample period to show packet loss and latency:

Windows PowerShell
Copyright (C) 2014 Microsoft Corporation. All rights reserved.

PS C:\Users\Administrator> pathping 199.91.189.36

Tracing route to 199.91.189.36 over a maximum of 30 hops

  0  lab-hv01 [172.16.1.180]
  1  172.16.1.1
  2  cpe-65-190-176-1.nc.res.rr.com [65.190.176.1]
  3  cpe-65-190-176-1.nc.res.rr.com [65.190.176.1]
  4  cpe-174-111-104-057.triad.res.rr.com [174.111.104.57]
  5  cpe-024-025-062-024.ec.res.rr.com [24.25.62.24]
  6  be31.chrcnctr01r.southeast.rr.com [24.93.64.186]
  7  bu-ether24.atlngamq46w-bcr00.tbone.rr.com [107.14.19.18]
  8  0.ae1.pr0.atl20.tbone.rr.com [66.109.6.177]
  9  twc-telus.pr0.chi10.tbone.rr.com [66.109.9.70]
 10  ae-11-11.car2.montreal2.level3.net [4.69.141.1]
 11  ae-11-11.car2.montreal2.level3.net [4.69.141.1]
 12  ormuco-comm.car2.montreal2.level3.net [4.59.178.74]
 13  192.34.76.10
 14  199.91.189.242
 15  199.91.189.36

Computing statistics for 375 seconds...
            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
  0                                           lab-hv01 [172.16.1.180]
                                0/ 100 =  0%   |
  1    0ms     0/ 100 =  0%     0/ 100 =  0%  172.16.1.1
                                0/ 100 =  0%   |
  2   13ms     0/ 100 =  0%     0/ 100 =  0%  cpe-65-190-176-1.nc.res.rr.com [65.190.176.1]
                                0/ 100 =  0%   |
  3   13ms     0/ 100 =  0%     0/ 100 =  0%  cpe-65-190-176-1.nc.res.rr.com [65.190.176.1]
                                0/ 100 =  0%   |
  4   19ms     0/ 100 =  0%     0/ 100 =  0%  cpe-174-111-104-057.triad.res.rr.com [174.111.104.57]
                                0/ 100 =  0%   |
  5   13ms     0/ 100 =  0%     0/ 100 =  0%  cpe-024-025-062-024.ec.res.rr.com [24.25.62.24]
                                0/ 100 =  0%   |
  6   20ms     0/ 100 =  0%     0/ 100 =  0%  be31.chrcnctr01r.southeast.rr.com [24.93.64.186]
                                0/ 100 =  0%   |
  7   25ms     0/ 100 =  0%     0/ 100 =  0%  bu-ether24.atlngamq46w-bcr00.tbone.rr.com [107.14.19.18]
                                0/ 100 =  0%   |
  8   30ms     0/ 100 =  0%     0/ 100 =  0%  0.ae1.pr0.atl20.tbone.rr.com [66.109.6.177]
                                0/ 100 =  0%   |
  9   27ms     0/ 100 =  0%     0/ 100 =  0%  twc-telus.pr0.chi10.tbone.rr.com [66.109.9.70]
                                0/ 100 =  0%   |
 10  ---     100/ 100 =100%   100/ 100 =100%  ae-11-11.car2.montreal2.level3.net [4.69.141.1]
                                0/ 100 =  0%   |
 11  ---     100/ 100 =100%   100/ 100 =100%  ae-11-11.car2.montreal2.level3.net [4.69.141.1]
                                0/ 100 =  0%   |
 12   55ms     0/ 100 =  0%     0/ 100 =  0%  ormuco-comm.car2.montreal2.level3.net [4.59.178.74]
                                0/ 100 =  0%   |
 13  ---     100/ 100 =100%   100/ 100 =100%  192.34.76.10
                                0/ 100 =  0%   |
 14  ---     100/ 100 =100%   100/ 100 =100%  199.91.189.242
                                0/ 100 =  0%   |
 15   51ms     0/ 100 =  0%     0/ 100 =  0%  199.91.189.36

Trace complete.
PS C:\Users\Administrator>

Please feel free to contact me directly if you'd like to coordinate remote testing or observation.

[Mases]

• Name of your Internet Service Provider (ISP): Comcast
• Ticket or reference number from your ISP: Comcast says that there's no issue on their end.
• State/Country: Virginia / USA
• Date/Time: I noticed it for the first time last Friday (November 27th), it persisted that Saturday and got somewhat better through the week with only occasional spikes until tonight around 7-8pm eastern.
• Frequency: Primarily a serious issue during the weekends. I've recently returned to the game last week, and didn't have any problem until the 27th of November.
• Content: All content: dungeons, doing anything in the world. Disconnected while in Idyllshire twice before giving up on playing for the night (RIP Friday Night Fights). I also noticed issues in Alexander earlier in the evening.
• World: Exodus.
• Traceroute records:

Tracing route to 199.91.189.39 over a maximum of 30 hops

  1     1 ms     2 ms     1 ms  10.0.0.1
  2    22 ms    18 ms    11 ms  96.120.80.201
  3     9 ms     9 ms     9 ms  xe-5-2-0-sur01.lynchburg.va.richmond.comcast.net [68.85.214.249]
  4    13 ms    11 ms    12 ms  ae-21-0-ar02.charlvilleco.va.richmond.comcast.net [68.85.214.113]
  5    24 ms    17 ms    15 ms  hu-1-1-0-5-10-cr02.ashburn.va.ibone.comcast.net [68.86.93.45]
  6    28 ms    12 ms    14 ms  23.30.207.94
  7   117 ms   225 ms   220 ms  ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]
  8   108 ms   213 ms   227 ms  ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]
  9    28 ms    26 ms    27 ms  ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
 10    58 ms    46 ms    77 ms  192.34.76.10
 11    53 ms    46 ms    51 ms  199.91.189.242
 12    54 ms    72 ms    86 ms  199.91.189.39

Trace complete.

[Samezuka]

Name of your Internet Service Provider (ISP): AT&T U-Verse
Ticket or reference number from your ISP: They didn't give me one, though I contacted them multiple times about the issue.
State/Country: FL / USA
Date/Time: Started after the move to the Chaos Data center (around 10/20), improved a bit, then the lag got far far worse this week since this Tuesday (12/01).
Frequency: It occurs about every few seconds. It's awful during EST primetime. Lots of freezing and characters not moving only to speed up quickly.
Content: Everything - Thordan Ex, Ravana Ex, Ex roulette, low level instances, traveling out in the world, crafting, FATES, etc.
Which in-game World were you playing on when the issue occurred? Aether - Jenova
Traceroute records: TBA

[Nbelheim]

Name of your Internet Service Provider (ISP): Comcast
Ticket or reference number from your ISP.: did not provide one
State/Country: VA / USA
Date/Time: Starts 1900 to 2100 EDT
Frequency: daily
Content: Ex roulette, over world areas. Anywhere
Which in-game World were you playing on when the issue occurred? Leviathan -Primal Data Center
Traceroute records: NA

[Milapurr]

Name of your Internet Service Provider (ISP) Verizon
Ticket or reference number from your ISP: They place the blame on the servers not the internet they're providing.
State/Country: Pennsylvania/US
Date/Time: Since I've played so at least 3 months
Frequency: Between 6pm EST and 12 AM EST
Content: Any and all instances, bosses, actual game play.
Which in-game World were you playing on when the issue occurred? Jenova
Traceroute records
C:\Windows\system32>tracert 4.59.178.74

Tracing route to ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.1.1
2 27 ms 31 ms 27 ms 10.12.13.1
3 28 ms 26 ms 26 ms P15-2.HRBGPA-LCR-01.verizon-gni.net [130.81.44.180]
4 33 ms 35 ms 40 ms T0-7-0-0.HRBGPA-LCR-21.verizon-gni.net [130.81.151.157]
5 * * * Request timed out.
6 * * * Request timed out.
7 32 ms 32 ms 33 ms 0.ae1.BR1.IAD8.ALTER.NET [140.222.229.163]
8 * * * Request timed out.
9 * * * Request timed out.
10 41 ms 41 ms 41 ms ae-101-101.ebr1.Washington12.Level3.net [4.69.204.229]
11 41 ms 40 ms 41 ms ae-1-100.ebr2.Washington12.Level3.net [4.69.143.214]
12 * * * Request timed out.
13 * 34 ms * ae-1-11.ebr4.NewYork1.Level3.net [4.69.204.198]
14 78 ms 254 ms 70 ms ae-5-5.car1.Montreal2.Level3.net [4.69.141.5]
15 40 ms 40 ms 39 ms ae-11-11.car2.Montreal2.Level3.net [4.69.141.1]
16 70 ms 69 ms 71 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]

Question from me: if there's a level 3 server in Washington DC or in New York City? Why are we not connecting to those and instead to Montreal which is further away?