TATA Communications - the reason we're lagging

[Kitai]

Been suffering lately from high latency, packet loss and disconnects? You're not alone. The issue seems to be TATA Communication routers - the CSP that most of us that lag use to reach the Montreal Datacenter. This is what a common tracert might look like to one of us.

1 <1 ms <1 ms <1 ms Voidzone [192.168.1.1]
2 1 ms 5 ms 1 ms gw-n1-u-d2.ias.bredband.telia.com [195.198.146.129]
3 3 ms 3 ms 3 ms s-b4-pos14-0.telia.net [213.248.101.142]
4 4 ms 4 ms 4 ms s-bb4-link.telia.net [213.155.131.32]
5 4 ms 4 ms 4 ms s-b3-link.telia.net [213.155.136.53]
6 4 ms 4 ms 4 ms vsnl-ic-124239-s-b1.c.telia.net [213.248.104.162]
7 283 ms 260 ms 260 ms if-7-1-3-0.tcore1.FR0-Frankfurt.as6453.net [195.219.50.145]
8 121 ms 119 ms 118 ms if-4-2.tcore1.PVU-Paris.as6453.net [80.231.153.10]
9 136 ms 137 ms 138 ms if-2-2.tcore1.PYE-Paris.as6453.net [80.231.154.18]
10 133 ms 141 ms 133 ms if-3-6.tcore1.L78-London.as6453.net [80.231.130.85]
11 133 ms 131 ms 129 ms if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
12 139 ms 137 ms 136 ms if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
13 146 ms * 138 ms if-2-2.tcore2.MTT-Montreal.as6453.net [64.86.226.13]
14 133 ms 134 ms 133 ms if-0-2.tcore1.MTT-Montreal.as6453.net [216.6.115.89]
15 134 ms * * if-5-2.tcore1.W6C-Montreal.as6453.net [64.86.31.6]
16 196 ms 196 ms * 66.198.96.50
17 209 ms 211 ms 213 ms 192.34.76.2
18 199 ms 207 ms 192 ms 199.91.189.234
19 209 ms * 211 ms 199.91.189.26

From hop 7 to 17 I'm using exclusively TATA Communication routers (as6453.net). Typically, the as6453.net backbone routers handle the load without issue. Even hop 7, which spiked during the tracert, showed a steady 30ms ping when I manually pinged it later.

The issue occurs at hop 16. A follow-up extended ping showed a steady 20% packetloss with high ping spikes. Funnily enough, this router, just like all the as6453.net routers, belongs to TATA Communications, but it's not part of the backbone. I believe it's one of many link routers that connects the Montreal Datacenter (starts at 192.34.76.2) to the TATA backbone, and is devoted specifically for the use of Ormuco Communications.

Obviously, they underestimated the amount of traffic going through this node, and it's suffering for it. They (meaning TATA Communications) need to split the load to other routers, or upgrade their hardware at this node. Sadly, it's nothing we endpoint customers can do anything about. TATA Communications work as a CSP, meaning they sell their services to ISP's. ISP's in turn sell their services to us, and to SQEX (The Montreal Datacenter uses Ormuco Communications as an ISP).

What we can do is lean on SQEX, who in turn can lean on Ormuco, who in turn can lean on TATA Communications, who are not providing the service they should. We can also lean on our respective ISP's who are employing TATA Communications to route their traffic to Montreal. The more feedback TATA gets about this issue, the more likely it's going to get resolved as soon as possible.

So send your support tickets, to SQEX and to your ISP, and try to include as much information as possible. When your problems occur, which router is responsible (use tracert and ping, or alternatively a third-party program like WinTrace to make things easier), and anything else you can think of. Unless we voice this issue we're going to get brushed off as a minority, or that the problem somehow is at our end.

[Sissyfox]

I don't know if it helps, but I'm in Sweden too and my trace looked like this.

1 <1 ms 3 ms 1 ms router []
2 7 ms <1 ms <1 ms skovde1-ge-1-3-0.sunet.se [130.242.82.177]
3 42 ms 8 ms 8 ms t1tug-ge-7-3-2.sunet.se [130.242.82.117]
4 9 ms 8 ms 9 ms se-tug.nordu.net [109.105.102.17]
5 7 ms 7 ms 19 ms se-fre.nordu.net [109.105.97.1]
6 7 ms 7 ms 7 ms s-b3-link.telia.net [213.248.97.17]
7 10 ms 8 ms 7 ms vsnl-ic-124239-s-b1.c.telia.net [213.248.104.162]
8 140 ms 136 ms 145 ms if-7-1-3-0.tcore1.FR0-Frankfurt.as6453.net [195.219.50.145]
9 137 ms 138 ms 138 ms if-5-2.tcore1.PVU-Paris.as6453.net [80.231.153.121]
10 146 ms 145 ms 144 ms if-12-2.tcore1.PYE-Paris.as6453.net [80.231.154.69]
11 144 ms 144 ms 145 ms if-3-6.tcore1.L78-London.as6453.net [80.231.130.85]
12 143 ms 145 ms 147 ms if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
13 140 ms 141 ms 159 ms if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
14 157 ms 179 ms 153 ms if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
15 163 ms 167 ms 162 ms if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
16 161 ms 163 ms 162 ms if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
17 225 ms 234 ms 235 ms 66.198.96.50
18 232 ms 245 ms 230 ms 192.34.76.2
19 226 ms * 231 ms 199.91.189.234
20 239 ms 241 ms 241 ms 199.91.189.26

Trace complete.

[Noshpan]

I live in California and I've been experiencing perpetual rubber-banding since the previous maintenance a could of days ago. At first I thought it was my settings, so I lowered everything to as much as possible, but there was no change in performance. There wasn't any change in performance when I turned off background running programs save my anti-virus and anti-malware programs. So I'm not even running Steam, the Battle.net launcher, not even Trillian.

I ran a tracert and here are my results:

Tracing route to traceip.ffxiv.com [202.67.52.81]
over a maximum of 50 hops:

1 2 ms 2 ms 2 ms 192.168.1.1
2 37 ms 30 ms 29 ms 98.192.180.1
3 11 ms 12 ms 10 ms te-7-2-ur01.hanford.ca.ccal.comcast.net [68.85.105.177]
4 18 ms 25 ms 17 ms te-8-1-ur01.vanness.ca.ccal.comcast.net [68.87.200.41]
5 24 ms 25 ms 21 ms 68.86.90.21
6 22 ms 22 ms 22 ms be-21-pe02.529bryant.ca.ibone.comcast.net [68.86.82.149]
7 48 ms 20 ms 21 ms 124.215.192.125
8 22 ms 23 ms 22 ms pajbb002.int-gw.kddi.ne.jp [111.87.3.69]
9 128 ms 128 ms 127 ms otejbb206.int-gw.kddi.ne.jp [203.181.100.205]
10 127 ms 138 ms 127 ms jc-ote301.int-gw.kddi.ne.jp [118.155.197.178]
11 139 ms 166 ms 138 ms 106.187.28.198
12 121 ms 136 ms 126 ms 61.195.56.133
13 121 ms 121 ms 122 ms 219.117.144.74
14 137 ms 138 ms 138 ms 219.117.144.45
15 131 ms 129 ms 129 ms 219.117.146.129
16 129 ms 129 ms 133 ms 219.117.146.38
17 131 ms 129 ms 129 ms 202.67.52.81

Trace complete.

[Lumerein]

I've noticed a similar issue from Virginia in the US on Gilgamesh (NA/EU). The past few times my latency has become particularly bad (sometimes coupled with Error 90000) I've done a pathping to the server IP shown by Resource Monitor. Each time I notice 5 - 20% or so packet loss at hops starting with those owned by Global Telecom & Technology (which apparently bought Tinet, the network showing in hops 7 - 9 below).

An example:

            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address

  1    1ms     0/ 100 =  0%     0/ 100 =  0%  192.168.1.1
                                0/ 100 =  0%   |
  2   10ms     0/ 100 =  0%     0/ 100 =  0%  96.120.18.41
                                0/ 100 =  0%   |
  3   11ms     0/ 100 =  0%     0/ 100 =  0%  te-1-4-ur01.frontroyal.va.richmond.comcast.net [68.86.126.41]
                                0/ 100 =  0%   |
  4   17ms     0/ 100 =  0%     0/ 100 =  0%  xe-5-0-2-0-ar02.charlvilleco.va.richmond.comcast.net [69.139.165.165]
                                0/ 100 =  0%   |
  5   18ms     0/ 100 =  0%     0/ 100 =  0%  pos-1-4-0-0-cr01.ashburn.va.ibone.comcast.net [68.86.95.21]
                                0/ 100 =  0%   |
  6   16ms     0/ 100 =  0%     0/ 100 =  0%  be-10-pe04.ashburn.va.ibone.comcast.net [68.86.84.202]
                                0/ 100 =  0%   |
  7   17ms     0/ 100 =  0%     0/ 100 =  0%  xe-7-0-2.was10.ip4.tinet.net [77.67.71.193]
                                0/ 100 =  0%   |
  8  113ms     5/ 100 =  5%     5/ 100 =  5%  xe-1-3-0.mtl10.ip4.tinet.net [89.149.184.74]
                                0/ 100 =  0%   |
  9  109ms    14/ 100 = 14%    14/ 100 = 14%  ormuco-gw.ip4.tinet.net [216.221.156.110]
                                0/ 100 =  0%   |
 10  113ms     3/ 100 =  3%     3/ 100 =  3%  192.34.76.2
                                0/ 100 =  0%   |
 11   35ms     0/ 100 =  0%     0/ 100 =  0%  199.91.189.234
                                1/ 100 =  1%   |
 12   34ms     1/ 100 =  1%     0/ 100 =  0%  199.91.189.24

This trace wasn't as bad as some of the others I've seen. I've seen 15 - 30% loss on three different hops before, usually starting at hop 8 or 9.

Tracing route to traceip.ffxiv.com [202.67.52.81]

Are you sure that's your server IP? Judging by your forum info, you play on Leviathan, which should be in the NA datacenter and probably have an IP in a similar range to Gilgamesh. However, there are .jp domains along the route, not to mention the traceip.ffxiv.com domain name there. While logged into the game on your character, open Resource Monitor, go to the Network tab, and check ffxiv.exe as a filter in the top panel. You should see two connections listed in the second panel, showing the IP address of your server in the Remote Address column. Try running a pathping to that IP.

[Noshpan]

Are you sure that's your server IP? Judging by your forum info, you play on Leviathan, which should be in the NA datacenter and probably have an IP in a similar range to Gilgamesh. However, there are .jp domains along the route, not to mention the traceip.ffxiv.com domain name there. While logged into the game on your character, open Resource Monitor, go to the Network tab, and check ffxiv.exe as a filter in the top panel. You should see two connections listed in the second panel, showing the IP address of your server in the Remote Address column. Try running a pathping to that IP.

I copied the wrong one and didn't realize it until now. ;^_^ Thanks for pointing that out. I ran it again to be more current.

Pathping

Tracing route to 199.91.189.46 over a maximum of 30 hops

  1  192.168.1.1
  2  98.192.180.1
  3  te-7-2-ur01.hanford.ca.ccal.comcast.net [68.85.105.177]
  4  te-8-1-ur01.vanness.ca.ccal.comcast.net [68.87.200.41]
  5  68.86.92.181
  6  pos-0-1-0-0-pe01.529bryant.ca.ibone.comcast.net [68.86.87.2]
  7  ix-2-3-0-0.tcore1.PDI-PaloAlto.as6453.net [66.198.127.17]
  8     *     if-2-2.tcore2.PDI-PaloAlto.as6453.net [66.198.127.2]
  9  if-11-3.tcore2.CT8-Chicago.as6453.net [66.198.144.58]
 10  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
 11  66.198.96.50
 12  192.34.76.2
 13  199.91.189.234
 14  199.91.189.46

Computing statistics for 350 seconds...

Tracert

Tracing route to 199.91.189.46 over a maximum of 30 hops

  1     1 ms     2 ms     2 ms  192.168.1.1
  2    22 ms    29 ms    30 ms  98.192.180.1
  3    12 ms    11 ms    13 ms  te-7-2-ur01.hanford.ca.ccal.comcast.net [68.85.105.177]
  4    18 ms    17 ms    20 ms  te-8-1-ur01.vanness.ca.ccal.comcast.net [68.87.200.41]
  5    25 ms    23 ms    23 ms  68.86.90.21
  6    41 ms    27 ms    26 ms  pos-0-1-0-0-pe01.529bryant.ca.ibone.comcast.net
[68.86.87.2]
  7    23 ms    22 ms    22 ms  ix-2-3-0-0.tcore1.PDI-PaloAlto.as6453.net [66.198.127.17]
  8     *      110 ms     *     if-2-2.tcore2.PDI-PaloAlto.as6453.net [66.198.127.2]
  9   123 ms   109 ms   114 ms  if-11-3.tcore2.CT8-Chicago.as6453.net [66.198.144.58]
 10   117 ms   110 ms   109 ms  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
 11    96 ms    96 ms    98 ms  66.198.96.50
 12    97 ms    96 ms    97 ms  192.34.76.2
 13    95 ms    97 ms    96 ms  199.91.189.234
 14    97 ms    96 ms    96 ms  199.91.189.46

Trace complete.

It's like once it leaves California, everything goes down the crapper. Mind you, I do have Comcast business class.

[Lumerein]

Yup, looks like you're probably experiencing a similar problem with latency spikes and dropped packets.

Also, in the future you should let pathping finish to get the most useful results for what it's meant to do. The "Computing statistics for 350 seconds..." line indicates that pathping is running a set of ping tests to each hop along the route. That way you can see if there are intermittent packet loss issues in certain places that a tracert or a short ping may not find. After that finishes, you'll get a table like the one I posted in my reply above, where you can see "x/100" dropped packets and a loss percentage for each hop.

[Noshpan]

Yup, looks like you're probably experiencing a similar problem with latency spikes and dropped packets.

Also, in the future you should let pathping finish to get the most useful results for what it's meant to do. The "Computing statistics for 350 seconds..." line indicates that pathping is running a set of ping tests to each hop along the route. That way you can see if there are intermittent packet loss issues in certain places that a tracert or a short ping may not find. After that finishes, you'll get a table like the one I posted in my reply above, where you can see "x/100" dropped packets and a loss percentage for each hop.

I waited 45 minutes for that to finish. It never did. I was like "Wellp, I'll just copy what I've been able to get since it's just gonna sit there with it's thumb up its backside."

[Raist]

may have gotten held up at that palo alto stop that suddenly went into full timeout on you---likely congestion reached the point at that hop to force it to ignore the echo requests, so you got hit with the long timeouts on each ping... and it was running 100 of them. With TATA's track record over the years, it may have started ignoring the requests at more of their hops you were going through as well, compounding the problem further.

[XainTheGreat]

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Users\Jeff>ping 184.107.107.176

Pinging 184.107.107.176 with 32 bytes of data:
Reply from 184.107.107.176: bytes=32 time=30ms TTL=52
Reply from 184.107.107.176: bytes=32 time=33ms TTL=52
Reply from 184.107.107.176: bytes=32 time=30ms TTL=52
Reply from 184.107.107.176: bytes=32 time=32ms TTL=52

Ping statistics for 184.107.107.176:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 30ms, Maximum = 33ms, Average = 31ms

C:\Users\Jeff>pathping 184.107.107.176

Tracing route to 184.107.107.176 over a maximum of 30 hops

0 JeffsBeast [192.168.0.3]
1 142.254.152.145
2 gig2-5.lkgnwilkgn.wi.rr.com [24.160.229.158]
3 tge1-7-0-9.milzwift-ccr01.midwest.rr.com [65.29.45.102]
4 ae-6-0.cr0.chi30.tbone.rr.com [66.109.6.206]
5 ae-2-0.a0.buf00.tbone.rr.com [66.109.1.42]
6 if-17-0.tcore1.CT8-Chicago.as6453.net [63.243.129.21]
7 if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
8 if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
9 66.198.96.58
10 te7-4.dr9.mtl.iweb.com [184.107.1.110]
11 72.55.128.44
12 184.107.107.176

Computing statistics for 300 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 JeffsBeast [192.168.0.3]
0/ 100 = 0% |
1 8ms 0/ 100 = 0% 0/ 100 = 0% 142.254.152.145
0/ 100 = 0% |
2 8ms 0/ 100 = 0% 0/ 100 = 0% gig2-5.lkgnwilkgn.wi.rr.com [24.16
0.229.158]
0/ 100 = 0% |
3 13ms 0/ 100 = 0% 0/ 100 = 0% tge1-7-0-9.milzwift-ccr01.midwest.
rr.com [65.29.45.102]
0/ 100 = 0% |
4 15ms 0/ 100 = 0% 0/ 100 = 0% ae-6-0.cr0.chi30.tbone.rr.com [66.
109.6.206]
0/ 100 = 0% |
5 14ms 0/ 100 = 0% 0/ 100 = 0% ae-2-0.a0.buf00.tbone.rr.com [66.1
09.1.42]
0/ 100 = 0% |
6 15ms 0/ 100 = 0% 0/ 100 = 0% if-17-0.tcore1.CT8-Chicago.as6453.
net [63.243.129.21]
0/ 100 = 0% |
7 14ms 0/ 100 = 0% 0/ 100 = 0% if-22-2.tcore2.CT8-Chicago.as6453.
net [64.86.79.1]
0/ 100 = 0% |
8 32ms 0/ 100 = 0% 0/ 100 = 0% if-3-2.tcore1.W6C-Montreal.as6453.
net [66.198.96.45]
0/ 100 = 0% |
9 33ms 99/ 100 = 99% 99/ 100 = 99% 66.198.96.58
0/ 100 = 0% |
10 --- 100/ 100 =100% 100/ 100 =100% te7-4.dr9.mtl.iweb.com [184.107.1.
110]
0/ 100 = 0% |
11 --- 100/ 100 =100% 100/ 100 =100% 72.55.128.44
0/ 100 = 0% |
12 30ms 0/ 100 = 0% 0/ 100 = 0% 184.107.107.176

Trace complete.

this right here is proof that it is not SE's fault, and im sick and tired of hearing people complain about the .3 positional update. Its clear that it is directly related to the clients ISP/location where they live. Ive said it a hundred times, i dont have problems dodging anything EVER, and its not SE's fault for any of the problems people claim to be having. It also seems like a lot of the people who claim to have these problems are located in the same states, which again, proves its not SE's fault

[whilke]

Microsoft Windows [Version 6.2.9200]
this right here is proof that it is not SE's fault, and im sick and tired of hearing people complain about the .3 positional update. Its clear that it is directly related to the clients ISP/location where they live. Ive said it a hundred times, i dont have problems dodging anything EVER, and its not SE's fault for any of the problems people claim to be having. It also seems like a lot of the people who claim to have these problems are located in the same states, which again, proves its not SE's fault

It is SE's issue to deal with. It's their network protocol that is unable to handle the internet. The issue is not that people are getting some pack loss or jitter spikes, it's that SE's game is unable to deal with those very common internet issues. This is also why people are not seeing problems with other games, they deal with it properly.

It comes down to this: Either SE fixes it, or people who are affected by this are going to stop paying them money. It's up to SE to decide what the cost benefit analysis for that is.