TATA Communications - the reason we're lagging

Been suffering lately from high latency, packet loss and disconnects? You're not alone. The issue seems to be TATA Communication routers - the CSP that most of us that lag use to reach the Montreal Datacenter. This is what a common tracert might look like to one of us.

1 <1 ms <1 ms <1 ms Voidzone [192.168.1.1]
2 1 ms 5 ms 1 ms gw-n1-u-d2.ias.bredband.telia.com [195.198.146.129]
3 3 ms 3 ms 3 ms s-b4-pos14-0.telia.net [213.248.101.142]
4 4 ms 4 ms 4 ms s-bb4-link.telia.net [213.155.131.32]
5 4 ms 4 ms 4 ms s-b3-link.telia.net [213.155.136.53]
6 4 ms 4 ms 4 ms vsnl-ic-124239-s-b1.c.telia.net [213.248.104.162]
7 283 ms 260 ms 260 ms if-7-1-3-0.tcore1.FR0-Frankfurt.as6453.net [195.219.50.145]
8 121 ms 119 ms 118 ms if-4-2.tcore1.PVU-Paris.as6453.net [80.231.153.10]
9 136 ms 137 ms 138 ms if-2-2.tcore1.PYE-Paris.as6453.net [80.231.154.18]
10 133 ms 141 ms 133 ms if-3-6.tcore1.L78-London.as6453.net [80.231.130.85]
11 133 ms 131 ms 129 ms if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
12 139 ms 137 ms 136 ms if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
13 146 ms * 138 ms if-2-2.tcore2.MTT-Montreal.as6453.net [64.86.226.13]
14 133 ms 134 ms 133 ms if-0-2.tcore1.MTT-Montreal.as6453.net [216.6.115.89]
15 134 ms * * if-5-2.tcore1.W6C-Montreal.as6453.net [64.86.31.6]
16 196 ms 196 ms * 66.198.96.50
17 209 ms 211 ms 213 ms 192.34.76.2
18 199 ms 207 ms 192 ms 199.91.189.234
19 209 ms * 211 ms 199.91.189.26

From hop 7 to 17 I'm using exclusively TATA Communication routers (as6453.net). Typically, the as6453.net backbone routers handle the load without issue. Even hop 7, which spiked during the tracert, showed a steady 30ms ping when I manually pinged it later.

The issue occurs at hop 16. A follow-up extended ping showed a steady 20% packetloss with high ping spikes. Funnily enough, this router, just like all the as6453.net routers, belongs to TATA Communications, but it's not part of the backbone. I believe it's one of many link routers that connects the Montreal Datacenter (starts at 192.34.76.2) to the TATA backbone, and is devoted specifically for the use of Ormuco Communications.

Obviously, they underestimated the amount of traffic going through this node, and it's suffering for it. They (meaning TATA Communications) need to split the load to other routers, or upgrade their hardware at this node. Sadly, it's nothing we endpoint customers can do anything about. TATA Communications work as a CSP, meaning they sell their services to ISP's. ISP's in turn sell their services to us, and to SQEX (The Montreal Datacenter uses Ormuco Communications as an ISP).

What we can do is lean on SQEX, who in turn can lean on Ormuco, who in turn can lean on TATA Communications, who are not providing the service they should. We can also lean on our respective ISP's who are employing TATA Communications to route their traffic to Montreal. The more feedback TATA gets about this issue, the more likely it's going to get resolved as soon as possible.

So send your support tickets, to SQEX and to your ISP, and try to include as much information as possible. When your problems occur, which router is responsible (use tracert and ping, or alternatively a third-party program like WinTrace to make things easier), and anything else you can think of. Unless we voice this issue we're going to get brushed off as a minority, or that the problem somehow is at our end.

I don't know if it helps, but I'm in Sweden too and my trace looked like this.

1 <1 ms 3 ms 1 ms router []
2 7 ms <1 ms <1 ms skovde1-ge-1-3-0.sunet.se [130.242.82.177]
3 42 ms 8 ms 8 ms t1tug-ge-7-3-2.sunet.se [130.242.82.117]
4 9 ms 8 ms 9 ms se-tug.nordu.net [109.105.102.17]
5 7 ms 7 ms 19 ms se-fre.nordu.net [109.105.97.1]
6 7 ms 7 ms 7 ms s-b3-link.telia.net [213.248.97.17]
7 10 ms 8 ms 7 ms vsnl-ic-124239-s-b1.c.telia.net [213.248.104.162]
8 140 ms 136 ms 145 ms if-7-1-3-0.tcore1.FR0-Frankfurt.as6453.net [195.219.50.145]
9 137 ms 138 ms 138 ms if-5-2.tcore1.PVU-Paris.as6453.net [80.231.153.121]
10 146 ms 145 ms 144 ms if-12-2.tcore1.PYE-Paris.as6453.net [80.231.154.69]
11 144 ms 144 ms 145 ms if-3-6.tcore1.L78-London.as6453.net [80.231.130.85]
12 143 ms 145 ms 147 ms if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
13 140 ms 141 ms 159 ms if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
14 157 ms 179 ms 153 ms if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
15 163 ms 167 ms 162 ms if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
16 161 ms 163 ms 162 ms if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
17 225 ms 234 ms 235 ms 66.198.96.50
18 232 ms 245 ms 230 ms 192.34.76.2
19 226 ms * 231 ms 199.91.189.234
20 239 ms 241 ms 241 ms 199.91.189.26

Trace complete.

I live in California and I've been experiencing perpetual rubber-banding since the previous maintenance a could of days ago. At first I thought it was my settings, so I lowered everything to as much as possible, but there was no change in performance. There wasn't any change in performance when I turned off background running programs save my anti-virus and anti-malware programs. So I'm not even running Steam, the Battle.net launcher, not even Trillian.

I ran a tracert and here are my results:

Tracing route to traceip.ffxiv.com [202.67.52.81]
over a maximum of 50 hops:

1 2 ms 2 ms 2 ms 192.168.1.1
2 37 ms 30 ms 29 ms 98.192.180.1
3 11 ms 12 ms 10 ms te-7-2-ur01.hanford.ca.ccal.comcast.net [68.85.105.177]
4 18 ms 25 ms 17 ms te-8-1-ur01.vanness.ca.ccal.comcast.net [68.87.200.41]
5 24 ms 25 ms 21 ms 68.86.90.21
6 22 ms 22 ms 22 ms be-21-pe02.529bryant.ca.ibone.comcast.net [68.86.82.149]
7 48 ms 20 ms 21 ms 124.215.192.125
8 22 ms 23 ms 22 ms pajbb002.int-gw.kddi.ne.jp [111.87.3.69]
9 128 ms 128 ms 127 ms otejbb206.int-gw.kddi.ne.jp [203.181.100.205]
10 127 ms 138 ms 127 ms jc-ote301.int-gw.kddi.ne.jp [118.155.197.178]
11 139 ms 166 ms 138 ms 106.187.28.198
12 121 ms 136 ms 126 ms 61.195.56.133
13 121 ms 121 ms 122 ms 219.117.144.74
14 137 ms 138 ms 138 ms 219.117.144.45
15 131 ms 129 ms 129 ms 219.117.146.129
16 129 ms 129 ms 133 ms 219.117.146.38
17 131 ms 129 ms 129 ms 202.67.52.81

Trace complete.

I've noticed a similar issue from Virginia in the US on Gilgamesh (NA/EU). The past few times my latency has become particularly bad (sometimes coupled with Error 90000) I've done a pathping to the server IP shown by Resource Monitor. Each time I notice 5 - 20% or so packet loss at hops starting with those owned by Global Telecom & Technology (which apparently bought Tinet, the network showing in hops 7 - 9 below).

An example:

Code:

Source to Here This Node/Link Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address 1 1ms 0/ 100 = 0% 0/ 100 = 0% 192.168.1.1 0/ 100 = 0% | 2 10ms 0/ 100 = 0% 0/ 100 = 0% 96.120.18.41 0/ 100 = 0% | 3 11ms 0/ 100 = 0% 0/ 100 = 0% te-1-4-ur01.frontroyal.va.richmond.comcast.net [68.86.126.41] 0/ 100 = 0% | 4 17ms 0/ 100 = 0% 0/ 100 = 0% xe-5-0-2-0-ar02.charlvilleco.va.richmond.comcast.net [69.139.165.165] 0/ 100 = 0% | 5 18ms 0/ 100 = 0% 0/ 100 = 0% pos-1-4-0-0-cr01.ashburn.va.ibone.comcast.net [68.86.95.21] 0/ 100 = 0% | 6 16ms 0/ 100 = 0% 0/ 100 = 0% be-10-pe04.ashburn.va.ibone.comcast.net [68.86.84.202] 0/ 100 = 0% | 7 17ms 0/ 100 = 0% 0/ 100 = 0% xe-7-0-2.was10.ip4.tinet.net [77.67.71.193] 0/ 100 = 0% | 8 113ms 5/ 100 = 5% 5/ 100 = 5% xe-1-3-0.mtl10.ip4.tinet.net [89.149.184.74] 0/ 100 = 0% | 9 109ms 14/ 100 = 14% 14/ 100 = 14% ormuco-gw.ip4.tinet.net [216.221.156.110] 0/ 100 = 0% | 10 113ms 3/ 100 = 3% 3/ 100 = 3% 192.34.76.2 0/ 100 = 0% | 11 35ms 0/ 100 = 0% 0/ 100 = 0% 199.91.189.234 1/ 100 = 1% | 12 34ms 1/ 100 = 1% 0/ 100 = 0% 199.91.189.24

This trace wasn't as bad as some of the others I've seen. I've seen 15 - 30% loss on three different hops before, usually starting at hop 8 or 9.

Quote:

Originally Posted by Noshpan

Tracing route to traceip.ffxiv.com [202.67.52.81]

Are you sure that's your server IP? Judging by your forum info, you play on Leviathan, which should be in the NA datacenter and probably have an IP in a similar range to Gilgamesh. However, there are .jp domains along the route, not to mention the traceip.ffxiv.com domain name there. While logged into the game on your character, open Resource Monitor, go to the Network tab, and check ffxiv.exe as a filter in the top panel. You should see two connections listed in the second panel, showing the IP address of your server in the Remote Address column. Try running a pathping to that IP.

Quote:

Originally Posted by Lumerein

Are you sure that's your server IP? Judging by your forum info, you play on Leviathan, which should be in the NA datacenter and probably have an IP in a similar range to Gilgamesh. However, there are .jp domains along the route, not to mention the traceip.ffxiv.com domain name there. While logged into the game on your character, open Resource Monitor, go to the Network tab, and check ffxiv.exe as a filter in the top panel. You should see two connections listed in the second panel, showing the IP address of your server in the Remote Address column. Try running a pathping to that IP.

I copied the wrong one and didn't realize it until now. ;^_^ Thanks for pointing that out. I ran it again to be more current.

Pathping

Code:

Tracing route to 199.91.189.46 over a maximum of 30 hops 1 192.168.1.1 2 98.192.180.1 3 te-7-2-ur01.hanford.ca.ccal.comcast.net [68.85.105.177] 4 te-8-1-ur01.vanness.ca.ccal.comcast.net [68.87.200.41] 5 68.86.92.181 6 pos-0-1-0-0-pe01.529bryant.ca.ibone.comcast.net [68.86.87.2] 7 ix-2-3-0-0.tcore1.PDI-PaloAlto.as6453.net [66.198.127.17] 8 * if-2-2.tcore2.PDI-PaloAlto.as6453.net [66.198.127.2] 9 if-11-3.tcore2.CT8-Chicago.as6453.net [66.198.144.58] 10 if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45] 11 66.198.96.50 12 192.34.76.2 13 199.91.189.234 14 199.91.189.46 Computing statistics for 350 seconds...

Tracert

Code:

Tracing route to 199.91.189.46 over a maximum of 30 hops 1 1 ms 2 ms 2 ms 192.168.1.1 2 22 ms 29 ms 30 ms 98.192.180.1 3 12 ms 11 ms 13 ms te-7-2-ur01.hanford.ca.ccal.comcast.net [68.85.105.177] 4 18 ms 17 ms 20 ms te-8-1-ur01.vanness.ca.ccal.comcast.net [68.87.200.41] 5 25 ms 23 ms 23 ms 68.86.90.21 6 41 ms 27 ms 26 ms pos-0-1-0-0-pe01.529bryant.ca.ibone.comcast.net [68.86.87.2] 7 23 ms 22 ms 22 ms ix-2-3-0-0.tcore1.PDI-PaloAlto.as6453.net [66.198.127.17] 8 * 110 ms * if-2-2.tcore2.PDI-PaloAlto.as6453.net [66.198.127.2] 9 123 ms 109 ms 114 ms if-11-3.tcore2.CT8-Chicago.as6453.net [66.198.144.58] 10 117 ms 110 ms 109 ms if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45] 11 96 ms 96 ms 98 ms 66.198.96.50 12 97 ms 96 ms 97 ms 192.34.76.2 13 95 ms 97 ms 96 ms 199.91.189.234 14 97 ms 96 ms 96 ms 199.91.189.46 Trace complete.

It's like once it leaves California, everything goes down the crapper. Mind you, I do have Comcast business class.

Yup, looks like you're probably experiencing a similar problem with latency spikes and dropped packets.

Also, in the future you should let pathping finish to get the most useful results for what it's meant to do. The "Computing statistics for 350 seconds..." line indicates that pathping is running a set of ping tests to each hop along the route. That way you can see if there are intermittent packet loss issues in certain places that a tracert or a short ping may not find. After that finishes, you'll get a table like the one I posted in my reply above, where you can see "x/100" dropped packets and a loss percentage for each hop. :)

Quote:

Originally Posted by Lumerein

Yup, looks like you're probably experiencing a similar problem with latency spikes and dropped packets.

Also, in the future you should let pathping finish to get the most useful results for what it's meant to do. The "Computing statistics for 350 seconds..." line indicates that pathping is running a set of ping tests to each hop along the route. That way you can see if there are intermittent packet loss issues in certain places that a tracert or a short ping may not find. After that finishes, you'll get a table like the one I posted in my reply above, where you can see "x/100" dropped packets and a loss percentage for each hop. :)

I waited 45 minutes for that to finish. It never did. I was like "Wellp, I'll just copy what I've been able to get since it's just gonna sit there with it's thumb up its backside."

may have gotten held up at that palo alto stop that suddenly went into full timeout on you---likely congestion reached the point at that hop to force it to ignore the echo requests, so you got hit with the long timeouts on each ping... and it was running 100 of them. With TATA's track record over the years, it may have started ignoring the requests at more of their hops you were going through as well, compounding the problem further.

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Users\Jeff>ping 184.107.107.176

Pinging 184.107.107.176 with 32 bytes of data:
Reply from 184.107.107.176: bytes=32 time=30ms TTL=52
Reply from 184.107.107.176: bytes=32 time=33ms TTL=52
Reply from 184.107.107.176: bytes=32 time=30ms TTL=52
Reply from 184.107.107.176: bytes=32 time=32ms TTL=52

Ping statistics for 184.107.107.176:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 30ms, Maximum = 33ms, Average = 31ms

C:\Users\Jeff>pathping 184.107.107.176

Tracing route to 184.107.107.176 over a maximum of 30 hops

0 JeffsBeast [192.168.0.3]
1 142.254.152.145
2 gig2-5.lkgnwilkgn.wi.rr.com [24.160.229.158]
3 tge1-7-0-9.milzwift-ccr01.midwest.rr.com [65.29.45.102]
4 ae-6-0.cr0.chi30.tbone.rr.com [66.109.6.206]
5 ae-2-0.a0.buf00.tbone.rr.com [66.109.1.42]
6 if-17-0.tcore1.CT8-Chicago.as6453.net [63.243.129.21]
7 if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
8 if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
9 66.198.96.58
10 te7-4.dr9.mtl.iweb.com [184.107.1.110]
11 72.55.128.44
12 184.107.107.176

Computing statistics for 300 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 JeffsBeast [192.168.0.3]
0/ 100 = 0% |
1 8ms 0/ 100 = 0% 0/ 100 = 0% 142.254.152.145
0/ 100 = 0% |
2 8ms 0/ 100 = 0% 0/ 100 = 0% gig2-5.lkgnwilkgn.wi.rr.com [24.16
0.229.158]
0/ 100 = 0% |
3 13ms 0/ 100 = 0% 0/ 100 = 0% tge1-7-0-9.milzwift-ccr01.midwest.
rr.com [65.29.45.102]
0/ 100 = 0% |
4 15ms 0/ 100 = 0% 0/ 100 = 0% ae-6-0.cr0.chi30.tbone.rr.com [66.
109.6.206]
0/ 100 = 0% |
5 14ms 0/ 100 = 0% 0/ 100 = 0% ae-2-0.a0.buf00.tbone.rr.com [66.1
09.1.42]
0/ 100 = 0% |
6 15ms 0/ 100 = 0% 0/ 100 = 0% if-17-0.tcore1.CT8-Chicago.as6453.
net [63.243.129.21]
0/ 100 = 0% |
7 14ms 0/ 100 = 0% 0/ 100 = 0% if-22-2.tcore2.CT8-Chicago.as6453.
net [64.86.79.1]
0/ 100 = 0% |
8 32ms 0/ 100 = 0% 0/ 100 = 0% if-3-2.tcore1.W6C-Montreal.as6453.
net [66.198.96.45]
0/ 100 = 0% |
9 33ms 99/ 100 = 99% 99/ 100 = 99% 66.198.96.58
0/ 100 = 0% |
10 --- 100/ 100 =100% 100/ 100 =100% te7-4.dr9.mtl.iweb.com [184.107.1.
110]
0/ 100 = 0% |
11 --- 100/ 100 =100% 100/ 100 =100% 72.55.128.44
0/ 100 = 0% |
12 30ms 0/ 100 = 0% 0/ 100 = 0% 184.107.107.176

Trace complete.

this right here is proof that it is not SE's fault, and im sick and tired of hearing people complain about the .3 positional update. Its clear that it is directly related to the clients ISP/location where they live. Ive said it a hundred times, i dont have problems dodging anything EVER, and its not SE's fault for any of the problems people claim to be having. It also seems like a lot of the people who claim to have these problems are located in the same states, which again, proves its not SE's fault

Quote:

Originally Posted by XainTheGreat

Microsoft Windows [Version 6.2.9200]
this right here is proof that it is not SE's fault, and im sick and tired of hearing people complain about the .3 positional update. Its clear that it is directly related to the clients ISP/location where they live. Ive said it a hundred times, i dont have problems dodging anything EVER, and its not SE's fault for any of the problems people claim to be having. It also seems like a lot of the people who claim to have these problems are located in the same states, which again, proves its not SE's fault

It is SE's issue to deal with. It's their network protocol that is unable to handle the internet. The issue is not that people are getting some pack loss or jitter spikes, it's that SE's game is unable to deal with those very common internet issues. This is also why people are not seeing problems with other games, they deal with it properly.

It comes down to this: Either SE fixes it, or people who are affected by this are going to stop paying them money. It's up to SE to decide what the cost benefit analysis for that is.

Quote:

Originally Posted by whilke

It is SE's issue to deal with. It's their network protocol that is unable to handle the internet. The issue is not that people are getting some pack loss or jitter spikes, it's that SE's game is unable to deal with those very common internet issues. This is also why people are not seeing problems with other games, they deal with it properly.

It comes down to this: Either SE fixes it, or people who are affected by this are going to stop paying them money. It's up to SE to decide what the cost benefit analysis for that is.

You clearly don't understand how the internet works. It's a massive series of pipelines with all the data passing through routers to move all the internet traffic around. All it takes is 1 router to route data incorrectly or to be overloaded for a large swath of the internet to have issues. This is specifically obvious in cases such as this one. Most traffic must pass through node A to reach a ARR. However node A isn't working correctly. It is now perceived that the service is faulty when in fact it was the node the entire time. It has absolutely nothing what-so-ever to do with SE's network protocol or the server protocol used. It is entirely due to that faulty node. Once the node is repaired normal functionality will return. It may be argued that Square should take up the mantle and confront the company that owns this router in order to achieve a resolution, but that doesn't explicitly make it their fault either.

Quote:

Originally Posted by whilke

BLAME SE!!!!1!

Quote:

Originally Posted by Tiggy

You clearly don't understand how the internet works.

I don't think he knows how life works.

"Theres a problem, i can't fix it... BETTER BLAME SOMEONE!"

And i understand SE should really come out and say something so that people understand that user connections to their server isn't a cup phone with a string attached from their modem to their datacenters. Thats the only thing i hold SE accountable for. Not communicating why this problem exists, even though im sure their working on their side so that they have a definitive answer because people will unsub because of these problems. Not knowing its not entirely SE's fault.

But the problem here is that they have to obtain a definitive answer since they just can't say "Blame your ISP for your horrible internet."

There are also a couple threads where some guy explains the problem in detail showing his own Tracerts, outlining that TATA (whatever that is) is the middle man possibly causing these issues.

its SE fault for getting a terrible provider

Quote:

Originally Posted by Thunderz

its SE fault for getting a terrible provider

You can't be too picky when it comes to ISPs. You are limited by your location since there isn't much demand for multiple large ISPs in many areas. As far as you know this was the only option for them.

Quote:

Originally Posted by XainTheGreat

this right here is proof that it is not SE's fault, and im sick and tired of hearing people complain about the .3 positional update. Its clear that it is directly related to the clients ISP/location where they live. Ive said it a hundred times, i dont have problems dodging anything EVER, and its not SE's fault for any of the problems people claim to be having. It also seems like a lot of the people who claim to have these problems are located in
the same states, which again, proves its not SE's fault

Your tracert doesn't prove a thing, you're not even using the same route (have a look at the last ip-addresses of as6453.net for example) your route could be completely fine while the next person takes a slightly different route and ends up with a congested router. You're not even doing a tracert of any of the game server's by the way, that IP is used by http://iweb.com/ (using the same data center by the looks of it).

I'm sick and tired of people stating that they have no lag issue and that there is nothing wrong and that people should check their PCs / network equipment / ISPs and what not to find a solution, talk about being clueless.

It is not SE's faulty directly but they will have to communicate with their ISP and their partners, in this case TATA (i'm personally directed through cogentco and have the exact same problem as others do here) to solve the routing issues which are clearly evident in this case. The lack of communication from SE for over two months now while the issue persists to me and i guess most of the people having the problem is that SE does not give a damn about it (maybe they do and try their best to sort it out but then why not communicate it and instead leave people in the dark, which is indeed very unprofessional).

I can personally get rid of the issue by using a tunneling service and will be using it till the issue is solved or i get tired of having to pay extra to experience a lagfree game, sadly tho this is not a solution for everyone.

Quote:

Originally Posted by Chizzum

The lack of communication from SE for over two months now while the issue persists to me and i guess most of the people having the problem is that SE does not give a damn about it (maybe they do and try their best to sort it out but then why not communicate it and instead leave people in the dark, which is indeed very unprofessional).

Threads like this http://forum.square-enix.com/ffxiv/t...P2P-throttling make it clear that they are aware of the problem and are already looking into solutions with the help of the player base. I don't think them telling us "We are aware of the problem and are currently looking for a solution. We deeply apologize for the inconvenience." would make people much happier.

SE..

Please give TATA a new router for Christmas. They have been a good boy/girl.

Tyvm.

Love,
-allofus whopayyou

Greetings,

It seems that there have not been any further posts about this concern being an ongoing issue. Because of this, we are considering it to be resolved. If this does continue to be a concern, please create another thread to further discuss resolutions or contact the Support Center at support.na.square-enix.com for assistance.