Issue of all AT&T Uverse lag possibly identified - SE Confirm

[Docfiord_Fowling]

So, after doing a few investigation threads on the issue, one IP address in particular keeps coming up in many people's trace routes to their servers :

192.34.76.2

Information on this IP suggests it is in Monteral, Canada.

Here is a sample of my traceroute. You can see as soon as it hits that IP, the latecy rises significantly.

 1:  192.168.1.103                                         0.115ms pmtu 1500
 1:  192.168.1.254                                         0.886ms 
 1:  192.168.1.254                                         0.676ms 
 2:  172.6.220.2                                          25.639ms 
 3:  76.201.208.74                                        24.954ms 
 4:  76.201.208.125                                       30.000ms 
 5:  12.83.82.145                                         25.087ms 
 6:  12.122.141.233                                       28.614ms asymm  7 
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  no reply
14:  4.69.141.5                                           79.500ms asymm 13 
15:  4.69.141.1                                           82.823ms asymm 14 
16:  4.59.178.74                                          79.945ms asymm 15 
17:  no reply
18:  192.34.76.2                                         105.938ms asymm 14 
19:  no reply
20:  no reply
21:  no reply
22:  no reply
23:  no reply
24:  no reply
25:  no reply
26:  no reply
27:  no reply
28:  no reply
29:  no reply
30:  no reply
31:  no reply
     Too many hops: pmtu 1500
     Resume: pmtu 1500

Others' traceroute results show the same and can be found from the VErizon FiOS link below as well as this Reddit post:

http://forums.verizon.com/t5/FiOS-In.../622913/page/2

http://www.reddit.com/r/ffxiv/commen...rse_featuring/

[HidingYoshis]

I can confirm on my end that this happens:

3 21 ms 24 ms 21 ms 75.26.64.42
4 21 ms 21 ms 20 ms 75.26.64.201
5 22 ms 23 ms 23 ms 12.83.32.133
6 44 ms 30 ms 31 ms ggr4.cgcil.ip.att.net [12.122.133.33]
7 29 ms 30 ms 29 ms 192.205.37.150
8 84 ms 83 ms 84 ms vl-3508-ve-122.ebr1.Chicago2.Level3.net [4.69.15
8.142]
9 * * * Request timed out.
10 116 ms 93 ms 70 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]

11 64 ms 65 ms 66 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
12 * * * Request timed out.
13 153 ms 153 ms 149 ms 192.34.76.2
14 146 ms 137 ms 141 ms 199.91.189.234
15 154 ms 153 ms 152 ms 199.91.189.40

As you said, 192.34.76.2 appears to be the cause of my problems.

I hope Square Enix actually sees AND does something about this.

[Docfiord_Fowling]

So far it seems AT&T, Verizon FiOS, and Canada Bell are all having this issue and they all run through that hop. SE, please look into this if you have any ontrol over that address space.

[Squa11_Leonhart]

So far it seems AT&T, Verizon FiOS, and Canada Bell are all having this issue and they all run through that hop. SE, please look into this if you have any ontrol over that address space.

they don't - its a third party peering network

[Docfiord_Fowling]

they don't - its a third party peering network

Interesting. Mind sharing how you came about this information?

[Raist]

So, after doing a few investigation threads on the issue, one IP address in particular keeps coming up in many people's trace routes to their servers :

192.34.76.2

Information on this IP suggests it is in Monteral, Canada.

Here is a sample of my traceroute. You can see as soon as it hits that IP, the latecy rises significantly.

 1:  192.168.1.103                                         0.115ms pmtu 1500
 1:  192.168.1.254                                         0.886ms 
 1:  192.168.1.254                                         0.676ms 
 2:  172.6.220.2                                          25.639ms 
 3:  76.201.208.74                                        24.954ms 
 4:  76.201.208.125                                       30.000ms 
 5:  12.83.82.145                                         25.087ms 
 6:  12.122.141.233                                       28.614ms asymm  7 
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  no reply
14:  4.69.141.5                                           79.500ms asymm 13 
15:  4.69.141.1                                           82.823ms asymm 14 
16:  4.59.178.74                                          79.945ms asymm 15 
17:  no reply
18:  192.34.76.2                                         105.938ms asymm 14 
19:  no reply
20:  no reply
21:  no reply
22:  no reply
23:  no reply
24:  no reply
25:  no reply
26:  no reply
27:  no reply
28:  no reply
29:  no reply
30:  no reply
31:  no reply
     Too many hops: pmtu 1500
     Resume: pmtu 1500

Others' traceroute results show the same and can be found from the VErizon FiOS link below as well as this Reddit post:

http://forums.verizon.com/t5/FiOS-In.../622913/page/2

http://www.reddit.com/r/ffxiv/commen...rse_featuring/

I can confirm on my end that this happens:

3 21 ms 24 ms 21 ms 75.26.64.42
4 21 ms 21 ms 20 ms 75.26.64.201
5 22 ms 23 ms 23 ms 12.83.32.133
6 44 ms 30 ms 31 ms ggr4.cgcil.ip.att.net [12.122.133.33]
7 29 ms 30 ms 29 ms 192.205.37.150
8 84 ms 83 ms 84 ms vl-3508-ve-122.ebr1.Chicago2.Level3.net [4.69.15
8.142]
9 * * * Request timed out.
10 116 ms 93 ms 70 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]

11 64 ms 65 ms 66 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
12 * * * Request timed out.
13 153 ms 153 ms 149 ms 192.34.76.2
14 146 ms 137 ms 141 ms 199.91.189.234
15 154 ms 153 ms 152 ms 199.91.189.40

As you said, 192.34.76.2 appears to be the cause of my problems.

I hope Square Enix actually sees AND does something about this.

Interesting. Mind sharing how you came about this information?

Actually, if you look more closely, you will find the typical signs of congestion issues occurring earlier in the routes that may be at fault that need to be looked into first. For instance, look at the wide variance at Level3's hop shortly after coming around Chicago:

10 116 ms 93 ms 70 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]

That's a swing of over 65% variance in a short time frame. If this is occurring regularly, it is a sign of pending (or even actual) overload either at the IPX or coming through Level3's segments. Such an issue compounds itself the further you go down the route. The fact that that hop further down is having a much more consistent response time is actually encouraging--it means it is doing a better job of maintaining traffic flow. The fact the response times are markedly high could very well just be because the shaping policies in effect have put the ICMP ECHO request on a lower priority, which means it is working as intended--slowing down less important traffic to reserve capacity for higher priority traffic. This is done to try to maintain more consistent flow--which clearly is not happening at the other hop.

Some other examples from one of the linked threads:

3 25 ms 8 ms 11 ms G0-6-4-6.WASHDC-LCR-22.verizon-gni.net [130.81.212.144]

4 33 ms 11 ms 14 ms xe-1-1-1-0.PHIL-BB-RTR2.verizon-gni.net [130.81.209.18]

so-9-0-2-0.RES-BB-RTR1.verizon-gni.net 0 150 150 7 18 100 9
(best time of 7, average of 18, worst of 100)

And this one is a REAL doozy:

Host	%	Sent	Recv	Best	Avrg	Wrst	Last
Wireless_Broadband_Router.home	0	500	500	0	0	1	0
L100.BLTMMD-VFTTP-52.verizon-gni.net	0	500	500	5	7	114	7
G5-0-352.BLTMMD-LCR-03.verizon-gni.net	0	500	500	5	7	118	7
so-9-0-2-0.RES-BB-RTR1.verizon-gni.net	0	500	500	7	18	176	99
0.ae5.XL1.IAD8.ALTER.NET	3	457	446	7	10	48	42
0.xe-10-1-1.GW9.IAD8.ALTER.NET	0	500	500	7	11	123	12
tinet-gw.customer.alter.net	5	421	401	15	19	97	17
xe-1-0-0.mtl10.ip4.tinet.net	0	500	500	25	37	184	37
ormuco-gw.ip4.tinet.net	0	500	500	42	45	148	43
192.34.76.2	0	500	500	42	46	147	45
199.91.189.234	1	496	495	40	45	170	41
199.91.189.30	0	500	500	40	43	147	41

In that sample, it's ALL on Verizon and their Alter.net subsidiary before it even get to TI-Net, which is another offender at times as well.

The root of this is indeed (in the majority of cases) an issue of flow control, compounded by shoddy/outdated peering agreements between our last mile ISP's and their routing partners. It has been discussed over and over on various forums for years (not just XIV... I remember it going back to the early days of WoW, Diablo, and even Freespace: Descent). Articles have been published and briefs/complaints filed with the regulatory commissions trying to get the issue addressed. One such blog post was put up by Level 3 this past year (you can find more on the topic by simply googling "level3.net chicken", or follow the links that crop up with that blog post.) TATA also filed briefs on similar observations about 6 or 7 years ago. There was a big kerfuffle over Verizon, Comcast, and Netflix in more recent history. It's all bound to an issue with the ISP's cutting corners and trying to make more money. All the while fleecing us for all we are worth while providing us with sub-par service.

Need to hold your ISP's feet to the fire to address this. They have the resources to address the issue--need to get their Tier3 people involved so they can properly investigate the issues and push for the escalation to get them addressed.

It should be noted that a few months back, while working with my ISP's T3 support, we spotted an issue on the other side of Cogent's transit in a VLAN managed by Ormuco. It was routinely spiking upwards of 280ms from it's norm of around 100ms. We went straight to them and got it addressed without getting SE involved. This CAN and SHOULD be handled through your ISP, since you are paying them for the service of routing you across the internet, and THEIR routing policies are what is keeping you on these troublesome routes. If they cannot get the responsible party to address the problem directly, they have the option of changing you to alternate routing partners to try to find a cleaner route. To that end, TWC changes my route every few weeks trying to stay ahead of the congested hops. This is done entirely through my ISP's T3 support... no one else--I just email them with my reports, and they take care of it. Sometimes they catch before I do... kind of annoying when my modem bounces to re-align my channels when I am in game, but at least I am able to log right back in a few minutes later and continue playing with minimal issues.


Edit:
After eating dinner, I ran a trace from the back of the house (from laptop, on Wifi from about 30 feet away), and from here in South Carolina, at 7:30PM... trace is clean as it usually is. I don't experience this horrible lag you speak of, and I am going through that very same hop---but it is giving me mostly right around 60ms response times (rarely breaks 80 even under the busiest of times unless something is up along my route, which my ISP usually addresses pretty quickly now that I have gotten them to take action).

C:\Windows\System32>tracert 199.91.189.30

Tracing route to 199.91.189.30 over a maximum of 30 hops

  1     2 ms     1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    37 ms    26 ms    28 ms  cpe-066-026-112-001.sc.res.rr.com [66.26.112.1]
  3    17 ms    24 ms    29 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    13 ms    16 ms    14 ms  clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
  5    28 ms    25 ms    27 ms  be33.drhmncev01r.southeast.rr.com [24.93.64.180]
  6    32 ms    34 ms    35 ms  107.14.19.20
  7    33 ms    43 ms    33 ms  so-1-1-1.c1.buf00.tbone.rr.com [66.109.1.113]
  8    32 ms    31 ms    32 ms  ix-17-0.tcore2.AEQ-Ashburn.as6453.net [216.6.87.149]
  9    62 ms    64 ms    61 ms  if-11-2.tcore2.NJY-Newark.as6453.net [216.6.87.138]
 10    62 ms    76 ms    62 ms  if-4-4.tcore2.NYY-New-York.as6453.net [66.198.111.18]
 11    59 ms    60 ms    62 ms  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
 12    58 ms    61 ms    60 ms  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
 13    60 ms    69 ms    61 ms  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
 14    59 ms    73 ms    61 ms  66.198.96.50
 15    62 ms    62 ms    64 ms  192.34.76.2
 16    58 ms    59 ms    60 ms  199.91.189.234
 17    62 ms    62 ms    59 ms  199.91.189.30

Trace complete.

Ping statistics for 199.91.189.30:
    Packets: Sent = 30, Received = 30, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 58ms, Maximum = 69ms, Average = 60ms

[Docfiord_Fowling]

Actually, if you look more closely, you will find the typical signs of congestion issues occurring earlier in the routes that may be at fault that need to be looked into first. For instance, look at the wide variance at Level3's hop shortly after coming around Chicago:

10 116 ms 93 ms 70 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]

That's a swing of over 65% variance in a short time frame. If this is occurring regularly, it is a sign of pending (or even actual) overload either at the IPX or coming through Level3's segments. Such an issue compounds itself the further you go down the route. The fact that that hop further down is having a much more consistent response time is actually encouraging--it means it is doing a better job of maintaining traffic flow. The fact the response times are markedly high could very well just be because the shaping policies in effect have put the ICMP ECHO request on a lower priority, which means it is working as intended--slowing down less important traffic to reserve capacity for higher priority traffic. This is done to try to maintain more consistent flow--which clearly is not happening at the other hop.

Some other examples from one of the linked threads:

3 25 ms 8 ms 11 ms G0-6-4-6.WASHDC-LCR-22.verizon-gni.net [130.81.212.144]

4 33 ms 11 ms 14 ms xe-1-1-1-0.PHIL-BB-RTR2.verizon-gni.net [130.81.209.18]

so-9-0-2-0.RES-BB-RTR1.verizon-gni.net 0 150 150 7 18 100 9
(best time of 7, average of 18, worst of 100)

And this one is a REAL doozy:

Host	%	Sent	Recv	Best	Avrg	Wrst	Last
Wireless_Broadband_Router.home	0	500	500	0	0	1	0
L100.BLTMMD-VFTTP-52.verizon-gni.net	0	500	500	5	7	114	7
G5-0-352.BLTMMD-LCR-03.verizon-gni.net	0	500	500	5	7	118	7
so-9-0-2-0.RES-BB-RTR1.verizon-gni.net	0	500	500	7	18	176	99
0.ae5.XL1.IAD8.ALTER.NET	3	457	446	7	10	48	42
0.xe-10-1-1.GW9.IAD8.ALTER.NET	0	500	500	7	11	123	12
tinet-gw.customer.alter.net	5	421	401	15	19	97	17
xe-1-0-0.mtl10.ip4.tinet.net	0	500	500	25	37	184	37
ormuco-gw.ip4.tinet.net	0	500	500	42	45	148	43
192.34.76.2	0	500	500	42	46	147	45
199.91.189.234	1	496	495	40	45	170	41
199.91.189.30	0	500	500	40	43	147	41

In that sample, it's ALL on Verizon and their Alter.net subsidiary before it even get to TI-Net, which is another offender at times as well.

The root of this is indeed (in the majority of cases) an issue of flow control, compounded by shoddy/outdated peering agreements between our last mile ISP's and their routing partners. It has been discussed over and over on various forums for years (not just XIV... I remember it going back to the early days of WoW, Diablo, and even Freespace: Descent). Articles have been published and briefs/complaints filed with the regulatory commissions trying to get the issue addressed. One such blog post was put up by Level 3 this past year (you can find more on the topic by simply googling "level3.net chicken", or follow the links that crop up with that blog post.) TATA also filed briefs on similar observations about 6 or 7 years ago. There was a big kerfuffle over Verizon, Comcast, and Netflix in more recent history. It's all bound to an issue with the ISP's cutting corners and trying to make more money. All the while fleecing us for all we are worth while providing us with sub-par service.

Need to hold your ISP's feet to the fire to address this. They have the resources to address the issue--need to get their Tier3 people involved so they can properly investigate the issues and push for the escalation to get them addressed.

It should be noted that a few months back, while working with my ISP's T3 support, we spotted an issue on the other side of Cogent's transit in a VLAN managed by Ormuco. It was routinely spiking upwards of 280ms from it's norm of around 100ms. We went straight to them and got it addressed without getting SE involved. This CAN and SHOULD be handled through your ISP, since you are paying them for the service of routing you across the internet, and THEIR routing policies are what is keeping you on these troublesome routes. If they cannot get the responsible party to address the problem directly, they have the option of changing you to alternate routing partners to try to find a cleaner route. To that end, TWC changes my route every few weeks trying to stay ahead of the congested hops. This is done entirely through my ISP's T3 support... no one else--I just email them with my reports, and they take care of it. Sometimes they catch before I do... kind of annoying when my modem bounces to re-align my channels when I am in game, but at least I am able to log right back in a few minutes later and continue playing with minimal issues.


Edit:
After eating dinner, I ran a trace from the back of the house (from laptop, on Wifi from about 30 feet away), and from here in South Carolina, at 7:30PM... trace is clean as it usually is. I don't experience this horrible lag you speak of, and I am going through that very same hop---but it is giving me mostly right around 60ms response times (rarely breaks 80 even under the busiest of times unless something is up along my route, which my ISP usually addresses pretty quickly now that I have gotten them to take action).

C:\Windows\System32>tracert 199.91.189.30

Tracing route to 199.91.189.30 over a maximum of 30 hops

  1     2 ms     1 ms    <1 ms  LPTSRV [10.10.100.1]
  2    37 ms    26 ms    28 ms  cpe-066-026-112-001.sc.res.rr.com [66.26.112.1]
  3    17 ms    24 ms    29 ms  cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
  4    13 ms    16 ms    14 ms  clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
  5    28 ms    25 ms    27 ms  be33.drhmncev01r.southeast.rr.com [24.93.64.180]
  6    32 ms    34 ms    35 ms  107.14.19.20
  7    33 ms    43 ms    33 ms  so-1-1-1.c1.buf00.tbone.rr.com [66.109.1.113]
  8    32 ms    31 ms    32 ms  ix-17-0.tcore2.AEQ-Ashburn.as6453.net [216.6.87.149]
  9    62 ms    64 ms    61 ms  if-11-2.tcore2.NJY-Newark.as6453.net [216.6.87.138]
 10    62 ms    76 ms    62 ms  if-4-4.tcore2.NYY-New-York.as6453.net [66.198.111.18]
 11    59 ms    60 ms    62 ms  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
 12    58 ms    61 ms    60 ms  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
 13    60 ms    69 ms    61 ms  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
 14    59 ms    73 ms    61 ms  66.198.96.50
 15    62 ms    62 ms    64 ms  192.34.76.2
 16    58 ms    59 ms    60 ms  199.91.189.234
 17    62 ms    62 ms    59 ms  199.91.189.30

Trace complete.

Ping statistics for 199.91.189.30:
    Packets: Sent = 30, Received = 30, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 58ms, Maximum = 69ms, Average = 60ms

You've done some exellent work, thank you. As you said, there does not appear to be any lag... tonight at least. Part of what is frustrating is catching this in action. I'm hovering between 50-60 ms tongiht, which is outstanding compared to the normal laggy 150+ms

[kcgreg77]

I'm also on ATT Uverse and seeing sort of the same thing with the Montreal Level3 hop being the bottleneck

C:\Windows\System32>tracert 199.91.189.30

Tracing route to 199.91.189.30 over a maximum of 30 hops

  1     1 ms     1 ms    <1 ms  homeportal [192.168.1.254]
  2    28 ms    27 ms    25 ms  99-51-176-3.lightspeed.kscymo.sbcglobal.net [99.51.176.3]
  3     *        *        *     Request timed out.
  4    25 ms    29 ms    26 ms  12.83.42.157
  5    39 ms    40 ms    39 ms  gar13.cgcil.ip.att.net [12.122.132.121]
  6    39 ms     *       38 ms  192.205.37.150
  7    59 ms    60 ms    59 ms  vl-3506-ve-120.ebr1.Chicago2.Level3.net [4.69.158.134]
  8     *        *        *     Request timed out.
  9   131 ms    59 ms    60 ms  ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
 10    67 ms    66 ms    66 ms  ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
 11     *        *        *     Request timed out.
 12    60 ms    59 ms    59 ms  192.34.76.2
 13    59 ms    58 ms    58 ms  199.91.189.234
 14    56 ms    57 ms    57 ms  199.91.189.30

Trace complete.

The part I'm trying to figure out and I'm currently online with ATT Support, why can't I ping my own Hyperion server from my PC??

Pinging 199.91.189.57 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Tracing route to 199.91.189.57 over a maximum of 30 hops

  1     1 ms     1 ms     1 ms  homeportal [192.168.1.254]
  2    27 ms    25 ms    25 ms  99-51-176-3.lightspeed.kscymo.sbcglobal.net [99.
51.176.3]
  3     *        *        *     Request timed out.
  4    27 ms    28 ms    27 ms  12.83.42.133
  5    41 ms    39 ms    40 ms  gar13.cgcil.ip.att.net [12.122.132.121]
  6    39 ms    38 ms    38 ms  192.205.37.150
  7    61 ms    62 ms    60 ms  vl-3507-ve-121.ebr1.Chicago2.Level3.net [4.69.15
8.138]
  8     *        *        *     Request timed out.
  9   118 ms   227 ms   207 ms  ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]

 10    63 ms    62 ms    60 ms  ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
 11     *        *        *     Request timed out.
 12    61 ms    60 ms    60 ms  192.34.76.2
 13    58 ms    58 ms    58 ms  199.91.189.234
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.
 24     *        *        *     Request timed out.
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.

Trace complete.

Just got off the phone with ATT Uverse support, interesting they are not able to ping the Hyperion server either suggesting that it is blocked. Do I have the correct IP address? Does anyone else get the same results with Hyperion IP 199.91.189.57?

[Raist]

.57 may not be the one you are actually using. SE leases a large chunk of IP's--it's a big subnet for them. While that may be a valid IP that may be getting used for the game, it may not be the one your client connects directly to. You can run netstat while logged into the game to see which ones you actually connect to, or run resmon and expand the network/tcp sections to find it also.