So, after doing a few investigation threads on the issue, one IP address in particular keeps coming up in many people's trace routes to their servers :
192.34.76.2
Information on this IP suggests it is in Monteral, Canada.
Here is a sample of my traceroute. You can see as soon as it hits that IP, the latecy rises significantly.
Others' traceroute results show the same and can be found from the VErizon FiOS link below as well as this Reddit post:Code:1: 192.168.1.103 0.115ms pmtu 1500
1: 192.168.1.254 0.886ms
1: 192.168.1.254 0.676ms
2: 172.6.220.2 25.639ms
3: 76.201.208.74 24.954ms
4: 76.201.208.125 30.000ms
5: 12.83.82.145 25.087ms
6: 12.122.141.233 28.614ms asymm 7
7: no reply
8: no reply
9: no reply
10: no reply
11: no reply
12: no reply
13: no reply
14: 4.69.141.5 79.500ms asymm 13
15: 4.69.141.1 82.823ms asymm 14
16: 4.59.178.74 79.945ms asymm 15
17: no reply
18: 192.34.76.2 105.938ms asymm 14
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
31: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
http://forums.verizon.com/t5/FiOS-In.../622913/page/2
http://www.reddit.com/r/ffxiv/commen...rse_featuring/
-
10-14-2014, 11:35 AMDocfiord_FowlingIssue of all AT&T Uverse lag possibly identified - SE Confirm
-
10-14-2014, 12:13 PMHidingYoshis
I can confirm on my end that this happens:
3 21 ms 24 ms 21 ms 75.26.64.42
4 21 ms 21 ms 20 ms 75.26.64.201
5 22 ms 23 ms 23 ms 12.83.32.133
6 44 ms 30 ms 31 ms ggr4.cgcil.ip.att.net [12.122.133.33]
7 29 ms 30 ms 29 ms 192.205.37.150
8 84 ms 83 ms 84 ms vl-3508-ve-122.ebr1.Chicago2.Level3.net [4.69.15
8.142]
9 * * * Request timed out.
10 116 ms 93 ms 70 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
11 64 ms 65 ms 66 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
12 * * * Request timed out.
13 153 ms 153 ms 149 ms 192.34.76.2
14 146 ms 137 ms 141 ms 199.91.189.234
15 154 ms 153 ms 152 ms 199.91.189.40
As you said, 192.34.76.2 appears to be the cause of my problems.
I hope Square Enix actually sees AND does something about this. -
10-14-2014, 12:28 PMDocfiord_Fowling
So far it seems AT&T, Verizon FiOS, and Canada Bell are all having this issue and they all run through that hop. SE, please look into this if you have any ontrol over that address space.
-
10-14-2014, 10:14 PMSqua11_Leonhartthey don't - its a third party peering networkQuote:
Originally Posted by Docfiord_Fowling
-
10-14-2014, 10:49 PMDocfiord_FowlingInteresting. Mind sharing how you came about this information?Quote:
Originally Posted by Squa11_Leonhart
-
10-15-2014, 01:15 AMRaistQuote:
Originally Posted by Docfiord_Fowling
Quote:
Originally Posted by HidingYoshis
Actually, if you look more closely, you will find the typical signs of congestion issues occurring earlier in the routes that may be at fault that need to be looked into first. For instance, look at the wide variance at Level3's hop shortly after coming around Chicago:Quote:
Originally Posted by Docfiord_Fowling
10 116 ms 93 ms 70 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
That's a swing of over 65% variance in a short time frame. If this is occurring regularly, it is a sign of pending (or even actual) overload either at the IPX or coming through Level3's segments. Such an issue compounds itself the further you go down the route. The fact that that hop further down is having a much more consistent response time is actually encouraging--it means it is doing a better job of maintaining traffic flow. The fact the response times are markedly high could very well just be because the shaping policies in effect have put the ICMP ECHO request on a lower priority, which means it is working as intended--slowing down less important traffic to reserve capacity for higher priority traffic. This is done to try to maintain more consistent flow--which clearly is not happening at the other hop.
Some other examples from one of the linked threads:
3 25 ms 8 ms 11 ms G0-6-4-6.WASHDC-LCR-22.verizon-gni.net [130.81.212.144]
4 33 ms 11 ms 14 ms xe-1-1-1-0.PHIL-BB-RTR2.verizon-gni.net [130.81.209.18]
so-9-0-2-0.RES-BB-RTR1.verizon-gni.net 0 150 150 7 18 100 9
(best time of 7, average of 18, worst of 100)
And this one is a REAL doozy:
In that sample, it's ALL on Verizon and their Alter.net subsidiary before it even get to TI-Net, which is another offender at times as well.Code:Host % Sent Recv Best Avrg Wrst Last
Wireless_Broadband_Router.home 0 500 500 0 0 1 0
L100.BLTMMD-VFTTP-52.verizon-gni.net 0 500 500 5 7 114 7
G5-0-352.BLTMMD-LCR-03.verizon-gni.net 0 500 500 5 7 118 7
so-9-0-2-0.RES-BB-RTR1.verizon-gni.net 0 500 500 7 18 176 99
0.ae5.XL1.IAD8.ALTER.NET 3 457 446 7 10 48 42
0.xe-10-1-1.GW9.IAD8.ALTER.NET 0 500 500 7 11 123 12
tinet-gw.customer.alter.net 5 421 401 15 19 97 17
xe-1-0-0.mtl10.ip4.tinet.net 0 500 500 25 37 184 37
ormuco-gw.ip4.tinet.net 0 500 500 42 45 148 43
192.34.76.2 0 500 500 42 46 147 45
199.91.189.234 1 496 495 40 45 170 41
199.91.189.30 0 500 500 40 43 147 41
The root of this is indeed (in the majority of cases) an issue of flow control, compounded by shoddy/outdated peering agreements between our last mile ISP's and their routing partners. It has been discussed over and over on various forums for years (not just XIV... I remember it going back to the early days of WoW, Diablo, and even Freespace: Descent). Articles have been published and briefs/complaints filed with the regulatory commissions trying to get the issue addressed. One such blog post was put up by Level 3 this past year (you can find more on the topic by simply googling "level3.net chicken", or follow the links that crop up with that blog post.) TATA also filed briefs on similar observations about 6 or 7 years ago. There was a big kerfuffle over Verizon, Comcast, and Netflix in more recent history. It's all bound to an issue with the ISP's cutting corners and trying to make more money. All the while fleecing us for all we are worth while providing us with sub-par service.
Need to hold your ISP's feet to the fire to address this. They have the resources to address the issue--need to get their Tier3 people involved so they can properly investigate the issues and push for the escalation to get them addressed.
It should be noted that a few months back, while working with my ISP's T3 support, we spotted an issue on the other side of Cogent's transit in a VLAN managed by Ormuco. It was routinely spiking upwards of 280ms from it's norm of around 100ms. We went straight to them and got it addressed without getting SE involved. This CAN and SHOULD be handled through your ISP, since you are paying them for the service of routing you across the internet, and THEIR routing policies are what is keeping you on these troublesome routes. If they cannot get the responsible party to address the problem directly, they have the option of changing you to alternate routing partners to try to find a cleaner route. To that end, TWC changes my route every few weeks trying to stay ahead of the congested hops. This is done entirely through my ISP's T3 support... no one else--I just email them with my reports, and they take care of it. Sometimes they catch before I do... kind of annoying when my modem bounces to re-align my channels when I am in game, but at least I am able to log right back in a few minutes later and continue playing with minimal issues.
Edit:
After eating dinner, I ran a trace from the back of the house (from laptop, on Wifi from about 30 feet away), and from here in South Carolina, at 7:30PM... trace is clean as it usually is. I don't experience this horrible lag you speak of, and I am going through that very same hop---but it is giving me mostly right around 60ms response times (rarely breaks 80 even under the busiest of times unless something is up along my route, which my ISP usually addresses pretty quickly now that I have gotten them to take action).
Code:C:\Windows\System32>tracert 199.91.189.30
Tracing route to 199.91.189.30 over a maximum of 30 hops
1 2 ms 1 ms <1 ms LPTSRV [10.10.100.1]
2 37 ms 26 ms 28 ms cpe-066-026-112-001.sc.res.rr.com [66.26.112.1]
3 17 ms 24 ms 29 ms cpe-024-031-198-005.sc.res.rr.com [24.31.198.5]
4 13 ms 16 ms 14 ms clmasoutheastmyr-rtr2.sc.rr.com [24.31.196.210]
5 28 ms 25 ms 27 ms be33.drhmncev01r.southeast.rr.com [24.93.64.180]
6 32 ms 34 ms 35 ms 107.14.19.20
7 33 ms 43 ms 33 ms so-1-1-1.c1.buf00.tbone.rr.com [66.109.1.113]
8 32 ms 31 ms 32 ms ix-17-0.tcore2.AEQ-Ashburn.as6453.net [216.6.87.149]
9 62 ms 64 ms 61 ms if-11-2.tcore2.NJY-Newark.as6453.net [216.6.87.138]
10 62 ms 76 ms 62 ms if-4-4.tcore2.NYY-New-York.as6453.net [66.198.111.18]
11 59 ms 60 ms 62 ms if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
12 58 ms 61 ms 60 ms if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
13 60 ms 69 ms 61 ms if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
14 59 ms 73 ms 61 ms 66.198.96.50
15 62 ms 62 ms 64 ms 192.34.76.2
16 58 ms 59 ms 60 ms 199.91.189.234
17 62 ms 62 ms 59 ms 199.91.189.30
Trace complete.
Ping statistics for 199.91.189.30:
Packets: Sent = 30, Received = 30, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 58ms, Maximum = 69ms, Average = 60ms
-
10-15-2014, 09:52 AMkcgreg77
I'm also on ATT Uverse and seeing sort of the same thing with the Montreal Level3 hop being the bottleneck
The part I'm trying to figure out and I'm currently online with ATT Support, why can't I ping my own Hyperion server from my PC??Code:C:\Windows\System32>tracert 199.91.189.30
Tracing route to 199.91.189.30 over a maximum of 30 hops
1 1 ms 1 ms <1 ms homeportal [192.168.1.254]
2 28 ms 27 ms 25 ms 99-51-176-3.lightspeed.kscymo.sbcglobal.net [99.51.176.3]
3 * * * Request timed out.
4 25 ms 29 ms 26 ms 12.83.42.157
5 39 ms 40 ms 39 ms gar13.cgcil.ip.att.net [12.122.132.121]
6 39 ms * 38 ms 192.205.37.150
7 59 ms 60 ms 59 ms vl-3506-ve-120.ebr1.Chicago2.Level3.net [4.69.158.134]
8 * * * Request timed out.
9 131 ms 59 ms 60 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
10 67 ms 66 ms 66 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.74]
11 * * * Request timed out.
12 60 ms 59 ms 59 ms 192.34.76.2
13 59 ms 58 ms 58 ms 199.91.189.234
14 56 ms 57 ms 57 ms 199.91.189.30
Trace complete.
Just got off the phone with ATT Uverse support, interesting they are not able to ping the Hyperion server either suggesting that it is blocked. Do I have the correct IP address? Does anyone else get the same results with Hyperion IP 199.91.189.57?Code:Pinging 199.91.189.57 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Tracing route to 199.91.189.57 over a maximum of 30 hops
1 1 ms 1 ms 1 ms homeportal [192.168.1.254]
2 27 ms 25 ms 25 ms 99-51-176-3.lightspeed.kscymo.sbcglobal.net [99.
51.176.3]
3 * * * Request timed out.
4 27 ms 28 ms 27 ms 12.83.42.133
5 41 ms 39 ms 40 ms gar13.cgcil.ip.att.net [12.122.132.121]
6 39 ms 38 ms 38 ms 192.205.37.150
7 61 ms 62 ms 60 ms vl-3507-ve-121.ebr1.Chicago2.Level3.net [4.69.15
8.138]
8 * * * Request timed out.
9 118 ms 227 ms 207 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
10 63 ms 62 ms 60 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
11 * * * Request timed out.
12 61 ms 60 ms 60 ms 192.34.76.2
13 58 ms 58 ms 58 ms 199.91.189.234
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
-
10-15-2014, 10:36 AMRaist
.57 may not be the one you are actually using. SE leases a large chunk of IP's--it's a big subnet for them. While that may be a valid IP that may be getting used for the game, it may not be the one your client connects directly to. You can run netstat while logged into the game to see which ones you actually connect to, or run resmon and expand the network/tcp sections to find it also.
-
10-15-2014, 12:20 PMDocfiord_FowlingYou've done some exellent work, thank you. As you said, there does not appear to be any lag... tonight at least. Part of what is frustrating is catching this in action. I'm hovering between 50-60 ms tongiht, which is outstanding compared to the normal laggy 150+msQuote:
Originally Posted by Raist
-
10-15-2014, 01:34 PMRaistActually, that is a pretty typical trace for me now when they first put me on TATA...usually holds for 3-5 weeks. Sometimes get a bit better with Cogent via Atlanta, but that starts to go south after about 3 weeks. Level3 via Charlotte/Raleigh is getting worse each time they shoot me through that corridor with them. Had to move away from them after just 1 week last time...even the roku was freaking out.Quote:
Originally Posted by Docfiord_Fowling
I routinely run a quick test prior to logging in, sometimes run another before planning anything important. As I stated, even in the busiest times it stays low...as in around the 80's. Even if it were to break 100ms response time that is still quite manageable, considering it would still be under 1/8 a second. Every time I have felt lag issues creeping up though it is either at an early TWC gateway, or out around the peering handoffs. Both of which are resolved through my ISP. -
10-17-2014, 10:16 AMghostx1984
It has been lagging every night at 8pm cst without fail. Any other time of day it is completely perfect but as soon as 8:00 rolls around the fun begins. And by fun i mean horribly depressing stop motion lag. It usually takes around at least half an hour to an hour to go away. I personally can raid 24.7 but the people i raid with only have a few hours and this falls within that timeframe.
My internet shows a 30mpbs download with 4-5mpbs upload during this time as if there was nothing wrong. What is going on here?
I called ATAT a few nights ago and they said they would check it out and monitor it. Is this a problem on the service providers end or SE?
Tracing route to 199.91.189.27 over a maximum of 30 hops
1 1 ms 1 ms <1 ms 192.168.1.254
2 24 ms 19 ms 20 ms 107-219-200-2.lightspeed.cicril.sbcglobal.net [1
07.219.200.2]
3 21 ms 21 ms 20 ms 71.145.65.170
4 23 ms 23 ms 23 ms 12.83.43.29
5 23 ms 23 ms 24 ms ggr4.cgcil.ip.att.net [12.122.133.33]
6 * 20 ms 21 ms 192.205.37.150
7 125 ms 123 ms 124 ms vl-3505-ve-119.ebr1.Chicago2.Level3.net [4.69.15
8.130]
8 * * * Request timed out.
9 * 124 ms 119 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
10 122 ms 121 ms 124 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
11 * * * Request timed out.
12 266 ms 268 ms 271 ms 192.34.76.2
13 270 ms * 273 ms 199.91.189.234
14 268 ms 272 ms 272 ms 199.91.189.27
Trace complete.
Tracing route to 199.91.189.27 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.1.254
2 39 ms 26 ms 21 ms 107-219-200-2.lightspeed.cicril.sbcglobal.net [1
07.219.200.2]
3 21 ms 20 ms 28 ms 71.145.65.170
4 22 ms 24 ms 24 ms 12.83.43.29
5 22 ms 23 ms 24 ms ggr4.cgcil.ip.att.net [12.122.133.33]
6 22 ms * * 192.205.37.150
7 121 ms 122 ms 125 ms vl-3505-ve-119.ebr1.Chicago2.Level3.net [4.69.15
8.130]
8 * * * Request timed out.
9 200 ms 214 ms 216 ms ae-10-10.car2.Montreal2.Level3.net [4.69.153.86]
10 129 ms 129 ms 130 ms ORMUCO-COMM.car2.Montreal2.Level3.net [4.59.178.
74]
11 * * * Request timed out.
12 195 ms 223 ms 195 ms 192.34.76.2
13 196 ms 196 ms 193 ms 199.91.189.234
14 196 ms 195 ms 197 ms 199.91.189.27
Trace complete.
Ping statistics for 199.91.189.27:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 205ms, Maximum = 208ms, Average = 206ms -
10-17-2014, 11:15 AMSqua11_LeonhartRead Raists posts, and look at your trace.Quote:
Originally Posted by ghostx1984
-
10-20-2014, 08:55 AMDocfiord_Fowling
So, after idetifying extreme latency between two of the same hops (one owned by AT&T my ISP, the other usually Level 3) I tried contacting my ISP and asking if there was nything we can do. Perhaps an alternate route? Nope. they are not budging and insist this is beyond their control. This... is very annoying.
-
10-20-2014, 09:31 AMSqua11_Leonhart
will continue to happen until either level 3 crumbles and pays them, or the big isps in america get a massive wake up call and the internet becomes at utility service.