Square, your NA/EU data centre has serious issues...

[Astralen]

This is odd:

8 38 ms 41 ms 37 ms eidos-vpn.gigabitethernet3-39.ar4.lon3.gblx.net [64.213.77.134] - This is an authentication check hop on the Square side, but why through a VPN?
9 * * * Request timed out. - I wonder if it times out because the check failed or the SSH crashed midway

13 118 ms 118 ms 122 ms te0-1-0-4.ccr21.ymq02.atlas.cogentco.com [154.54.26.141]
14 125 ms 125 ms 125 ms 38.122.42.34
15 * * * Request timed out. - This is on the Square side. Maybe a route table error?
16 113 ms 113 ms 113 ms 192.34.76.2
17 110 ms 110 ms 110 ms 199.91.189.234
18 113 ms 113 ms 110 ms 199.91.189.28

The EU hops are much more direct and the drops there may be influenced by passing through the VPN for authentication. Are you using a VPN? Is there anyway you can get away from that? That's definitely host side. Save that trace and send it to Square with a ticket.

The NA hops you have a ton of. The longer the route, the less stable the connection becomes. Notice that your latency doubles between hop 12 and hop 13 (Same city, just different hub) That's influencing the drop on hop 15. It may not be the catalyst, but it's definitely not helping.

Regarding the first trace, indeed it's strange that it goes to a VPN.

Regarding the second one, no I'm not using a VPN, I promise

Ok, I'll make a ticket to SE. Question is if they answer them at all?

[Jougen]

Here's my tracert, 100m\bit wire (NetByNet), Moscow:

1 <1 мс <1 мс <1 мс 192.168.1.1
2 1 ms 1 ms 1 ms msk-b13-s49.ti.ru [212.1.254.115]
3 1 ms 1 ms 1 ms 212.1.252.77
4 9 ms 7 ms 7 ms 78.25.83.161
5 66 ms 66 ms 65 ms 10.222.241.38
6 * 53 ms 53 ms ae52.edge7.Frankfurt1.Level3.net [195.16.162.53]

7 53 ms 53 ms 54 ms ae-3-80.edge4.Frankfurt1.Level3.net [4.69.154.13
6]
8 54 ms 54 ms 53 ms 4.68.111.26
9 157 ms 157 ms 156 ms if-5-2.tcore1.PVU-Paris.as6453.net [80.231.153.1
21]
10 190 ms 170 ms 165 ms if-2-2.tcore1.PYE-Paris.as6453.net [80.231.154.1
8]
11 * 174 ms 175 ms if-5-2.tcore1.L78-London.as6453.net [80.231.130.
1]
12 176 ms 179 ms 173 ms if-2-2.tcore2.L78-London.as6453.net [80.231.131.
1]
13 167 ms 166 ms 171 ms if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.
13]
14 183 ms 180 ms 181 ms if-2-2.tcore2.MTT-Montreal.as6453.net [64.86.226
.13]
15 191 ms 182 ms 185 ms if-0-2.tcore1.MTT-Montreal.as6453.net [216.6.115
.89]
16 181 ms 177 ms 177 ms if-5-2.tcore1.W6C-Montreal.as6453.net [64.86.31.
6]
17 236 ms 241 ms * 66.198.96.50
18 246 ms 251 ms 257 ms 192.34.76.2
19 243 ms 252 ms 246 ms 199.91.189.234
20 278 ms * * 199.91.189.57
21 228 ms 226 ms 220 ms 199.91.189.57

Either they buy a datacenter in eu, or i'm done when sub expires.

[Julusp]

Here's my tracert, 100m\bit wire (NetByNet), Moscow:

1 <1 мс <1 мс <1 мс 192.168.1.1
2 1 ms 1 ms 1 ms msk-b13-s49.ti.ru [212.1.254.115]
3 1 ms 1 ms 1 ms 212.1.252.77
4 9 ms 7 ms 7 ms 78.25.83.161
5 66 ms 66 ms 65 ms 10.222.241.38
6 * 53 ms 53 ms ae52.edge7.Frankfurt1.Level3.net [195.16.162.53]

7 53 ms 53 ms 54 ms ae-3-80.edge4.Frankfurt1.Level3.net [4.69.154.13
6]
8 54 ms 54 ms 53 ms 4.68.111.26
9 157 ms 157 ms 156 ms if-5-2.tcore1.PVU-Paris.as6453.net [80.231.153.1
21]
10 190 ms 170 ms 165 ms if-2-2.tcore1.PYE-Paris.as6453.net [80.231.154.1
8]
11 * 174 ms 175 ms if-5-2.tcore1.L78-London.as6453.net [80.231.130.
1]
12 176 ms 179 ms 173 ms if-2-2.tcore2.L78-London.as6453.net [80.231.131.
1]
13 167 ms 166 ms 171 ms if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.
13]
14 183 ms 180 ms 181 ms if-2-2.tcore2.MTT-Montreal.as6453.net [64.86.226
.13]
15 191 ms 182 ms 185 ms if-0-2.tcore1.MTT-Montreal.as6453.net [216.6.115
.89]
16 181 ms 177 ms 177 ms if-5-2.tcore1.W6C-Montreal.as6453.net [64.86.31.
6]
17 236 ms 241 ms * 66.198.96.50
18 246 ms 251 ms 257 ms 192.34.76.2
19 243 ms 252 ms 246 ms 199.91.189.234
20 278 ms * * 199.91.189.57
21 228 ms 226 ms 220 ms 199.91.189.57

Either they buy a datacenter in eu, or i'm done when sub expires.

Man, Level 3 really sucks - it takes 90 ms from frankfurt to montreal via cogent, 100ms via telia/tinet, and Level 3 needs 100 ms just from Frankfurt to Paris..

[Julusp]

This is odd:

8 38 ms 41 ms 37 ms eidos-vpn.gigabitethernet3-39.ar4.lon3.gblx.net [64.213.77.134] - This is an authentication check hop on the Square side, but why through a VPN?
9 * * * Request timed out. - I wonder if it times out because the check failed or the SSH crashed midway


13 118 ms 118 ms 122 ms te0-1-0-4.ccr21.ymq02.atlas.cogentco.com [154.54.26.141]
14 125 ms 125 ms 125 ms 38.122.42.34
15 * * * Request timed out. - This is on the Square side. Maybe a route table error?
16 113 ms 113 ms 113 ms 192.34.76.2
17 110 ms 110 ms 110 ms 199.91.189.234
18 113 ms 113 ms 110 ms 199.91.189.28

The EU hops are much more direct and the drops there may be influenced by passing through the VPN for authentication. Are you using a VPN? Is there anyway you can get away from that? That's definitely host side. Save that trace and send it to Square with a ticket.

The NA hops you have a ton of. The longer the route, the less stable the connection becomes. Notice that your latency doubles between hop 12 and hop 13 (Same city, just different hub) That's influencing the drop on hop 15. It may not be the catalyst, but it's definitely not helping.

The latency double between hop 12 and 13 is called Atlantic ocean, lpl = liverpool , ymq = montreal (they are codenamed like airports)

[Mang]

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Phillip>tracert 199.91.189.34

Tracing route to 199.91.189.34 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms router.belkin [192.168.2.1]
2 9 ms 10 ms 12 ms 73.88.148.1
3 8 ms 11 ms 8 ms te-0-2-0-5-ur04.eugene.or.bverton.comcast.net [68.85.242.153]
4 20 ms 11 ms 11 ms ae-5-0-ar03.troutdale.or.bverton.comcast.net [68.87.216.121]
5 17 ms 18 ms 19 ms he-2-3-0-0-11-cr01.seattle.wa.ibone.comcast.net [68.86.95.17]
6 34 ms 15 ms 15 ms ix-0-2-2-0.tcore1.00S-Seattle.as6453.net [64.86.123.9]
7 21 ms 20 ms 24 ms if-0-0-0-2.core1.00S-Seattle.as6453.net [64.86.123.2]
8 101 ms 112 ms 117 ms if-9-3-3-0.tcore2.CT8-Chicago.as6453.net [64.86.124.34]
9 91 ms 93 ms 109 ms if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
10 143 ms 153 ms 134 ms 66.198.96.50
11 145 ms 148 ms 144 ms 192.34.76.2
12 132 ms 132 ms 144 ms 199.91.189.234
13 157 ms 143 ms 153 ms 199.91.189.34

Trace complete.

C:\Users\Phillip>tracert 199.91.189.34

Tracing route to 199.91.189.34 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms router.belkin [192.168.2.1]
2 10 ms 10 ms 10 ms 73.88.148.1
3 9 ms 9 ms 12 ms te-0-2-0-5-ur04.eugene.or.bverton.comcast.net [68.85.242.153]
4 18 ms 13 ms 11 ms ae-5-0-ar03.troutdale.or.bverton.comcast.net [68.87.216.121]
5 18 ms 23 ms 16 ms he-2-3-0-0-11-cr01.seattle.wa.ibone.comcast.net [68.86.95.17]
6 16 ms 17 ms 18 ms ix-0-2-2-0.tcore1.00S-Seattle.as6453.net [64.86.123.9]
7 22 ms 22 ms 22 ms if-0-0-0-2.core1.00S-Seattle.as6453.net [64.86.123.2]
8 114 ms 97 ms 98 ms if-9-3-3-0.tcore2.CT8-Chicago.as6453.net [64.86.124.34]
9 122 ms 95 ms 90 ms if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
10 138 ms 135 ms 145 ms 66.198.96.50
11 129 ms 146 ms 144 ms 192.34.76.2
12 126 ms 125 ms 132 ms 199.91.189.234
13 157 ms 146 ms 152 ms 199.91.189.34

Trace complete.

C:\Users\Phillip>tracert 199.91.189.34

Tracing route to 199.91.189.34 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms router.belkin [192.168.2.1]
2 10 ms 10 ms 7 ms 73.88.148.1
3 11 ms 11 ms 8 ms te-0-2-0-5-ur04.eugene.or.bverton.comcast.net [68.85.242.153]
4 12 ms 41 ms 69 ms ae-5-0-ar03.troutdale.or.bverton.comcast.net [68.87.216.121]
5 17 ms 27 ms 19 ms he-2-1-0-0-11-cr01.seattle.wa.ibone.comcast.net [68.86.95.93]
6 16 ms 31 ms 19 ms ix-0-2-2-0.tcore1.00S-Seattle.as6453.net [64.86.123.9]
7 21 ms 29 ms 20 ms if-0-0-0-2.core1.00S-Seattle.as6453.net [64.86.123.2]
8 96 ms 97 ms 95 ms if-9-3-3-0.tcore2.CT8-Chicago.as6453.net [64.86.124.34]
9 89 ms 88 ms 87 ms if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
10 136 ms 136 ms 137 ms 66.198.96.50
11 140 ms 147 ms 131 ms 192.34.76.2
12 137 ms 145 ms 145 ms 199.91.189.234
13 146 ms 146 ms 147 ms 199.91.189.34

Trace complete.

C:\Users\Phillip>

I'm getting the severe lag and occasional 90000 but I'm not seeing any specific timeouts. I'm getting a call back from Comcast in 19-28 minutes, anyone of you networking gurus see anything in particular? Looks like the issue starts on jump 8 to Chicago and snowballs from there.

[Raist]

People have been seeing some weirdness off and on coming across both Washington and Illinois. Many TATA segments (as6453.net) are known for being weird when the web in general gets busy. Ran some tests through a Virginia TATA corridor Friday and it was showing a 24% packet loss at one hop in Ashburne.

Might see something if you run a pathping (might show issues in forwarding between hops that don't show in a tracert). It will take about 7 minutes or so to run---it will look like it stalled, but it hasn't. It will hammer each hop 100 times, and give you some additional info on how the routers are forwarding to each other, not just your ping results to each hop like a tracert. The lines that have the pipe character (|) are the stats for the forwarding between hops. If you see losses on those | lines, there is definitely something going on there--if they are just dropping partially with the regular pings (like a 10% loss or something), they are likely just getting periodically overloaded (too much web traffic) in general. A full on loss (99 or 100%) means it is likely just set to not respond to ICMP requests in the ping--what you are looking for is one that is showing partial responses.

Your 90k error is most likely because their server or other device on SE's end is overloaded and they are clamping down to try to get it under control. It should take some serious packet loss in route to cause a 90K event--more like pulling the plug than a 20% packet loss.

[harleq]

I hate calling people white knights but it does look like the OP has a fair point. From the very beginning he claimed that it was SE's DC issue. He didn't come here seeking for advice. He knew the issue like many of us and he came here looking for support from other people who have come to the same conclusion using the same methods everyone else used.

Sure sometimes traceroute shows a high ping hop near the subject, but those kinds of hops happen for every game on every kind of connectivity. No, the type of latency issue is far greater than the occasional blip that shows up on traceroute. I'm not a profession networking expert but I've built and worked with computers ever since I was 10. I know my way around a computer and a network. I know the diagnostic steps...and if I don't, I know where to find those answers. This issue is on SE's side. You can only ignore the masses for so long. Look at D3...lol

[Julusp]

I hate calling people white knights but it does look like the OP has a fair point. From the very beginning he claimed that it was SE's DC issue. He didn't come here seeking for advice. He knew the issue like many of us and he came here looking for support from other people who have come to the same conclusion using the same methods everyone else used.

Sure sometimes traceroute shows a high ping hop near the subject, but those kinds of hops happen for every game on every kind of connectivity. No, the type of latency issue is far greater than the occasional blip that shows up on traceroute. I'm not a profession networking expert but I've built and worked with computers ever since I was 10. I know my way around a computer and a network. I know the diagnostic steps...and if I don't, I know where to find those answers. This issue is on SE's side. You can only ignore the masses for so long. Look at D3...lol

If the issue was on SE side, everyone would be suffering from the lags/disconnects and so. But I am not. In fact, many people are not.

If you want to complain for network, this is IP of datacenter 192.34.76.2. This IP is common for all people who want reach the game, everything before it just a path to it..

So lets say you are going to New York, but road you are on in Pennsylvania is bad. Are you seriously going to complain New York that your road is bad?

[Raist]

I hate calling people white knights but it does look like the OP has a fair point. From the very beginning he claimed that it was SE's DC issue. He didn't come here seeking for advice. He knew the issue like many of us and he came here looking for support from other people who have come to the same conclusion using the same methods everyone else used.

Sure sometimes traceroute shows a high ping hop near the subject, but those kinds of hops happen for every game on every kind of connectivity. No, the type of latency issue is far greater than the occasional blip that shows up on traceroute. I'm not a profession networking expert but I've built and worked with computers ever since I was 10. I know my way around a computer and a network. I know the diagnostic steps...and if I don't, I know where to find those answers. This issue is on SE's side. You can only ignore the masses for so long. Look at D3...lol

...because I can play fine on JP server.

So yeah must be my ISP or equipment or or or or

LOL-worthy

So um please advise?

Actually, the OP did ask for advice in the matter... Just didn't like the advice being given I guess.


Just because someone has a theory that runs contrary to someone's belief that it is all SE's fault, does not make them a White Knight for SE. Ever consider the facts that support the theories in question?

Network congestion is network congestion, regardless of where it resides---at the endpoint server, or a router in between. A lot of those routers have been documented to break the 20% packet loss boundary---some have even breached 50% at times under high load. That means they are loosing packets that have to be retransmitted--or otherwise get discarded (creating a different kind of problem). This can potentially have a profound impact on your latency because of how it can dramatically compound other issues. If you are already past 300ms latency (which is almost 1/3 second delay in receiving broadcasted data from the server), and that is further delayed in route by an additional 20-50% because of issues in route that delay the delivery further, you really feel it. Stack that on top of server delays due to high congestion, and you can easily hit delays reaching into seconds and not milliseconds--especially when you are talking round trips. Remember, you have to get your data to the server, have it processed, and then receive that response from the server. Delays going both ways can really wreak havoc on game play.

Some of these problems are out of SE's hands. Some are even out of your ISP's hands to some extent (short of them choosing to route you to someone else's routers, trying to force a third party's hand to resolve issues on their hardware/media). Even if SE finally resolves their load balancing issues on their server and their networks, these problems may still exist in areas over which they have no influence/control. The point is, if one can further isolate where the problem lies they can get that information to more appopriate people to try to affect change. Why not try to work towards that resolution while we wait for SE to come up with theirs?

Also, as already stated, some simply aren't experiencing ANY issues... until there are congestion issues in route. For those players, it has nothing to do with SE--it is an ISP handling segments in between the player and the server. May not be their ISP, but it IS an Internet Service Provider that is at fault.

[Pangtao]

...because I can play fine on JP server.

So yeah must be my ISP or equipment or or or or

LOL-worthy

So um please advise?

everything seems fine for me, hows the JP servers?