Square, your NA/EU data centre has serious issues...

[blackbluesky]

I am also consistently timing out when my trace hits 38.122.42.34. My connection was fine before Monday's maintenance.

Here is the Whois for 38.122.42.34: http://myip.ms/info/whois/38.122.42.34

Some company called Ormuco owns it. Though there's also mention of cogentco.

It also seems that 38.122.42.34 is the last step before it hits the NA/EU datacenter.

[Raist]

C:\Users\Wyse>ping 184.107.107.176

Pinging 184.107.107.176 with 32 bytes of data:
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51
Reply from 184.107.107.176: bytes=32 time=56ms TTL=51
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51
Reply from 184.107.107.176: bytes=32 time=56ms TTL=51

Ping statistics for 184.107.107.176:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 55ms, Maximum = 56ms, Average = 55ms

C:\Users\Wyse>ping 184.107.107.176

Pinging 184.107.107.176 with 32 bytes of data:
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51

Ping statistics for 184.107.107.176:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 55ms, Maximum = 55ms, Average = 55ms

C:\Users\Wyse>ping 184.107.107.176

Pinging 184.107.107.176 with 32 bytes of data:
Reply from 184.107.107.176: bytes=32 time=54ms TTL=51
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51
Reply from 184.107.107.176: bytes=32 time=54ms TTL=51
Reply from 184.107.107.176: bytes=32 time=55ms TTL=51

Ping statistics for 184.107.107.176:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 54ms, Maximum = 55ms, Average = 54ms


In game: sending 20-100, Receiving 200-5600

Both these tests were done one during 12PM CTR, the 2nd was at 5pm CTR. Please explain to me how this is an ISP problem when according to this test I am connecting to the server flawlessly, but in game I'm getting lag spikes out of the wazoo

PS: It's not just one place either. I've been hearing people complaining about this in game all day long from NYC to LA USA, and im in TX. So whatever the hell the problem is, this needs fixed and SE needs to stop ignoring us.

your hitting the wrong IP... that is a WEBSERVER... old Eidos webserver, with lots of their library aliased to it:
http://myip.ms/browse/sites/1/ipID/1...84.107.107.176

You can go ahead and do a ping or nslookup on some of those names... hitman.com, justcause.com... even the old www.eidos.com points to it.

C:\Windows\System32>nslookup
Default Server: resolver1.opendns.com
Address: 208.67.222.222

> www.hitman.com
Non-authoritative answer:
Name: www.hitman.com
Address: 184.107.107.176

> www.justcause.com
Non-authoritative answer:
Name: www.justcause.com
Address: 184.107.107.176

> www.eidos.com
Non-authoritative answer:
Name: www.eidos.com
Address: 184.107.107.176

> www.deusex.com
Non-authoritative answer:
Name: www.deusex.com
Address: 184.107.107.176

Not only is this IP not in the same subnet, it is also managed by a different company, whose registered address is about 16 blocks away from the one registered to the IP's of the XIV game servers.

IDK who originally posted these IP's... they may have appeared correct at the time, but they are NOT the right ones to use now:

NA - 184.107.107.176
EU - 209.130.141.243
JP - 202.67.53.202

These are hosting websites, NOT XIV... you need to use the IP that your client is actually using while in the game. For the NA regions, these should be starting with 199.

[Twilite]

I am also consistently timing out when my trace hits 38.122.42.34. My connection was fine before Monday's maintenance.

Yay, some one else, besides me has noticed this!

[Raist]

I am also consistently timing out when my trace hits 38.122.42.34. My connection was fine before Monday's maintenance.

Here is the Whois for 38.122.42.34: http://myip.ms/info/whois/38.122.42.34

Some company called Ormuco owns it. Though there's also mention of cogentco.

It also seems that 38.122.42.34 is the last step before it hits the NA/EU datacenter.

Perhaps it is owned/managed by PSINet, which is Cogent Communications--but leased to Ormuco.

According to the ARIN info, it is Cogent Communications for that entire block of IP's:
http://whois.arin.net/rest/net/NET-38-0-0-0-1/pft

That is for 38.0.0.0 through 38.255.255.255--so, it looks a lot like Cogent's hardware/media in play here.

[Jambalaya]

Yay, some one else, besides me has noticed this!

Try tracert -d <ip address of the server>

[Appleh4x]

Perhaps it is owned/managed by PSINet, which is Cogent Communications--but leased to Ormuco.

According to the ARIN info, it is Cogent Communications for that entire block of IP's:
http://whois.arin.net/rest/net/NET-38-0-0-0-1/pft

That is for 38.0.0.0 through 38.255.255.255--so, it looks a lot like Cogent's hardware/media in play here.

Would it surprise you if SE used Cogent to host their servers?

[Noah_]

Perhaps it is owned/managed by PSINet, which is Cogent Communications--but leased to Ormuco.

According to the ARIN info, it is Cogent Communications for that entire block of IP's:
http://whois.arin.net/rest/net/NET-38-0-0-0-1/pft

That is for 38.0.0.0 through 38.255.255.255--so, it looks a lot like Cogent's hardware/media in play here.

Cogent is listed under Data Specialty Providers in Psi Network own website as partners, whatever that means.

[Astralen]

It's not so much the IP address that matters but more what port range you're connecting to and the trace route it takes to get there. If you have the port range and the trace routes, I can look for you and tell you what I think it looks like.

The client will have two open connections. One is to the lobby server while the other is to the host. This is normal.

The issue you describe doesn't sound like high latency, but rather time out requests. When traffic is sent, there is first a handshake with the client and host (basically authenticating and establishing a connection). Once done, the host sends the traffic to the client with an end packet essentially containing a library of what's sent. The host then waits for the client to send it a confirmation of the data sent and that it has all the data where the connection is then terminated. If the flow of data is interrupted in anyway, the host will by default raise a flag and resend the data until it can confirm the data sent. This is essentially a MD5 checksum.

If the MD5 flag is raised, the host and client have to basically rebond with one another and transfer data until the packets are confirmed and complete. Normally, this isn't an issue but with events happening in real time, your client becomes desynchronized with the hosts clock. The rush of data that you see is your client synching back up with the host (host trumps the clients clock and controls the flow).

Hope this helps!

Tracing route to eu.square-enix.com [209.130.141.243] over a maximum of 30 hops:

1 <1 ms 1 ms <1 ms 192.168.1.1
2 1 ms <1 ms <1 ms ua-213-112-240-1.cust.bredbandsbolaget.se [213.112.240.1]
3 9 ms 9 ms 9 ms ti3153d400-xe8-1-1.ti.telenor.net [146.172.85.169]
4 9 ms 9 ms 10 ms ti3153c400-ae3-0.ti.telenor.net [146.172.100.81]
5 9 ms 13 ms 9 ms ti3002c400-ae12-0.ti.telenor.net [146.172.102.86]
6 9 ms 9 ms 9 ms ti3002b400-ae0-0.ti.telenor.net [146.172.105.5]
7 12 ms 12 ms 12 ms globalcrossing-sto-1.ti.telenor.net [148.122.8.230]
8 38 ms 41 ms 37 ms eidos-vpn.gigabitethernet3-39.ar4.lon3.gblx.net [64.213.77.134]
9 * * * Request timed out.
10 39 ms 38 ms 38 ms eu.square-enix.com [209.130.141.243]

Trace complete.

------------------------------------------------------------------------
Tracing route to 199.91.189.28 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.1
2 <1 ms <1 ms <1 ms ua-213-112-240-1.cust.bredbandsbolaget.se [213.112.240.1]
3 9 ms 10 ms 9 ms ti3153d400-xe8-1-1.ti.telenor.net [146.172.85.169]
4 9 ms 16 ms 9 ms ti3153c400-ae3-0.ti.telenor.net [146.172.100.81]
5 9 ms 9 ms 9 ms ti3002c400-ae12-0.ti.telenor.net [146.172.102.86]
6 9 ms 9 ms 9 ms ti3001b400-ae1-0.ti.telenor.net [146.172.105.29]
7 9 ms 9 ms 9 ms be2001.ccr21.sto01.atlas.cogentco.com [130.117.14.97]
8 9 ms 9 ms 9 ms te0-7-0-27.ccr21.sto03.atlas.cogentco.com [154.54.77.130]
9 25 ms 25 ms 25 ms te0-1-0-3.ccr41.ham01.atlas.cogentco.com [130.117.51.54]
10 34 ms 34 ms 34 ms be2188.mpd21.ams03.atlas.cogentco.com [154.54.74.129]
11 41 ms 41 ms 41 ms te0-0-0-0.mpd21.lon13.atlas.cogentco.com [130.117.1.98]
12 51 ms 51 ms 51 ms te0-2-0-0.ccr21.lpl01.atlas.cogentco.com [154.54.60.46]
13 118 ms 118 ms 122 ms te0-1-0-4.ccr21.ymq02.atlas.cogentco.com [154.54.26.141]
14 125 ms 125 ms 125 ms 38.122.42.34
15 * * * Request timed out.
16 113 ms 113 ms 113 ms 192.34.76.2
17 110 ms 110 ms 110 ms 199.91.189.234
18 113 ms 113 ms 110 ms 199.91.189.28

Trace complete.

IDK who originally posted these IP's... they may have appeared correct at the time, but they are NOT the right ones to use now:

NA - 184.107.107.176
EU - 209.130.141.243
JP - 202.67.53.202

These are hosting websites, NOT XIV... you need to use the IP that your client is actually using while in the game. For the NA regions, these should be starting with 199.

Im in EU region and still connect to 199 alas the EU/NA datacenter. And I didn't know about the thing you said that the above IPs are just hosting servers. Makes perfect sense!

[Marishi-Ten]

Marishi is a bit overconfident in his/her own analysis. I have already gone over many of the aspects of why it isn't client side in several different topics. Traceroute is clean, ISP says I'm straight and never throttled, no issues outside of this game etc. While it's true that what I said is a bold claim (about jobless people claiming to be network admins,) it is also true for the most part. People make pretty crazy claims about themselves to bolster their arguments online, job related lies being a very common one. Your own quick defense of using Google as a means to becoming a "network guru" seems to support my claim. Cogent is not the issue; 200ms of lag =/= 5+ seconds and losing several packets =/= a timed out connection.

It's not an issue of overconfidence and I usually put disclaimers in my posts stating my opinion and the data supporting it. They are simply data points. I attempt to be as unbiased as possible to avoid any kind of slanted view.

I have also gone over some data points and I have come up with data that directly conflicts your statement. I've provided my data below. Can you provide your own so we can see if there is a point of common connection?

"Dear Customer,

Regarding your request for account support. Please find your answer below.

If you are receiving an Error 90000 when attempting to access FINAL FANTASY XIV: A Realm Reborn, please try again later. This error simply means that the server is congested and you will need to keep trying until you are able to login. We apologize for any inconvenience this may cause you.

Thank you for contacting the SQUARE ENIX Support Center."

I posted in your thread you made about this email, I'll repost it here as well:

The email you received also looks to be either a canned response or an auto generated reply. Normally, these emails are used as a blanket email by tech firms that are overloaded with support requests. Instead of their agents typing out individual responses, they use a blanket statement to essentially dissuade people from contacting them with egregious requests and/or repeat inquiries from the same login. If you read between the lines, that email actually reads as a "stop contacting us" email.

Accounts don't get stolen by someone's external IP becoming known, stop being alarmist.

Posting traceroutes in no way 'proves' anything if it doesn't show packet loss, chances are your ISP is throttling you and that doesn't show in this kind of network tool.

I play on two servers in Canada, one NA one EU from UK with ZERO noticeable server lag. Last night a couple in my FC said they were lagging, no one else was EVEN IN THE SAME ZONE, so I would assert this proves it's NOT SE's fault!

ISP throttling occurs on early hops where the ISP can control the bandwidth. Latency is also the symptom of throttling. Clients wouldn't be dropping packets all together. The network admins of the hubs should be able to tell if you're being throttled or not. You will more than likely have to file a trouble ticket with the tech rep to have them send to application engineering.

You don't read well do you? When a server is congested it doesn't limit all of the incoming connections, just enough of them to make it run well for the others.

This would be true if latency is seen past the Montreal hops and there wasn’t packet loss. Packet loss isn’t a symptom of throttling. It’s a symptom of over loading. That would also indicate there is an algorithm in place to throttle some users. If so, then thee is a reason why some are being throttled. There is a pattern. Do you happen to have data that supports this?

Except SE explicitly stated that the congestion was on their server. You all have very selective reading habits, don't you?

I assume you’re referring to the email you received from Square that basically told you to stop emailing them right? This is the exact reason why canned responses exist. To weed out egregious requests.

You aren't getting how this works, are you? The type of congestion control used by SE's server is limiting certain ranges of incoming transmissions in order to maintain a higher QOS for the other people connected. It's called congestion control. Do I know their algorithm? No, I didn't set up their server.

I don’t think you understand how hops works. Square cannot control third party nodes. That can limit the amount of traffic onto their rack ports, but this is done via the queue system (hence why you have two active IP’s on one client). There would be no way possible to throttle at any other hop except after Montreal which I have yet to see.

Yes, they limit their connections. They do this via the queue process, not at the server level or a remote hop level. This isn’t what’s creating packet loss. I/O simply doesn’t work like that.

Would it surprise you if SE used Cogent to host their servers?

They don't. Edios has their own racks they are using to host FFXIV. Square is however, using Cogent/Ormuco as the fiber/ISP provider which gives even more credibility to the root issue not being associated with Square but one of the third parties involved.

As promised, here is some of the data I was able to collect that shows evidence that the problem is not with the client network nor with the host network. I've highlighted the problem areas in red for ease of viewing:

4 boid-agw1.inet.qwest.net (184.99.65.65) 17.865 ms 17.774 ms 18.633 ms
5 sea-brdr-02.inet.qwest.net (67.14.41.18) 31.640 ms 31.770 ms 31.015 ms
6 ix-1-0-0-0.tcore1.00s-seattle.as6453.net (64.86.123.77) 36.179 ms 30.107 ms 30.110 ms
7 if-13-0-0-4.core1.00s-seattle.as6453.net (64.86.124.13) 99.434 ms
if-11-0-0-5.core1.00s-seattle.as6453.net (64.86.124.25) 105.043 ms
if-1-0-0-3.core1.00s-seattle.as6453.net (64.86.123.6) 202.544 ms
8 if-9-3-3-0.tcore2.ct8-chicago.as6453.net (64.86.124.34) 95.411 ms 94.850 ms 95.723 ms
9 if-3-2.tcore1.w6c-montreal.as6453.net (66.198.96.45) 110.629 ms 351.581 ms 96.759 ms - NOT SQUARE ENIX
10 66.198.96.50 (66.198.96.50) 164.126 ms 100.402 ms 98.446 ms
11 192.34.76.2 (192.34.76.2) 100.919 ms 100.394 ms 99.517 ms

sudo ping neolobby02.ffxiv.com
PING neolobby02.ffxiv.com (199.91.189.74): 56 data bytes
Request timeout for icmp_seq 0 - HI PACKET LOSS
64 bytes from 199.91.189.74: icmp_seq=1 ttl=54 time=537.147 ms
64 bytes from 199.91.189.74: icmp_seq=2 ttl=54 time=458.225 ms
64 bytes from 199.91.189.74: icmp_seq=3 ttl=54 time=379.185 ms
64 bytes from 199.91.189.74: icmp_seq=4 ttl=54 time=386.048 ms
64 bytes from 199.91.189.74: icmp_seq=5 ttl=54 time=169.980 ms
64 bytes from 199.91.189.74: icmp_seq=6 ttl=54 time=134.348 ms

--- neolobby02.ffxiv.com ping statistics ---
457 packets transmitted, 454 packets received, 0.7% packet loss
round-trip min/avg/max/stddev = 99.605/113.363/664.401 - BAD/51.532 ms

4 boid-agw1.inet.qwest.net (184.99.65.65) 17.779 ms 18.350 ms 17.408 ms
5 sea-brdr-02.inet.qwest.net (67.14.41.18) 30.822 ms 30.159 ms 30.602 ms
6 ix-1-0-0-0.tcore1.00s-seattle.as6453.net (64.86.123.77) 31.615 ms 32.681 ms 30.775 ms
7 if-0-0-0-2.core1.00s-seattle.as6453.net (64.86.123.2) 30.830 ms
if-1-0-0-3.core1.00s-seattle.as6453.net (64.86.123.6) 31.310 ms
if-0-0-0-2.core1.00s-seattle.as6453.net (64.86.123.2) 31.403 ms
8 if-4-1-0-0.tcore2.ct8-chicago.as6453.net (64.86.124.10) 96.160 ms 516.145 ms 699.067 ms - NOT SQUARE ENIX
9 if-3-2.tcore1.w6c-montreal.as6453.net (66.198.96.45) 330.758 ms 222.031 ms 96.910 ms - NOT SQUARE ENIX
10 66.198.96.50 (66.198.96.50) 100.557 ms 101.032 ms 100.031 ms
11 192.34.76.2 (192.34.76.2) 100.524 ms 101.284 ms 99.957 ms

[Marishi-Ten]

Tracing route to eu.square-enix.com [209.130.141.243] over a maximum of 30 hops:

1 <1 ms 1 ms <1 ms 192.168.1.1
2 1 ms <1 ms <1 ms ua-213-112-240-1.cust.bredbandsbolaget.se [213.112.240.1]
3 9 ms 9 ms 9 ms ti3153d400-xe8-1-1.ti.telenor.net [146.172.85.169]
4 9 ms 9 ms 10 ms ti3153c400-ae3-0.ti.telenor.net [146.172.100.81]
5 9 ms 13 ms 9 ms ti3002c400-ae12-0.ti.telenor.net [146.172.102.86]
6 9 ms 9 ms 9 ms ti3002b400-ae0-0.ti.telenor.net [146.172.105.5]
7 12 ms 12 ms 12 ms globalcrossing-sto-1.ti.telenor.net [148.122.8.230]
8 38 ms 41 ms 37 ms eidos-vpn.gigabitethernet3-39.ar4.lon3.gblx.net [64.213.77.134]
9 * * * Request timed out.
10 39 ms 38 ms 38 ms eu.square-enix.com [209.130.141.243]

Trace complete.

------------------------------------------------------------------------
Tracing route to 199.91.189.28 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.1
2 <1 ms <1 ms <1 ms ua-213-112-240-1.cust.bredbandsbolaget.se [213.112.240.1]
3 9 ms 10 ms 9 ms ti3153d400-xe8-1-1.ti.telenor.net [146.172.85.169]
4 9 ms 16 ms 9 ms ti3153c400-ae3-0.ti.telenor.net [146.172.100.81]
5 9 ms 9 ms 9 ms ti3002c400-ae12-0.ti.telenor.net [146.172.102.86]
6 9 ms 9 ms 9 ms ti3001b400-ae1-0.ti.telenor.net [146.172.105.29]
7 9 ms 9 ms 9 ms be2001.ccr21.sto01.atlas.cogentco.com [130.117.14.97]
8 9 ms 9 ms 9 ms te0-7-0-27.ccr21.sto03.atlas.cogentco.com [154.54.77.130]
9 25 ms 25 ms 25 ms te0-1-0-3.ccr41.ham01.atlas.cogentco.com [130.117.51.54]
10 34 ms 34 ms 34 ms be2188.mpd21.ams03.atlas.cogentco.com [154.54.74.129]
11 41 ms 41 ms 41 ms te0-0-0-0.mpd21.lon13.atlas.cogentco.com [130.117.1.98]
12 51 ms 51 ms 51 ms te0-2-0-0.ccr21.lpl01.atlas.cogentco.com [154.54.60.46]
13 118 ms 118 ms 122 ms te0-1-0-4.ccr21.ymq02.atlas.cogentco.com [154.54.26.141]
14 125 ms 125 ms 125 ms 38.122.42.34
15 * * * Request timed out.
16 113 ms 113 ms 113 ms 192.34.76.2
17 110 ms 110 ms 110 ms 199.91.189.234
18 113 ms 113 ms 110 ms 199.91.189.28

Trace complete.

This is odd:

8 38 ms 41 ms 37 ms eidos-vpn.gigabitethernet3-39.ar4.lon3.gblx.net [64.213.77.134] - This is an authentication check hop on the Square side, but why through a VPN?
9 * * * Request timed out. - I wonder if it times out because the check failed or the SSH crashed midway

13 118 ms 118 ms 122 ms te0-1-0-4.ccr21.ymq02.atlas.cogentco.com [154.54.26.141]
14 125 ms 125 ms 125 ms 38.122.42.34
15 * * * Request timed out. - This is on the Square side. Maybe a route table error?
16 113 ms 113 ms 113 ms 192.34.76.2
17 110 ms 110 ms 110 ms 199.91.189.234
18 113 ms 113 ms 110 ms 199.91.189.28

The EU hops are much more direct and the drops there may be influenced by passing through the VPN for authentication. Are you using a VPN? Is there anyway you can get away from that? That's definitely host side. Save that trace and send it to Square with a ticket.

The NA hops you have a ton of. The longer the route, the less stable the connection becomes. Notice that your latency doubles between hop 12 and hop 13 (Same city, just different hub) That's influencing the drop on hop 15. It may not be the catalyst, but it's definitely not helping.