Square, your NA/EU data centre has serious issues...

[Myschaella]

I have been suffering with bad lag during high load times ie I can play all day no lag but then in the evening as more people start logging on my lag begins and gradually gets worse. This lag only started for me after last weeks maintenance, before that I had pretty much zero lag problems at all times of the day and night. Now after yesterdays maintenance I tried an AK run, thank god I was with a tolerant party because my lag was 10x worse than ever and I even got error 9k 2x during the run (I have had error 9k only 1 time since the error 9k/ 3102 debarcle during beta).
All my other MMOs (Tera, WOW, DCUO, TSW) I have zero lag problems at all times of the day and as I said until last week I had zero lag problems in FFXIV.
I am not an IT expert or a tech Wiz however I am an MMO vet of almost 10yrs having played pretty much all the AAA titles and I can safely say in all my experience whenever there has been a lag issue similar to this it has always been addressed by the game company, despite all the sudden tech wizards that appear stating it is the player or their isp fault. I agree for some there may be a problem with their isp but this has happened to too many people at the same time after last weeks maintenance (in both NA and EU, I am in EU), for this to be primarily the fault of the players isp.
I sent in a tech report but as my sub is up in next few days and I was considering 3 or 6 month renewal (which I am now probably not going to do), I just hope something is sorted out for the benefit of all us players who want to keep playing a fun game.

[Appleh4x]

Network admins coming out in doves defending SE. SE doesn't intend to do any investigation on their side.

I'm sick and tired of disagreeing with people. Forget that there are people who have traced and found no issues (timeout/packet loss/high ping), and yet still experience strange lag. Or that there are people who have no problems with any other games during prime time. Time to be negative, take the safe bet, and sound smart (hey 3 birds, 1 stone).

"Learn to call your ISP."

There: problem solved for thousands of people. I'm done keeping up with the lag issue. It doesn't even affect me. I can dodge just fine looking at circles.

Complainers are always wrong. The big company that doesn't even bother to give a response to a 72 page thread is doing it right.

Lol who cares about responsiveness anyways.

I think you mean jobless people claiming to be network admins because they can use google.

[Marishi-Ten]

Raist, you're caught in a devil's proof. You can't establish that this issue is not SE's fault.

1. You have to prove that the reason this issue is occurring is because of infrastructure and not SE server configuration.
2. The only way to prove that is to get a massive amount of data/logs from each user who is having this problem (as 1 ping/trace does nothing)
3. You have no means of obtaining this massive amount of information. The reports you're doing here is like a grain of sand in the desert.

Thus you cannot prove that this is not SE's fault.

Likewise,we are caught in Hempel's Raven

1. If lag affects our game it's SE's fault
2. If we don't have lag, it's not SE's fault.
3. To qualify 1, in the past , when there was a lag, SE has been able to resolve it.
4. We have no lag SE is doing a good job.

By the same reasoning, the statement would be: If we have Lag, SE is not doing a good job. This is paradoxical - I can think of 100s of occasions where I can have lag and it's not SE's fault.

So what is the solution? The problem is that WE'RE LACKING IN INFORMATION. If SE would be so kind to provide us with more INFORMATION we can start finding a solution. It could very well be our fault. However, we don't know without INFORMATION. And there's literally nothing we can do.

1.) Square is not affiliated with Cogent, i-web, as6453, Level3, and the countless other hubs where the bottleneck occurs. I would say it's a fair assumption that the issue is not on the end point.

2.) I have logged some data showing the packet loss and icmp time outs on between client and host. I feel confident that I have enough aggregate data to, at the very least, show a pattern of behavior.

3.) Perhaps, but it shows a pattern. Patterns can lead to educated guesses that are naturally near the mark. Users don't time out at the end IP, they time out at a remote hub. Generally where there's smoke, there is almost always fire.

4.) Lag can occur from the local router all the way to the end point. Latency on a network means nothing until a pattern emerges to narrow down the root cause.

5.) Unless the bottle neck occurred in the Montreal datacenter, it wouldn't be Squares responsibility to fix and they wouldn't be able to even if they wanted.

6.) No, they have been able to mask it. Dumping the DNS cache and refreshing the ports can help alleviate some congestion but the issue will return until the hubs/fiber are physically upgraded. Treating a symptom isn't the same thing as curing the disease.

7.) The hubs and fiber are running efficiently. Routing traffic at full capacity while also maintaining low packet loss is the goal. If this is accomplished, all parties involved are performing satisfactory.

8.) I believe Square can't provide us with said information for a couple reasons. One, to have access to that data they would need to give us root access into their networks which no sane person would allow as this would also give the user access to their ruby/command console as well. Two, Square may not be able to provide it as even they don't have access to it as most hubs and infrastructures are locked down even to clients using the pipeline.

9.) We can educate and inform. We can answer questions and assist to the best of our abilities. The more people know, the more people can field tickets to the right places. Knowledge is power.

Network admins coming out in doves defending SE. SE doesn't intend to do any investigation on their side.

I'm sick and tired of disagreeing with people. Forget that there are people who have traced and found no issues (timeout/packet loss/high ping), and yet still experience strange lag. Or that there are people who have no problems with any other games during prime time. Time to be negative, take the safe bet, and sound smart (hey 3 birds, 1 stone).

"Learn to call your ISP."

There: problem solved for thousands of people. I'm done keeping up with the lag issue. It doesn't even affect me. I can dodge just fine looking at circles.

Complainers are always wrong. The big company that doesn't even bother to give a response to a 72 page thread is doing it right.

Lol who cares about responsiveness anyways.

It's not defense when It's been shown the issue isn't on the end ports. It's point A to point C where data is lost on point B. Is it point C's problem that point B can't send it information? Rhetorical question.

Debate is healthy for issues that are difficult to resolve. I prefer that others disagree as it makes me look over the data points again and make the case for what I think is the cause. I have some data that I was able to store that shows a pattern though you are right, to diagnose a network problem, you have to continuously run network monitoring scripts and store the data. Luckily, Terminal does this for me and runs icmp dumps with custom scripting.

Contacting your ISP and asking for a re route if possible may clear the issue, though I don't think this is what you meant. ISP's have control of their hubs and networks. Century Link can't fix Level3's problem. Why would an individual call a company with no affiliation to the problem end and expect them to fix it? The best thing we can do is request a route change. If this isn't possible, contact the owner of the hub you are seeing the most problems with.

I also don't have any issues with latency save for the mild hiccup every now and then. I post here to help and educate others. Just because a user doesn't have a problem isn't a reason to be apathetic to others that may be experiencing issues.

What would you like Square to say? They are doing the best they can spending millions of dollar upgrading racks for more re routes to help with congestion? We know they are. Would you like them to say that their fiber provider doesn't have enough bandwidth and possibly violating their service agreement costing them money that would better go to making FFXIV better? Or would you like them to say that the issue is on another company without solid proof (let's face it, they don't have root access into their hubs either) possibly causing a law suit that could end up closing FFXIV? Would that really make anyone here feel better? This is the opposite of what anyone wants. Square is a business and by holding back any information they may have is either due to contractual obligations or legal implications. It's easier to be transparent about an issue instead of covering it up. Why go to all this trouble if there isn't a valid reason behind it?

Transparency is a good thing. But people have to understand that just because they aren't saying anything doesn't mean anything is happening.

This isn't a jab at you personally and I'm not saying you're wrong in your thinking. It's more of a general outline of what I think may be happening from my own experiences and technical backround. As always, take it with a grain of salt.

I think you mean jobless people claiming to be network admins because they can use google.

That's an extremely blanked statement. Though Google can answer most queries, we all come from different walks of life and may actually have experience dealing with large scale networks and server clusters under load that no search engine can replicate.

Also, as a rule of thumb, I try not to stereotype others as it makes me sound ignorant of the topic at hand and only serves to agitate and intrench the other party into their line of thinking which does no one any good.

Do you happen to have any insight you care to share? Are you having an issue with latency? If so, can you describe the issue? Users on this forum may be able to shed some light and offer some steps to perform to alleviate it.

[Gonzothegreat198]

He had a 90k issue and was give the generic 90k automated response for the 90k issues. Now because that response came from SE he is under the impression that it is now confirmed that ALL the connection issue are hereby related to server issues. See: http://forum.square-enix.com/ffxiv/t...side-CONFIRMED

[Astralen]

I'm very interested in understanding how the different datacenters IP addresses work.

We know it looks like this:
NA - 184.107.107.176
EU - 209.130.141.243
JP - 202.67.53.202

If I ping the EU datacenter I get 40ms. But when FFXIV is launched and I check the Resource monitor in Win8 then I see that the game is connected to a different IP address (I'm at work and forgot to bring this info with me) which points to Montreal (whois). Resource monitor tells me that those two connections (yes two, why?) give me ~170 and ~200 ping which would be considered normal for me living in Europe (I could compare this by checking the ping of a Battlefield 3 server in the US which range from 150-220ms).

Now, 170ms is much but not that much and it's definitely not the 1000-2500ms I presume I had the day before yesterday (casting a 2sec spell then immediately jumping and seeing it get interrupted when the meter says 0.5sec or not getting interrupted at all). Wouldn't this be another proof that the datacenters are causing the lag and not our ISP or the connection between EU and Canada?

Sorry for being Cpt. Obvious here but I'm just trying to understand (and maybe helping others understand too).

Good news is that there was less lag yesterday evening Quite playable but the rubberbanding was present from time to time.

[Malitias]

Starting to have the same problems as OP... Can log-in on JP servers but not EU/NA servers.
No idea why. Had no problems for 2 weeks.

[Marishi-Ten]

I'm very interested in understanding how the different datacenters IP addresses work.

We know it looks like this:
NA - 184.107.107.176
EU - 209.130.141.243
JP - 202.67.53.202

If I ping the EU datacenter I get 40ms. But when FFXIV is launched and I check the Resource monitor in Win8 then I see that the game is connected to a different IP address (I'm at work and forgot to bring this info with me) which points to Montreal (whois). Resource monitor tells me that those two connections (yes two, why?) give me ~170 and ~200 ping which would be considered normal for me living in Europe (I could compare this by checking the ping of a Battlefield 3 server in the US which range from 150-220ms).

Now, 170ms is much but not that much and it's definitely not the 1000-2500ms I presume I had the day before yesterday (casting a 2sec spell then immediately jumping and seeing it get interrupted when the meter says 0.5sec or not getting interrupted at all). Wouldn't this be another proof that the datacenters are causing the lag and not our ISP or the connection between EU and Canada?

Sorry for being Cpt. Obvious here but I'm just trying to understand (and maybe helping others understand too).

Good news is that there was less lag yesterday evening Quite playable but the rubberbanding was present from time to time.

It's not so much the IP address that matters but more what port range you're connecting to and the trace route it takes to get there. If you have the port range and the trace routes, I can look for you and tell you what I think it looks like.

The client will have two open connections. One is to the lobby server while the other is to the host. This is normal.

The issue you describe doesn't sound like high latency, but rather time out requests. When traffic is sent, there is first a handshake with the client and host (basically authenticating and establishing a connection). Once done, the host sends the traffic to the client with an end packet essentially containing a library of what's sent. The host then waits for the client to send it a confirmation of the data sent and that it has all the data where the connection is then terminated. If the flow of data is interrupted in anyway, the host will by default raise a flag and resend the data until it can confirm the data sent. This is essentially a MD5 checksum.

If the MD5 flag is raised, the host and client have to basically rebond with one another and transfer data until the packets are confirmed and complete. Normally, this isn't an issue but with events happening in real time, your client becomes desynchronized with the hosts clock. The rush of data that you see is your client synching back up with the host (host trumps the clients clock and controls the flow).

Hope this helps!

[Astralen]

It's not so much the IP address that matters but more what port range you're connecting to and the trace route it takes to get there. If you have the port range and the trace routes, I can look for you and tell you what I think it looks like.

The client will have two open connections. One is to the lobby server while the other is to the host. This is normal.

The issue you describe doesn't sound like high latency, but rather time out requests. When traffic is sent, there is first a handshake with the client and host (basically authenticating and establishing a connection). Once done, the host sends the traffic to the client with an end packet essentially containing a library of what's sent. The host then waits for the client to send it a confirmation of the data sent and that it has all the data where the connection is then terminated. If the flow of data is interrupted in anyway, the host will by default raise a flag and resend the data until it can confirm the data sent. This is essentially a MD5 checksum.

If the MD5 flag is raised, the host and client have to basically rebond with one another and transfer data until the packets are confirmed and complete. Normally, this isn't an issue but with events happening in real time, your client becomes desynchronized with the hosts clock. The rush of data that you see is your client synching back up with the host (host trumps the clients clock and controls the flow).

Hope this helps!

Thanks for the great explanation!

Regarding ports they are:

UDP
55296-55551

TCP
80, 8080, 443, 55296-55551

I'll give you a tracert of the EU Datacenter and the Canadian IP when I come home from work.

[Appleh4x]

Marishi is a bit overconfident in his/her own analysis. I have already gone over many of the aspects of why it isn't client side in several different topics. Traceroute is clean, ISP says I'm straight and never throttled, no issues outside of this game etc. While it's true that what I said is a bold claim (about jobless people claiming to be network admins,) it is also true for the most part. People make pretty crazy claims about themselves to bolster their arguments online, job related lies being a very common one. Your own quick defense of using Google as a means to becoming a "network guru" seems to support my claim. Cogent is not the issue; 200ms of lag =/= 5+ seconds and losing several packets =/= a timed out connection.

[ZohnoReecho]

Yes, because you know exactly how SE manages packets loss in the game. You are right, people make bold claims on internet.