Why the netcode issue exists, and why it cannot be fixed.

[Sinth]

I posted this in response to another thread on this topic but I felt it would be much more visible as an OP than it would be buried in a mountain of replies on the subject.

I used to work for several years as a systems programmer in a data center which hosted several MMOGs. Perhaps I can give a bit of insight into this problem.

The issue everyone is complaining about stems from an architectural design decision made by SE in the early stages of the game's development. The decision made was to design the game in a way which makes the server dictate the live-state of the game as opposed to the client. This is contradictory to the vast majority of online games on the market, presently, as well as in the past. I will explain below the two different methodologies.

Client Live-State:
Most of the game actions are driven by the client. However, the entities and environment are controlled by the server. When you give a command to jump, your client sends a command to the server, which acknowledges, and replies back to the client with a response to jump. Once the communication has completed, your character then jumps on your screen. Say, for instance you are fighting a monster. You attack the monster but your network connection is poor. So what happens is: You send the command to attack. There is a delay in sending the command to the server. The monster continues to attack you (successfully), as it is controlled by the server. When your connection resolves, you finally make your attack. This results in the player's experience being - 'I'm giving commands but they are delayed on my screen.' However, if a monster creates an AoE circle, it will be in real-time on the client because the server sends the command "monster A is doing an AoE attack at such and such time" When your client receives this information, it syncs to the time provided by the server, so the AoE may APPEAR late, but your reaction to it will be on time, because the client is in a live-state. So if you step out of the AoE on your client, when it goes off, your client will tell the server "The AoE landed, but the client was out of it's range", so in the end, even if you are lagging a little bit, you can still dodge the AoE.

Server Live-State:
Most of the game actions are driven by the server. When you give a command to jump, your client jumps on your screen, but in the live-state of the server you have not jumped until it receives your command to jump. So if you're lagging, it may look like you're jumping on your screen, but to the server, you are standing still. This gives the player a false sense of what is happening in the game. So how is this any different than client live-state? Well, for things like jumping its not much different, but when it comes to combat and time-intensive actions, its a whole other animal. Say for instance, monster A drops an AoE. The AoE is live on the server but you're lagging and haven't received the info yet (monster appears to be sitting there). As soon as you get the information, the monster appears to do his cast and the AoE circle appears. The difference is, you are out of sync with the server and the server rules all, so even if you step outside of the AoE, if your client doesn't get the command to the server to update your position before the AoE lands, you get hit by it. And that, my friends, is why you all died to landslide / bombs / any AoE that you were out of on your screen.

The reasons why SE would choose server dictated live-state:
-Prevents desync bugs (on the server)
-Prevents countless potential duping bugs
-Prevents (in theory) teleportation and other unauthorized client commands from being accepted
-If the player has a good connection to the server, little to no delay is observed by the player

The detractors of server live-state:
-Poor connections results in nearly unplayable conditions, specifically conditions which require fast reaction time
-Server infrastructure is more costly (servers must do much more data processing)

In the past, server live-state was extremely rare because internet connections were still poor / narrow-band.

Can it be changed?
Short answer: No.
Long Answer: Technically, yes, but it would be extremely costly and time consuming. SE Would lose millions and it would take months to over a year to accomplish.

I hope I explained things well enough for most people to understand. Let me know if you want clarification on anything.

Cheers.

[Ryios]

Choosing to make the game server sided completely and decided to only have one data center in Canada were two horrible ideas that will kill this game. Luckily they can limp along until they
A: Fix it so there is client side checking
or
B:
Open 3 more data centers and free transfers.

[Alcyon_Densetsu]

Wow! That is probably the most enlightening explanation I read on the issue. Thank you very much for such an interesting read!

I have a couple of follow-up questions if you don't mind

1. Considering that in Japan, most customers are connected to the internet through very good connections (100Mbps symmetrical with quite low latency), do you think it could explain why SE chose a server live-state architecture? And do you think it's a wise choice considering the quality of worldwide connections as of 2013 (NA, SA, EU, Asia…), bearing in mind that's there's only 1 datacenter for each half of the world?

2. Do you know of any other games that are also based on such a server live-state architecture?

[Sinth]

Wow! That is probably the most enlightening explanation I read on the issue. Thank you very much for such an interesting read!

I have a couple of follow-up questions if you don't mind

1. Considering that in Japan, most customers are connected to the internet through very good connections (100Mbps symmetrical with quite low latency), do you think it could explain why SE chose a server live-state architecture? And do you think it's a wise choice considering the quality of worldwide connections as of 2013 (NA, SA, EU, Asia…), bearing in mind that's there's only 1 datacenter for each half of the world?

2. Do you know of any other games that are also based on such a server live-state architecture?

Thanks, glad I could help. To answer your questions:

1a. I would say your guess is likely to be quite accurate as to why SE would choose this model of networking in their game. In addition to the advanced network infrastructure in Japan, the geographical size of the country also provides for a reduced latency due to less lengthy hops a connection has to make between server and client.

1b. I would say it was an unwise decision based on the limited availability of localized servers (2 data centers for the whole planet). If SE had licensed servers to data centers all over the world in order to serve local communities, these issues would not be so apparent.

2. No MMOs that I know of, specifically, for the reasons stated above.

[Alcyon_Densetsu]

Answers

Thank you for these answers.

(no more double-posts!)

I suspected something was off, but not at this magnitude. Much like Sunarie, I didn't know the technical side, though. At least now I can put words on it. That's a mild consolation, even if it somehow fails to offset my whole disappointment over this realisation, but… Oh well, at least I'm left with an interesting topic to further dig into in my spare time. I suppose we'll play nothing but server live-state games a decade or two from now!

Who said SE isn't focused on the future? Though, admittedly, perhaps a bit too much for their own sake…

______
EDIT

This doesn't make sense to me.

QA testing could easily be done by Square Enix Headquarters USA, which is located in California, and more than likely was. Granted, this is California to Japan up until the later Beta testing. However I wouldn't go so far as to presume that they did not test, at least in the mid stages, the connection quality of the netcode outside of region.

How their connections would be solid while others are having difficulties, I cannot know. Perhaps this issue did not articulate itself gravely until stress tests began, which at that point it would be too late. But this is all just speculation on my end.

These are fair questions, especially after Phil Rogers (SE's NA/EU CEO) stated that teams from all around the world worked on ARR's development.

Also, they involved players fairly early in the process (since alpha testing), and the issue was mentioned to SE repeatedly (I only arrived during beta 3 in june, I don't know first-hand about before though, but it was clearly a topic when I got hands-on the game). At the time they blamed "server-side checks that would be disabled come release"; yet very late during the beta process they did offset the whole red zone by several hundreds milliseconds (making it appear earlier than previously), as it used to be much more penalising in the beta, downright impossible to dodge. At the same time it somehow broke immersion though —the obvious "dodge the red zone rather than the actual spell animation", which somehow seems ridiculous that you have to dodge a UI indicator but still can dance in the subsequent fire without concerns…

[IzumiRaito]

2. Do you know of any other games that are also based on such a server live-state architecture?

Path of exile is like this. And suffers from desyncs and rubber banding. Although its not same game type. Diablo 2 and 3, but its blizzard so problems are minimal.

[Dedayius]

I think at this point in the discussion few people don't understand what it is SE did. They just want it fixed. The easy fix is to make cast times longer so that if you happened to get a bit of lag, you would still have a reasonable amount of time to get out of the AoE. Unfortunately, the games entire difficulty revolves around getting out of AoE and so if they did this fix, it would trivialize nearly all the content, especially Titan HM.

I'm not sure what the solution is, but as more and more people hit the brick wall that is Titan HM if you have even a smidge of lag, subscribers are going to drop the game because they can't progress their character. Perhaps if you want to introduce OHKO mechanics the proper way would be more things like the first boss of Temple of Qarn where if you don't get to the properly lit square in time, you're dead.

[beforethechasm]

...I'm not sure what the solution is, but as more and more people hit the brick wall that is Titan HM if you have even a smidge of lag, subscribers are going to drop the game...

I believe this is why there was talk of adding buffs to a party if they continuously die.

[Dedayius]

I believe this is why there was talk of adding buffs to a party if they continuously die.

Interesting. Is this a community idea or one that the devs tossed out?

[xanj]

so basicly what your saying is what we have always known se doesn't care about its over seas market