Why the netcode issue exists, and why it cannot be fixed.

[Sinth]

I posted this in response to another thread on this topic but I felt it would be much more visible as an OP than it would be buried in a mountain of replies on the subject.

I used to work for several years as a systems programmer in a data center which hosted several MMOGs. Perhaps I can give a bit of insight into this problem.

The issue everyone is complaining about stems from an architectural design decision made by SE in the early stages of the game's development. The decision made was to design the game in a way which makes the server dictate the live-state of the game as opposed to the client. This is contradictory to the vast majority of online games on the market, presently, as well as in the past. I will explain below the two different methodologies.

Client Live-State:
Most of the game actions are driven by the client. However, the entities and environment are controlled by the server. When you give a command to jump, your client sends a command to the server, which acknowledges, and replies back to the client with a response to jump. Once the communication has completed, your character then jumps on your screen. Say, for instance you are fighting a monster. You attack the monster but your network connection is poor. So what happens is: You send the command to attack. There is a delay in sending the command to the server. The monster continues to attack you (successfully), as it is controlled by the server. When your connection resolves, you finally make your attack. This results in the player's experience being - 'I'm giving commands but they are delayed on my screen.' However, if a monster creates an AoE circle, it will be in real-time on the client because the server sends the command "monster A is doing an AoE attack at such and such time" When your client receives this information, it syncs to the time provided by the server, so the AoE may APPEAR late, but your reaction to it will be on time, because the client is in a live-state. So if you step out of the AoE on your client, when it goes off, your client will tell the server "The AoE landed, but the client was out of it's range", so in the end, even if you are lagging a little bit, you can still dodge the AoE.

Server Live-State:
Most of the game actions are driven by the server. When you give a command to jump, your client jumps on your screen, but in the live-state of the server you have not jumped until it receives your command to jump. So if you're lagging, it may look like you're jumping on your screen, but to the server, you are standing still. This gives the player a false sense of what is happening in the game. So how is this any different than client live-state? Well, for things like jumping its not much different, but when it comes to combat and time-intensive actions, its a whole other animal. Say for instance, monster A drops an AoE. The AoE is live on the server but you're lagging and haven't received the info yet (monster appears to be sitting there). As soon as you get the information, the monster appears to do his cast and the AoE circle appears. The difference is, you are out of sync with the server and the server rules all, so even if you step outside of the AoE, if your client doesn't get the command to the server to update your position before the AoE lands, you get hit by it. And that, my friends, is why you all died to landslide / bombs / any AoE that you were out of on your screen.

The reasons why SE would choose server dictated live-state:
-Prevents desync bugs (on the server)
-Prevents countless potential duping bugs
-Prevents (in theory) teleportation and other unauthorized client commands from being accepted
-If the player has a good connection to the server, little to no delay is observed by the player

The detractors of server live-state:
-Poor connections results in nearly unplayable conditions, specifically conditions which require fast reaction time
-Server infrastructure is more costly (servers must do much more data processing)

In the past, server live-state was extremely rare because internet connections were still poor / narrow-band.

Can it be changed?
Short answer: No.
Long Answer: Technically, yes, but it would be extremely costly and time consuming. SE Would lose millions and it would take months to over a year to accomplish.

I hope I explained things well enough for most people to understand. Let me know if you want clarification on anything.

Cheers.

[Ryios]

Choosing to make the game server sided completely and decided to only have one data center in Canada were two horrible ideas that will kill this game. Luckily they can limp along until they
A: Fix it so there is client side checking
or
B:
Open 3 more data centers and free transfers.

[Alcyon_Densetsu]

Wow! That is probably the most enlightening explanation I read on the issue. Thank you very much for such an interesting read!

I have a couple of follow-up questions if you don't mind

1. Considering that in Japan, most customers are connected to the internet through very good connections (100Mbps symmetrical with quite low latency), do you think it could explain why SE chose a server live-state architecture? And do you think it's a wise choice considering the quality of worldwide connections as of 2013 (NA, SA, EU, Asia…), bearing in mind that's there's only 1 datacenter for each half of the world?

2. Do you know of any other games that are also based on such a server live-state architecture?

[Dedayius]

I think at this point in the discussion few people don't understand what it is SE did. They just want it fixed. The easy fix is to make cast times longer so that if you happened to get a bit of lag, you would still have a reasonable amount of time to get out of the AoE. Unfortunately, the games entire difficulty revolves around getting out of AoE and so if they did this fix, it would trivialize nearly all the content, especially Titan HM.

I'm not sure what the solution is, but as more and more people hit the brick wall that is Titan HM if you have even a smidge of lag, subscribers are going to drop the game because they can't progress their character. Perhaps if you want to introduce OHKO mechanics the proper way would be more things like the first boss of Temple of Qarn where if you don't get to the properly lit square in time, you're dead.

[Blarp]

If this is true, can you explain why some people say they do not have this issue? Is it a perception issue?

[Sinth]

If this is true, can you explain why some people say they do not have this issue? Is it a perception issue?

Close proximity (or low amount of hops) to the data servers. In low-latency scenarios, the server live-state model is superior in all regards to client live-state.

[Sinth]

Wow! That is probably the most enlightening explanation I read on the issue. Thank you very much for such an interesting read!

I have a couple of follow-up questions if you don't mind

1. Considering that in Japan, most customers are connected to the internet through very good connections (100Mbps symmetrical with quite low latency), do you think it could explain why SE chose a server live-state architecture? And do you think it's a wise choice considering the quality of worldwide connections as of 2013 (NA, SA, EU, Asia…), bearing in mind that's there's only 1 datacenter for each half of the world?

2. Do you know of any other games that are also based on such a server live-state architecture?

Thanks, glad I could help. To answer your questions:

1a. I would say your guess is likely to be quite accurate as to why SE would choose this model of networking in their game. In addition to the advanced network infrastructure in Japan, the geographical size of the country also provides for a reduced latency due to less lengthy hops a connection has to make between server and client.

1b. I would say it was an unwise decision based on the limited availability of localized servers (2 data centers for the whole planet). If SE had licensed servers to data centers all over the world in order to serve local communities, these issues would not be so apparent.

2. No MMOs that I know of, specifically, for the reasons stated above.

[Sunarie]

If this is true, can you explain why some people say they do not have this issue? Is it a perception issue?

Everyone does have the issue. However, everyone does not perceive the issue because the amount of delay you get depends upon your latency. At best, you get about a .1 second delay (this has been stated by Yoshi).

Anyway, thank you for posting this. I suspected that's what was going on (without knowing anything about the back-end, nor knowing the technical side), but was unaware how easy/difficult it would be to fix. I will still hold out some shred of hope that SE will at least acknowledge it at some point.. but the hope of that is much much tinier now. I can resign myself to just playing until I find it more frustrating than fun to play. Currently I can dodge things a large majority of the time, some of my FC mates on the other hand aren't so lucky.

[TrystWildkey]

Thank you for providing detailed, logical information on this topic. Especially in what can be a very thankless environment.

Just wanted to let you know that I appreciate your post!

[beforethechasm]

...I'm not sure what the solution is, but as more and more people hit the brick wall that is Titan HM if you have even a smidge of lag, subscribers are going to drop the game...

I believe this is why there was talk of adding buffs to a party if they continuously die.