Letter from the Producer LIVE Part X: Will the netcode issue be honestly addressed?

[Sinth]

I used to work for several years as a systems programmer in a data center which hosted several MMOGs. Perhaps I can give a bit of insight into this problem.

The issue everyone is complaining about stems from an architectural design decision made by SE in the early stages of the game's development. The decision made was to design the game in a way which makes the server dictate the live-state of the game as opposed to the client. This is contradictory to the vast majority of online games on the market, presently, as well as in the past. I will explain below the two different methodologies.

Client Live-State:
Most of the game actions are driven by the client. However, the entities and environment are controlled by the server. When you give a command to jump, your client sends a command to the server, which acknowledges, and replies back to the client with a response to jump. Once the communication has completed, your character then jumps on your screen. Say, for instance you are fighting a monster. You attack the monster but your network connection is poor. So what happens is: You send the command to attack. There is a delay in sending the command to the server. The monster continues to attack you (successfully), as it is controlled by the server. When your connection resolves, you finally make your attack. This results in the player's experience being - 'I'm giving commands but they are delayed on my screen.' However, if a monster creates an AoE circle, it will be in real-time on the client because the server sends the command "monster A is doing an AoE attack at such and such time" When your client receives this information, it syncs to the time provided by the server, so the AoE may APPEAR late, but your reaction to it will be on time, because the client is in a live-state. So if you step out of the AoE on your client, when it goes off, your client will tell the server "The AoE landed, but the client was out of it's range", so in the end, even if you are lagging a little bit, you can still dodge the AoE.

Server Live-State:
Most of the game actions are driven by the server. When you give a command to jump, your client jumps on your screen, but in the live-state of the server you have not jumped until it receives your command to jump. So if you're lagging, it may look like you're jumping on your screen, but to the server, you are standing still. This gives the player a false sense of what is happening in the game. So how is this any different than client live-state? Well, for things like jumping its not much different, but when it comes to combat and time-intensive actions, its a whole other animal. Say for instance, monster A drops an AoE. The AoE is live on the server but you're lagging and haven't received the info yet (monster appears to be sitting there). As soon as you get the information, the monster appears to do his cast and the AoE circle appears. The difference is, you are out of sync with the server and the server rules all, so even if you step outside of the AoE, if your client doesn't get the command to the server to update your position before the AoE lands, you get hit by it. And that, my friends, is why you all died to landslide / bombs / any AoE that you were out of on your screen.

The reasons why SE would choose server dictated live-state:
-Prevents desync bugs (on the server)
-Prevents countless potential duping bugs
-Prevents (in theory) teleportation and other unauthorized client commands from being accepted
-If the player has a good connection to the server, little to no delay is observed by the player

The detractors of server live-state:
-Poor connections results in nearly unplayable conditions, specifically conditions which require fast reaction time
-Server infrastructure is more costly (servers must do much more data processing)

In the past, server live-state was extremely rare because internet connections were still poor / narrow-band.

Can it be changed?
Short answer: No.
Long Answer: Technically, yes, but it would be extremely costly and time consuming. SE Would lose millions and it would take months to over a year to accomplish.

I hope I explained things well enough for most people to understand. Let me know if you want clarification on anything.

Cheers.

[Bearssi]

Also, I must say that personally on Ifrit HM, as a stunt (PLD, DF), I must absolutely hit (complete the animation confirmed by the sound effect) before his Eruption casts reached ~75% or else the bar will keep on and eventually completes (on top of that, the bar usually appears already ~25% filled). Which means I have a fraction of a second to hit my keybind. It became trivial when I had memorised his skill rotation, but as I was learning it, boy did I fail numerous times… (and I'm used to interrupting, I did it for years in many other MMO's, with an average-to-good reaction time, and never in another game do I see a cast bar still filling up after I interrupted it).

The global cooldown which is twice longer than most casts adds up on the difficulty with the lag making twitch interrupting impossible and the interrupter is forced to have only one skill (the stun) on action bars. Pointing an example what these discussed issues effect.

[Mysteltain]

For someone who isn't a native English speaker, I've gotta hand it to you: you are amazingly eloquent. I don't think this issue could've been brought up any better than what you just did. So, well done, sir!

On topic, though, I hope this issue gets brought up as well. Having just run Stone Vigil this morning, I had a few small lag spikes, but one of them froze some of the patrolling monsters. While not normally an issue that results in a party wiping, it is still annoying when I'm chucking my axe at something (or attempting to, as the animation never finishes) that's clearly in front of me, but is in fact well behind me and on the other side of a wall. What's even worse is when a lag spike causes the other players to completely freeze in position on my screen, as then I have no idea of where it's best to tank, as well as if one of my party members have moved in front of the enemy for whatever reason. I've even had healers in my party suffer this issue, and need to spam Medica in order to heal anyone, and needless to say that just makes everyone's jobs tougher. Simply reloading the area map solves this issue (such as dying and respawning, relogging, using Return, etc.), but it's still an issue regardless.

Surely, there must be some way in which the server can help to 'catch up' without necessarily glitching and freezing character models in place.

[Hyrist]

Eloquently put, and also good on a fellow community member to explain his speculation on the issue. I do not think this issue will be addressed in the live letter. However if there was ever a well placed plea for a response, I have to acknowledge that this is it.

[Alcyon_Densetsu]

I used to work for several years as a systems programmer in a data center which hosted several MMOGs. Perhaps I can give a bit of insight into this problem.
[…]
I hope I explained things well enough for most people to understand. Let me know if you want clarification on anything.

Cheers.

Wow! That is probably the most enlightening explanation I read on the issue. Thank you very much for such an interesting read!

I have a couple of follow-up questions if you don't mind

1. Considering that in Japan, most customers are connected to the internet through very good connections (100Mbps symmetrical with quite low latency), do you think it could explain why SE chose a server live-state architecture? And do you think it's a wise choice considering the quality of worldwide connections as of 2013 (NA, SA, EU, Asia…), bearing in mind that's there's only 1 datacenter for each half of the world?

2. Do you know of any other games that are also based on such a server live-state architecture?

EDIT: Sinth made a thread of his own to make his explanation more visible, something which i wholeheartedly support, therefore I'll post my questions in there.

[BeePee]

Thank you Sinth, that's one of the all time best responses I've read anywhere on these forums. ^^

[Sinth]

Questions

Thanks, glad I could help. To answer your questions:

1a. I would say your guess is likely to be quite accurate as to why SE would choose this model of networking in their game. In addition to the advanced network infrastructure in Japan, the geographical size of the country also provides for a reduced latency due to less lengthy hops a connection has to make between server and client.

1b. I would say it was an unwise decision based on the limited availability of localized servers (2 data centers for the whole planet). If SE had licensed servers to data centers all over the world in order to serve local communities, these issues would not be so apparent.

2. No MMOs that I know of, specifically, for the reasons stated above.

[Alcyon_Densetsu]

For someone who isn't a native English speaker, I've gotta hand it to you: you are amazingly eloquent. I don't think this issue could've been brought up any better than what you just did. So, well done, sir!

Eloquently put, and also good on a fellow community member to explain his speculation on the issue. I do not think this issue will be addressed in the live letter. However if there was ever a well placed plea for a response, I have to acknowledge that this is it.

Thank you very much, both of you, for the very kind comments.

Answers

Thank you for the answers

(Also, sorry for double-posting your topic, I just wanted to switch over there to further discuss this, since you're right making your explanation an OP will make it much more visible to this community)

[Ghasto]

m-muh eu lagg
100 ping from EU, Titan HM is a breeze.

That being said, I would like them to investigate some other instance servers, story Titan is messed up beyond belief.

[theflo84]

Thank you Sinth for this Infos.

But i cannot understand why you had to do this and why not SE did this.

Why is SE still so silent about this?
Why dont they just say the things you explained to us?
Why is SE letting us fight each other about that in other threads...