Thread: .3 Second Positional Update Delay (Requesting Dev Response)

For those of you who might want to learn a bit more about the issue of server/client sync and concurrency, here a few (technical) sources and (cultural) remarks.

________________
This is one of the most accessible (for non-technical savvy people) documentation I've found on the topic —posted here by someone else, don't remember your name but if you're reading this: thanks! ^^
➔ http://www.gabrielgambetta.com/fast_...ltiplayer.html (you may want to switch directly to part IV if you understand these concepts and want to see the final word on client/server authority)
Note that this documentation applies to fast twitch multiplayer (FPS and the likes), therefore any MMORPG should be much easier to render coherently for the player.

There's also a Valve Developer Community paper that's quite enlightening, a bit more in-depth
➔ https://developer.valvesoftware.com/...yer_Networking

And a Wikipedia article on Lag (in online gaming) that may explain not the particular issues we're having in this game (it applies to all online games), but is good general overview of the challenges that a networked game presents, and explains why developers must find workarounds such as found in the two aforementioned papers.
➔ http://en.wikipedia.org/wiki/Lag_(online_gaming)

What you can gather, reading these, and looking at how other games perform on a regular basis (the level of "network quality" you expect in your multiplayer games, be it FPS, Fighting, or indeed MMORPG), is that there is no way to argue against the fact that ARR has a serious network programming problem.

***** THE REST OF THIS POST IS TOTALLY OFF-TOPIC *****
Should you wish to discuss this any further, you may quote me and begin a new thread in this forum. Please don't do what I do, digressing way too much.
________________
To widen the issue a bit, here are few facts.

• The bulk of populated Japan areas are very urbanised and very small (geographically), many buildings are connected to the internet through symmetrical 100Mbps akin-to-LAN optic networks; thus neither latency nor bandwidth are issues there. It paints a whole different world than EU or NA for developers, who may not know first-hand what it is to deal with low-connectivity.
• However this is the country of fighting games and high-twitch arcade games, hence why companies such as Capcom do know (even innovated with) state-of-the-art network code. My sources suggest, however, that there's few turnover between studios; aside from joint projects, most workers there tend to stick to one company, even one IP within that company (the guys making PES for instance have been at it for 20 years, likewise for DMC, Bio Hazard and so on; evidently as well for FF).
• Japanese companies, for cultural (including linguistic) reasons are very closed to foreign workers; hence why it's highly unprobable they hire westerners (who would, in the case of network coding, probably be more up to the task of dealing with distant/outsourced worldwide datacenters). There are likely many issues in communicating with their Canadian subsets and partners. Notice how Japan, despite being an IT leader, is overall absent from the worldwide IT market of services —be it cloud, apps, or even mail. They are also very prone to using proprietary technologies, especially software wise, which may hinder their solution-finding processes.
• Another issue is their culture of management: it's very vertical, definitely not flat, and very uncommunicative. To over-simplify, subordinates tend not to criticise or give feedback to their bosses and team leaders because it's considered disrespectful to challenge a higher authority's opinion. Essentially, "right and wrong" matter less than "social entitlement". You can observe this in how customers respond to products: they "eat it" much more than they assess quality from a personal point of view, they follow the press take much more than they criticise, and overall there's a degree of complacency between the media and the companies (PR) that we would deem flat out fallacious in the west.

This is not to diminish or judge negatively the value of Japan's take on business; they also are very diligent workers and quite outstandingly skilled individuals; but all this hard work may not always be put to good use. I think ARR may be fairly indicative of that. Granted also, this environment is changing; albeit very slowly and not without meeting a lot of resistance, not necessarily from the people you would expect so —we sociologists observe that such a setting not only made things easy for higher-ups all through Japan's history, it also gave a sense of security to the lower end of the spectrum: "if you do well the work you're assigned, you can't be wrong about it, so questioning it is not only a waste of time, it undermines your own position". The problem being that, in a very competitive environment, it's highly improbable that a single individual may be right about everything —be it politics, economics, or… game design. You know the rest of the story.

________________
In the end, the whole SE's stance towards issues such as this (network/game design choices, silence towards customers, unwillingness to compromise over feedback that they don't already agree with in the first place, etc.) could be explained by all these facts. Then again, I'm not saying I'm right, but I can't say I'm surprised about it, knowing all this — none of it is exactly new in my book.

I should however conclude this already too long post by saying that I did (and still do to some extent) have faith in an individual such as Naoki Yoshida, but I'm not really sure he alone can change the whole culture at SE, especially when he's "just" a producer/director, and not a CEO. Yet. It may be a while…

Originally Posted by Sunarie

It's still kind of on topic. I mean, we do want Yoshida to answer. I did post a question (nicely, and hopefully well worded) in the ask thread thing for the next live letter deal.

To be honest, I am not counting on that question to be picked. Why? Because it's a live letter and they wouldn't have time to prepare a reply. Which is kinda the point, but they have never picked such important questions before, unless it also made some big waves on the Japanese side (see: Warrior). So I'm not holding my breath.

The Japanese culture probably does discourage people from speaking up against the lead guy, but it's hardly unique to Japan. I remember seeing Sid Meier giving a speech on Civilization mechanics that was completely wrong but who is going to tell Sid Meier he's wrong about Civilization? And since fans like to back the 'expert' that's probably why the said experts are never proven wrong. Nintendo was very late to the whole online thing and I'm sure their fanboys always said 'Nintendo obviously knows something better than you!', but of course now we know they really do not. Not having significant online capability on a console is just a bad idea.

Now back to the original subject. Because anyone can make a laundry list of games that do not have FF14's lag problem, that makes just about everyone more qualified than the entire FF14 development team. You don't have to know how network coding even works. It is sufficient to know that literally every MMORPG out there does not have this issue to conclude that it cannot be very hard to have an acceptable behavior for lag, and FF14's behavior is not acceptable. So the whole 'network is hard' argument doesn't hold water, because no other game had the same difficulty handling such a fundamental part of the game.

A quick note on timestamps. They're definitely not how other games avoid the 'but I dodged that' problem. If you assume the 0.3s interval to pool data is normal for MMORPG (and I'll give SE the benefit of the doubt here), then if you rely on timestamps, that means the server may have to wait up to 0.3s before relaying any information back. After all, the next update with the current client position could be up to 0.3s late, so you can't possibly make the correct decision until then. This means if this is how other MMORPG handled their lag, you can see up to 0.3s (or whatever their interval is) of lag for the same issue as FF14 when you do fail to dodge the AE. Since people obviously still fail to dodge AE some of the time but none exhibits the same lag, this cannot be the case.

Originally Posted by Astarica

** Considerations on timestamps **

Here's the particular extract I feel is worthy of attention in general sync design. You may replace "performing a headshot" with "dodging an AoE" or "interrupting the enemy's cast", in all cases it's a matter of positional assessment, it works for any action performed as/on a moving element (a mobile, or "mob", as we call them in MMORPG, here including Playing Characters).

Originally Posted by Gabriel Gambetta

Lag Compensation

So you’re aiming perfectly at the target’s head with your sniper rifle. You shoot - it’s a shot you can’t miss.

But you miss.

Why does this happen?

Because of the client-server architecture explained before, you were aiming at where the enemy’s head was 100ms before you shot - not when you shot!

In a way, it’s like playing in an universe where the speed of light is really, really slow; you’re aiming at the past position of your enemy, but he’s long gone by the time you squeeze the trigger.

Fortunately, there’s a relatively simple solution for this, which is also pleasant for most players most of the time (with the one exception discussed below).

Here’s how it works:

• When you shoot, client sends this event to the server with full information: the exact timestamp of your shot, and the exact aim of the weapon.
• Here’s the crucial step. Since the server gets all the input with timestamps, it can authoritatively reconstruct the world at any instant in the past. In particular, it can reconstruct the world exactly as it looked like to any client at any point in time.
• This means the server can know exactly what was on your weapon’s sights the instant you shot. It was the past position of your enemy’s head, but the server knows it was the position of his head in your present.
• The server processes the shot at that point in time, and updates the clients.

And everyone is happy!

The server is happy because he’s the server. He’s always happy.

You’re happy because you were aiming at your enemy’s head, shot, and got a rewarding headshot!

The enemy may be the only one not entirely happy. If he was standing still when he got shot, it’s his fault, right? If he was moving… wow, you’re a really awesome sniper.

But what if he was in an open position, got behind a wall, and then got shot, a fraction of a second later, when he thought he was safe?

Well, that can happen. That’s the tradeoff you make. Because you shoot at him in the past, he may still be shot up to a few milliseconds after he took cover.

It is somewhat unfair, but it’s the most agreeable solution for everyone involved. It would be much worse to miss an unmissable shot!

Source: http://www.gabrielgambetta.com/fpm4.html

The timestamp of that headshot would happen between regular checks (between the 0.3 seconds), making it an extra check performed by the server, based on a "presumed" authority of the client, subsequently validated by the server.

Now, a more graphical take on the same matter. Consider the following picture: imagine that the 1st-person "shooter" is an AoE firing, and you (the player) would be the running character (here towards the left). This is a good representation of what the server "sees" (hitboxes) versus what we see on our clients (textured character). The server thinks you're the blue hitbox when in fact, on your screen, you're already in the clear. Without some form of interpolation generated by your client, the hit is confirmed by the server: bloodstains on the wall (resp. floor ^^), you're dead. But if your client can send a timestamped update of your position at any time, especially when an AoE is fired, then you're in the clear, it can be reconstructed, and the hit is not validated by the server. Currently, ARR's server doesn't perform such an interpolation, thus you always see yourself in the future of what the server knows, conversely you never see AoE and castbars in the present but always in the past (it's a two-way street).


Source: https://developer.valvesoftware.com/...yer_Networking

I don't see any other explanation for what we observe in-game. As we all can see, it's perfectly documented and there are number of workarounds —validating your idea that "network is hard to code" is an argument that holds no water in 2013, nor has it for a decade actually.

In terms of network, information regarding yourself is always in the present. Information regarding other players can be in the past but that's okay since people usually care the most about themselves. For example if you look at the lag thread, you'd find logs like:
1. Player use potion and heal 100 HP
2. Player took 80 damage
3. Player died.

This is because the information from #1 didn't relay to server yet. But, this only happens in FF14. I've never heard of any other MMORPG where the above scenario is possible, and this has to be because the client wins on information regarding yourself. Otherwise, just due to the fact the it takes nonzero time for information to travel to server, there has to be a case where to the server you ought to be dead even though the client just did something to prolong his lifespan. That is, in other MMORPG, the server sends the 'you took 80 damage' message, and client says, 'but I actually got 100 more HP that you don't know about', and the server trusts the client and says okay, and you live. In FF14, there is no second part, so you just die, and the reason why there isn't the second part is because the client is not trusted for anything. Despite the fact you can point to your exact log at where you got your extra 100 HP, the server doesn't care. And this is a really backwards way to do networking. You should trust the client because it is closer to the source, and if someone hacked the client, it's not exactly hard to do even a simple analysis on the client and note that there seems to be an unlimited amount of HP coming from nowhere and then flag that.

This needs to be fixed sooner than later

The combat is based on the mechanic of avoiding AoE damage and half the time or more we cant even stun AoE damage before it goes off because our instant skills aren't instant.


cant run out of the AoE in time (game latency)

cant stun it in time (skill animation)