Why should I need to post a video when literally everyone else is having the issue? I've been saying this shit since beta. I was one of the first people to bring this issue up. It isn't a client-side issue, it is a check-delay built in to the game to compensate for people that have terrible connections. It's like what they used to do with FPS' in the 90's. I have seen it first hand when people bring their laptops over to my house, or vice versa. If I jump, my jump doesn't even begin until well after I have LANDED. I have seen it on three different ISPs.Quote:
Originally Posted by Garnatian
If you guys actually want me to make a video, I will. I will post it tomorrow or monday.
EDIT: when I said I can mount while running I didn't mean I can literally complete the cast and go on my mount while still running, but the cast almost finishes before it actually interrupts.
Why the netcode issue exists, and why it cannot be fixed.
Printable View
-
11-17-2013, 02:41 PMGravehill
-
11-17-2013, 02:51 PMAlcyon_DensetsuTL;DR at the end if one wishes to respond quickly.Quote:
Originally Posted by Althea
This logic sounds nice, however I don't have a sufficient tech background and experience to give a "definitive answer", so I'll wait for someone to do that. I like the elegance of it, mathematically. ^^ However I can see a few concerns, based on my personal experience as a gamer, and what little I think I know (or assume… ^^) about networks.
________
Consider the following example:
http://i43.tinypic.com/35izsr4.png
• A, B, C, D are regular positional checks (each .3s)
• "exit" and "re-entry" are the key moments to determine whether or not a player might be hit by the spell that will occupy the red area.
• you can observe the deviation between a "perfect" escape (straight thin black line) and the actual path of an average player (curved thick black line).
This is a fairly realistic example of what a player does I suppose: starting from A, you move loosely in a straight line, passing by B, then C, and finally come back close to the mob to keep fighting.
Now in this example, one exits about .5s after seeing the AoE, but one fails to wait enough and re-enters the AoE before the circle (or rather, enemy cast bar) disappears, about .8s after seeing the AoE. Hence, one should be hit when the AoE fires, about 1s after the AoE appeared on one's screen (that's anything from 1.02s to 1.3s after the red circle appeared from the point of view of the server, depending on one's internet latency).
If I understand correctly, you suggest that in addition to regular positional checks (A…D), the client sends a timestamped additional check at the exact time of exiting (~.5s) —and, subsequently, at the exact time of re-entry, should it happen (here at ~.8s). Right?
________
Here are my concerns:
1 • The "simple math" (difference between A and "exit point") is not that easy, for you need twice the data currently exchanged: you're going to compare between the straight line theoretical minimum and the actual difference between A and the "exit" timestamp, but for this to happen the client needs to send both the "exit" timestamp and the timestamp of when it received A (and displayed the red area to the player) as well (or else you don't solve the latency problem between A on the server and A on the client).
2 • "Zigzags" may add a significant strain on the servers (here +50%; at least +25% in an ideal scenario, and it adds stress to the clients too —see below PS3 considerations in my 4th point).
This is an unpredictable stress, because you don't what players will do, and they might try (especially in PvP) to flirt with the limits (at the risk of being hit) to beat mobile opponents. Actually, the whole problem remains since in this server-live state model, everything you see is "in the past" on your client (your solution doesn't change that fact, it merely compensates for it by interpolating, which is probably the usual method, but it works much better in a client live-state model). Trying to compensate for that discrepancy ourselves, we actually intuitively tend already to re-enter former red areas asap, knowing that we're free of the consequences of the spell animation itself, and that we've already lost a lot of time just waiting for the AoE indicator to appear (we try to compensate for loss of DPS, tank threat, etc.)
Never mind that it further breaks immersion, which is related but yet another topic entirely :)
3 • Assuming this game does indeed use TCP instead of UDP, all checks must happen in order (no asynchronous resolution of requests or reconstruction of situations), therefore bottlenecking the whole succession of events (can't perform check C before "exit" is received for instance).
This might seem trivial, but how can the server know that a client-prompted "exit" check is about to happen? if "exit" is too close to C, it won't be taken into account (it would violate TCP order, and if causality is violated would force a short rollback, which is probably way too much stress for this server model; and at the very least could cause rubber banding and freezes).
Now taking into account the "re-entry" point, the server might have sent a "clear" confirmation to the client (which must happen before C); then the client must send a timestamped re-entry check request, and we're back with order/causality issues before check D happens and is sent to the client. Without UDP, it must all happen in order, there's no reconstruction or fixed displaying delay.
All things considered, I think it's a strong argument explaining why you can't have server live-state and client live-state simultaneously (regardless of TCP/UDP), it's just not an option for it would violate causality all over the place. Who gets the final word, client or server? It has to be one or the other. I think.
4 • A solution might be to delay C & D validations for "some time" before sending to the clients (perform the check but wait "some time") to make sure the client isn't about to send an additional check request; it may work but then this is equivalent to adding "some time" latency — which this game probably doesn't need.
Also, more client requests = more stress, as stated already, and from what we read of the developers, the PS3 clients are already pushing the limits of the console. It probably can't perform these additional check requests without impacting the display of the flow events (rubber banding issues likely to happen). And actually this lack of PS3 resources might explain why they went with server live-state (the console probably just can't be relied upon for authority). I'd very much like to hear Sinth (or anyone else versed in MMOG client/server sync) about this.
________
TL;DR: This method of yours addresses the discrepancy between "server reality" and "client delay" at the heavy cost of sending (probably too much) more data, which is that much unpredictable stress for (some) clients and servers. A ping of 200-300 ms, not rare when there's only 1 datacenter for each of Earth's hemispheres (and furthermore none for the southern one), may be enough to break the whole solution you propose, since such a ping is potentially more than the regular (TCP-ordered) positional check already in place in this server live-state model —back to causality issues for server live-state regular checks (A…D) to happen, short of (too) many rollbacks, which would probably result in other issues such as rubber banding display on the most latency-impacted clients, or resource-limited ones.
I would refer to this old post I made on the issue, before I knew about live-states: not my opinion, I was just sharing interesting sources (a programmer named Gabriel Gambetta, and a Valve doc). I suppose it provides an illustration of timestamp methodology and discrepancy between server/client point of view, and I now assume these sources consider a client live-state model in which the client has the authority (only to be confirmed or invalidated by the servers).
I really don't know if this post made sense. Which is why I tried to put the "concerns" in bold, but you can probably ignore the "explanations" I provide as they must be deeply flawed. haha ^^; -
11-17-2013, 02:52 PMTashigiIMO, you'd be wasting your time.Quote:
Originally Posted by Gravehill
If people can't "see" this issue, then I am not even sure what games/MMO other than this they've been playing.
In my 12+ years of MMO history I haven't seen anything like this, outside of FFXI; where it didn't matter anyway since the mechanics were totally different. FFXIV in the other hand, is a reaction based game, but we're all left playing a memory based game due to the issues.
And yes, I was mentioning this during all the BETA phases.
It was identified and the developers themselves have noted what causes it but haven't spoken nor address how they plan to fix, or if they plan to fix it at all. -
11-17-2013, 03:12 PMAlcyon_Densetsu
@Gravehill & Tashigi: I remember some of your posts. Back in open beta, I also remember a friend of mine (computer science engineer, though he works with databases and not at all with networks now, but he had a valuable tech education in general comp. science) commenting on the late and poor bandaid of a fix that was implemented (essentially to display the red areas ahead in time to give players more time). He said something along the lines of "notwithstanding that it kills immersion, it makes it generally too easy to avoid AoE's [he was referring to avoiding the red zone rather than the spell itself], and if there's latency it just won't work." Well, now I know what he meant…
Anyway, don't bother providing more evidence of the issue. It's now behind us, we know there is an issue, now's the time to hear from SE, as I'm relentlessly trying to suggest them to. -
11-17-2013, 04:39 PMAstarica
Timestamp doesn't get around the fact that server doesn't have the data at the time it makes its decision. Let us imagine this sequence:
Time 0.0s - Client sends position, player is in AE.
Time 0.01s - Player moved out of AE.
Time 0.15s - AE fires on server.
Time 0.30s - Client sends position, player is out of AE.
The server at time 0.15 only knows about your position at 0.0s (and earlier). It has no way of knowing your position at 0.01s. That information, even timestamped, comes at the next update at 0.30s. Currently, the server doesn't wait for the next update and just decides based on your last known position, so you get hit.
So I guess the natural thing to ask is, can't the server wait for the next update? Sure, it can do that, which means its decision will be rendered 0.15s later (and in general it can be up to 0.3s later). Now I'm sure all of us have seen the case where a guy managed to run really far after the castbar ended and then get hit half a screen away. Now you'd have even bigger delay if you indeed did not get out of the AE (because the server has to wait longer). While this method would yield the correct result, I'm not sure if you can add more lag to this game and have an acceptable playing experience, even if it is the correct one. It's common to get hit by an AE about half a second after you see the castbar end. With this method it can be up to 0.8s. It'd simply be a very jarring experience to get hit by an AE that late, even if the result is correct. If you indeed dodged the AE, this method would always credit you for dodging it. But if it turned out you actually didn't dodge the AE, you can expect even worse lag between when the AE hits compared to now. I find the lag in this game is already very disruptive, and I think adding more lag, even if it benefits the player, is likely going to produce an unplayable product. -
11-17-2013, 04:49 PMRisaeThats what you get when you use the same developers who made 1.0 and 11 do an action MMORPG.Quote:
Originally Posted by Tashigi
I don't think they know that "latency" matters here. -
11-17-2013, 06:18 PMAPilgrimI respectfully disagree that this is the case here. And here's some examples that anyone can observe to back up why I think the conclusion regarding the client-server dictation of the live-state is wrong:Quote:
Originally Posted by Sinth
1) Most recently, the gil hack where a player could purchase something off the market board using another players identification (and therefore be using another players gil to purchase something). This is/was only possible because the servers in FFXIV tend to implicitly trust the information from any given client to an extent that is not the norm in more modern MMOs. This type of hack is "old-school" because more sophisticated and secure (modern) MMO design does not implicitly trust the client. Simple hacks like this are accomplished by simply editing memory on the client platform (the easy way, and the same way trainers work in single-player games) or by editing outbound packets coming from the client (the harder way that usually requires a little technical knowledge regarding packet editing and obfuscation or cryptology). Either way, this type of hack is only possible because the client is telling the server what is happening and the server is simply trusting and accepting what the client is telling it. It's the kind of hack you'd find in older games like Maple Story and earlier.
But this hack has nothing to do with Client vs Server Live-State you may be saying to yourself? But it does, and here's another example that explains why and shows that FFXIV is essentially a Client Live-State specifically regarding player location:
2) Teleporting bots. Happily zipping around interacting with mining/botany nodes or farming NPCs. Once again, an "old-school" hack that is only possible because the server implicitly trusts what the client is telling it (i.e. only made possible by a client-dictated live-state). If it were the case that FFXIV operates in primarily a server live-state, we'd be seeing "speed hacks" at most instead of teleporting characters. Why? Because speed hacks are essentially the client only getting away with as much as the server allows - but ultimately the server has the final say and knowledge regarding player position and "knows" that what the client is telling it isn't possibly correct. Teleporting hacks occur only when the client has complete and ultimate say on character position within the game-world and the server trusts client to dictate what the live-state is (i.e. a client live-state like exists in FFXIV).
There are additional examples I could bring up but these 2 are the most well-known to the average player and prove the point regarding the client-server relationship in FFXIV - but I don't want to digress the topic too far into hacking FFXIV. Hacks just demonstrate the inner-workings of the client-server relationship better than subjective observances regarding AOE indicators from any given player that may be subject to varying degrees of lag/latency. And the hacks we see in this game are not the result of the server dictating the live-state of the game - in fact, the hacks we see result from the client being given too much trust in dictating the live-state of the game to the server. And which most modern MMOs have better designs to defeat and cases of where a server live-state would make extremely difficult to impossible.
Then what's the deal with all the complaints regarding avoiding AOE hits?
In this case, the server has to tell the client something that impacts the live-state of the game (NPC actions having a live-state impact on the client/character/player). And when it does, it doesn't match up with what the player is seeing on their client. Why? Because FFXIV primarily runs in a client-dictated live-state and not the other way around!
In a server dictated live-state, the server would be telling the client where the character is located (still in the hit-box of an AOE action) - and the result would be that the client would show the player's character still being located in a position to be hit by the AOE action when it occurs. Or in other words, the NPC action hitting the player would appear to the player as being coherent because their character would literally still be shown to them as still being within the area the AOE is hitting... the catch would be that their controls/input that were used in an attempt to move their character out of the AOE indicated area would appear to not work or be "laggy" to the point they were unable to get their character out in time. That, or their character would rubber-band back to being in a position to be hit.
In more of a client dictated live-state, the client would show the player as having responsively and clearly moved out of the AOE indicated area (if they actually do so)... and yet still possibly receive notice from the server that their character was still hit by the AOE. Which is the case here with all the complaints.
As an example, look at Coincounter (the 2nd boss of AV). He doesn't show AOE indicators (so we can even remove the indicators from the discussion) and has both a frontal AOE and a big 360 degree AOE. In the case where a tank sees he is gearing up a swipe attack (frontal AOE), the tank can run forward and through Coincounter and be standing behind him to avoid it.
If the tank/client/player is impacted by the issue and receives a hit irregardless of standing behind Coincounter (and in theory, making being hit impossible): the tank/character will be flung off in a certain direction. If the live-state were dictated by the server, the tank would be flung off in a direction away from Coincounter towards his front (as if the character were still standing in front of Coincounter and being flung away correctly). If the live-state were dictated by the client, the tank would be flung off in a direction behind Coincounter because this is where the client says the player is. And this is what happens in FFXIV in this fight: the client maintains where the character is located correctly and even responds to the hit correctly (by making the tank fly away in a direction behind Coincounter) even if it makes no coherent sense to the player.
The server doesn't care and isn't involved with where the player is located. Simple as that. All it is/was concerned about is very briefly being involved in the live state (likely even just one packet) by telling the client if the player was hit or not - and that's where the issues ultimately lay. And it's an issue that may even be fixable by involving the server more and not the other way around (causing the issue).
If nothing else, apply Occam's Razor: Is it more likely to a problem with SE using a more costly cutting edge server-client design... or more likely that SE is using a rather antiquated and cheaper design?
I'm going to go with SE using a more antiquated, hackable, cheaper, client-server design (and I'm not a hater of the game or SE). -
11-17-2013, 06:59 PMAlcyon_DensetsuNo offense, but your take on this seems more "deduced" and less "informed" by technical background in operating MMOG datacenters. I do respect your opinion but it doesn't seem as "definitive" as Sinth's —which was less discussing an opinion than a fact, at least in the way he worded it. Seems the man is quite positive on his analysis. That being said, I can't really comment on its validity, so I'll let others do that.Quote:
Originally Posted by APilgrim
However I have a few questions that don't seem addressed by your explanation.
1) How do you explain that people can cast while running, to the extent of mounting or completing Esuna (1s cast)?
I'm asking this because I suppose that, should the client have authority, it just wouldn't let you do that at all. Not for .1s or even .01, it just wouldn't cast at all. Which is what I've observed in pretty much all other MMO's on the market.
2) How is database hacking related to live-state?
I'm wondering because, as far as I know, the database of ARR was accepting SQL requests without any check whatsoever, which was the (rather rookie) security flaw about it. To rephrase that: it wouldn't check within its own data that a request was possible, it would just process it —no need to "justify" the likeliness of an event for it to be validated (no check, no validation), requests would simply be processed. Pretty much as if we were living in a world where cheaters wouldn't exist and SQL requests were so impossible to write that SE thought no one would try (and evidently succeed). Likewise, teleport hacks are just that from what I gathered: a form of database hacking, where you don't ask the server to process a movement (which I assume would not result in a teleport but instead in very fast movement), but simply update your position in the database which doesn't check at all if it's even possible based on your previous position. Also from what little I know about these early gil generation hacks (and instant-leveling, item generation, etc.), it seems the SQL requests were in plain, not encrypted at all (like your session ID was written in the .exe parameters, without any obfuscation or encryption whatsoever, in plain memory). Apparently this lack of protection, on top of a lack of checks, made the hacks ridiculously easy to perform for a tech-savvy individual.
3) Contrary to your AV example, how do you explain that against Titan, people sitting behind him at the time of a Landslide release would indeed be ejected in the opposite direction of their relative position? (you are behind but you get ejected in front; this rarely happens but I've seen it first-hand)
To me this rather shows that the server saw the player still in front of Titan, regardless of what the client displayed to said player. I would suppose that if the live-state were on the client, you'd always be thrown away "logically" on your screen; however what often happens is that what goes on the server just isn't what we see, or more precisely we see it with a delay.
4) Finally this is a less technical question pertaining to your answer based on Occam's Razor (I love this logic too). Considering that:- SE is known to be using cutting-edge technology in their development processes. They were the first game studio in the world to buy Silicon Graphics workstations back in the late 90's, they produced one of the most advanced CGI movie in the early 2000's, and to this day few, if any, can rival their mastery of 3D artistry.
- SE is also known to produce some of the most complicated, heavy coding in games, way too much so for the sake of (coding) elegance and efficiency, making me often call them Rube Goldberg machine-makers. This can also be seen with their web design and overall lacking knack for simplicity in UI's (just look at ARR's overcomplicated menus, absence of three-clic rules —which is often dubbed down to 2 in games— and so on). I think this post explains my feeling well.
___
EDIT: daily post limit reached (DOH!) — I'm so sorry APilgrim, I can't reply below :(
:)Quote:
Originally Posted by APilgrim
Alright. Thanks for taking the time to answer :) I can't solve either way.
I'm still not really convinced by the casting-while-running bit though, since I really never saw that happening. I mean, if you issue a command and the client has authority, it would simply refuse that command if it's forbidden, without even needing to check with the server (that's the whole point of client authority on live-state, as I understand it, much less server-checks required, thus much less stress on said server). I suppose. Unlike you, I don't have work experience in computing besides some very basic web dev (html/css with a bit of php/ajax occasionally) and a keen interest for all things technical, so I'm really not an authority on the matter.
About these database checks… :p … what I read (unreliable source, but the guy seemed to know first-hand), ARR's database wasn't protected at all, and you could directly send SQL requests. No hacking per se, just knowing the right syntax. Which made me laugh at SE, since even a basic website is more secure than that.
As for Occam… haha yeah, doesn't take much knowledge in ergonomics and web design to see that SE's just bad at it. However that's not just them, many designers I know repeatedly commented that the Japanese scene was often subpar to most UI / ergonomics standards out there (good graphics, bad ergonomics) (the US being notably better than most, since they focused so much on communication and efficiency for decades). I see what you mean about SE being a huge corp., but I think we agree it's an explanation, not an excuse ^^; I mean, Apple and Google are huge too…
Yeah, a good food for thought is always nice. :)Quote:
Originally Posted by Dyrstahl
-
11-17-2013, 08:21 PMWaliel
Would disabling TCP-ACK through registry help this problem any? Dodging the AoE, that is. I'm pretty sure it did help a lot for me when I did it a week or two after live. Seeing as this game uses TCP, having it on can increase the latency by up to 200ms or so.
-
11-17-2013, 08:22 PMAPilgrim
No offense taken, I didn't mention my technical background because it's the internet where anyone can be anything they want and I thought actual examples that everyone could observe would be more convincing and less argument prone. But I have a CCNA (routing and switching) background [and a misspent youth hacking games, if that means anything to anyone]. I'm by no means an MMO game developer - but fortunately no one needs to be to discuss this topic as it effects players. And we are all deducing here unfortunately since SE hasn't responded with any facts...
1) This issue could occur in any case regardless I'm sure (most likely due to latency/lag) - but I would point out that the client is getting away with something it shouldn't precisely because the server isn't doing anything to stop it... and still the server also has no problem with accepting the client telling the server that a spell was cast successfully and accepting the outcome among other clients.
2) Sounds like what you said and what I said are in agreement? But it's related because it shows that individual clients are running the show and telling the server what the live-state is. The database is just updated with what the clients say it should be even if the new information shouldn't make any sense to a server that may be playing any sort of an important role in the processes. I'd also go out on a limb and say the database would not be attacked directly with either hack though, and instead just normal client actions would be altered. Most probably the identity of one player was replaced with the identity of another player in the act of buying something on the market board (hence why some players were receiving insufficient gil messages if their character ID had been targeted for use with the hack), for example. The result being the server updates the database itself without question because it trusts what the client is telling it. No need to attack the database directly (I suppose it's possible - just not as likely).
3) I would think in that case (and if it's more of an exception rather than the rule), latency/lag might also be playing a role if the direction doesn't make coherent sense with what the client is showing you on screen, or your character position suddenly jumps in position. But programming could be doing some funky things in different situations, fights, effects. And I was only addressing the norm of what happens - there can always be exceptions of course.
4) This actually made me laugh because I was just ranting to a friend about how bad SE's website, customer-service, and account management still is - it hasn't changed much since at least FFXI came out (my first real exposure to SE in a big way). But yes, I absolutely agree SE has a tendency to make things be overly-complicated and unintuitive for the end-user. I'd say a lot of that though has to do with SE being a huge international corporation with many, many, departments and segmentation. I can't really speak for the programming (I'm an amateur programmer at best and haven't even thought about trying to dig into something as big as a game client such as FFXIV). BUT, considering the length of development (and redevelopment) I think it's likely the programming has gotten quite complicated by this point (and also is less likely to be something cutting edge considering the years involved).