Server Disconnecting Clients - Wireshark Proof - Server sending FIN TCP flag- BUG

[Moronica]

One more thing:
In case you use any kind of anti-virus software, Windows Defender or similar, please try deactivating it (completely if possible). I just remembered that we had a completely crazy incident years ago at work were crapware of that sort actually killed TCP connections when it detected a specific string in the data being transmitted.

[Einmimiria]

Good afternoon and thank you for posting on the Final Fantasy XIV Technical Support Forums DoomAngel.

To clarify, You only have this issue on your account, regardless of device. Do you only have this one character? Do you have other characters on other servers you've had this issue with? Would it be possible to create a character on another Data Center (preferably one where you take a different connection route, such as a European Server) to see if you continue to disconnect?

We hope we can be of assistance in helping you enjoy the exciting world of Final Fantasy XIV!

[DoomAngel]

Yes, this issue is only on my account, regardless of device. I'll create a character and sign up for a VPN service to test those options in the next day or so. Will keep you all posted.

[DoomAngel]

All TTL values from the FFXIV server (204.2.229.93) are the same at 54, after reviewing the wireshark data, including the FIN, followed by RST.

[Moronica]

A TTL of 54 makes sense. You’re about 10 hops away from the server and 64 is a common TTL default value. That definitely looks like the server closed the connection, either on purpose or the server process crashed and the operating system closed the TCP socket.

(The RST after the FIN looks like an effect I’ve seen before (on Linux): if a TCP socket is closed while there is still data in the receive buffer, or data arrives after the FIN has been sent, a RST is sent to let the other side know that not all data received was delivered to the application. In your case the last two chunks of data (336 bytes sent at 205.284423 and 112 bytes sent at 205.296661 were not acknowledged by the server.)

Everything up to the FIN looks fine: timely ACKs for received data from both sides and window sizes are in the green. I don’t see anything going wrong here. Of course we don’t know anything about the actual data being exchanged and whether it conforms to whatever application layer protocol FF XIV uses. The only thing different (compared to your other screenshots) is that this screenshot only shows data from one TCP connection — due to your filter (tcp.port eq 58899 and tcp.port eq 55026) — whereas your other screenshots showed that there are actually two TCP connections involved, both to port 55026 on the server. I have no idea why a FF XIV client needs/has two TCP connections to the same port on the server.

I’d say it’s time for someone from Square Enix or the data center to ask you for the exact timestamp of a FIN from your Wireshark capture so they can have a look at the server’s log and figure out why the heck the server closed the connection. To keep pointing fingers at a customer’s ISP doesn’t cut it this time IMHO…

PS: Thanks to https://convertio.co/ocr/ for their nice online OCR service . Worked like a charm on your screenshots.

[Packetdancer]

Everything up to the FIN looks fine: timely ACKs for received data from both sides and window sizes are in the green. I don’t see anything going wrong here. Of course we don’t know anything about the actual data being exchanged and whether it conforms to whatever application layer protocol FF XIV uses. The only thing different (compared to your other screenshots) is that this screenshot only shows data from one TCP connection — due to your filter (tcp.port eq 58899 and tcp.port eq 55026) — whereas your other screenshots showed that there are actually two TCP connections involved, both to port 55026 on the server. I have no idea why a FF XIV client needs/has two TCP connections to the same port on the server.

I haven't bothered to poke into the FFXIV networking protocol, so take this with a grain of salt, but... some games go for having multiple simpler networking protocols, like one for the state of a given session within an instance, another for the overall session. It wouldn't surprise me if FFXIV had something like that; one for your overall 'my character is logged in', and one for the connection to a given instance.

In a situation like that, the session service could tell the client something like "Okay, now you're zoning into Weeping City of Mhach, your instance connection should be to <raid instance server>" and it makes a second connection there. And when you leave the instance and it wants to put you back somewhere, the session server goes, "Okay, your instance connection should now be to <physical server that's running Limsa Lominsa for Spriggan>." It can be much simpler to maintain the one lasting session connection and then connect separately to each server that's running wherever/whatever you are, rather than having the server you talk to have to talk to the others behind the scenes, or having you forced to disconnect from the main server entirely to reconnect to other ones.

So two connections makes a lot of sense to me; it's how we were handling things with a game engine I helped write (before I burned out on game dev crunch time and bailed to less insane software development), back when we were modifying a fork of it to be used for an MMO. As the first game we were going to be doing atop the MMO variant of that engine got cancelled and I left before the second game atop it released, I don't know if that's the networking model they stuck with. But it was what we designed to start with while I was still in the room, and it wouldn't surprise me if FFXIV did something similar.

Honestly, it would also explain separate IPs for DoomAngel's account and DA's wife; the architecture we'd been discussing was that you had a 'world' server (in FFXIV terminology, this might be Spriggan), then several session/user servers (responsible for storing the data of specific individual users and managing their sessions) which the world server would point you at during login, and then the individual instance servers you'd be pointed to. If FFXIV uses a similar design, it might just be that the two characters are on different user/session servers.

My gut feeling—and this is spitballing, at this point—is that if we're only seeing one connection, and we're seeing packets go out and not get acknowledged, if something along the way is dropping outgoing packets (or mangling them in some way such that the FFXIV servers consider them invalid packets and drop them) when they're being sent to that specific IP address that seems to be authoritative for DA's account... well, I do know that if FFXIV doesn't receive valid heartbeat messages when it expects them, it will shut down the connection.

I'd still recommend trying the VPN approach, and the make-a-new-temporary-character approach. Making a new character might demonstrate the issue is IP-specific if that character ends up on a different user/session server within the same world and doesn't have the problem, or if the character ends up on the same one and does display the same problem. The VPN approach could show the issue is something along the route between DA and SquareEnix's data center (if approaching the server from a different network resolves the issue).

In some ways, I hope the VPN doesn't resolve it, because if it does it means the issue is in router configuration somewhere along the route between DA and the FFXIV servers, which would be a royal pain to track down (and presumably fall outside of SquareEnix's control). Which wouldn't be ideal, because I do imagine DA is not the only one affected by this, and if it's outside of SquareEnix's control there's not a lot they can do to resolve it for others. And even if the VPN works to solve things for DoomAngel right now, there'd be others who were affected by it who aren't posting here and wouldn't see such a solution get offered.

If the issue can be shown to exist regardless of how DA comes at the server IP (i.e. still exists over VPN), it means it's probably in SquareEnix's part of things; while that's less convenient for DoomAngel in the short run (because it means using a VPN isn't a workaround), it's probably better in the long run since then SE would have the ability to track it down in their stuff and presumably fix it for everyone affected.

Edit: unrelated, thank you for the mention of Convertio. I actually have been trying to find a web-accessible OCR service for a project that will need to do OCR as a stage of an automated pipeline, and poking at the Convertio site you linked to, their API suits my need 100%. So this thread has saved me a chunk of research I was about to need to do today on something wholly unrelated to FFXIV.

[DoomAngel]

Ok, so I created another character on a different Data Center, where the IP Address of the FFXIV server is different and I have had no disconnects yet. IP of new server is 195.82.50.49 Granted, the disconnects happened most frequently in Dungeons, Raids, etc, and my character is only lv 12 and hasn't opened any dungeons yet... so we'll see. But so far... few hours of letting the game sit idle, and some basic questing, so far no disconnects.

Moronica and Packetdancer, you guys are amazing! ...freaking geniuses... Even though the issue isn't resolved yet, I've gotten more help from you two in the last few days, than Sq Enix in months! They just keep closing my tickets stating it's a my PC, Router, packet loss, etc, and that they cant help me. At least now I feel like it's getting narrowed down! If there's any other data you need that's not too sensitive to post here, or any specific wireshark data that'll help, let me know.

[Moronica]

My gut feeling—and this is spitballing, at this point—is that if we're only seeing one connection, and we're seeing packets go out and not get acknowledged, if something along the way is dropping outgoing packets (or mangling them in some way such that the FFXIV servers consider them invalid packets and drop them) when they're being sent to that specific IP address that seems to be authoritative for DA's account... well, I do know that if FFXIV doesn't receive valid heartbeat messages when it expects them, it will shut down the connection.

The last screenshot only shows packets of one connection because Wireshark’s filter was “(ip.addr eq 192.168.1.146 and ip.addr eq 204.2.229.93) and (tcp.port eq 58899 and tcp.port eq 55026)”. Any additional TCP connection(s) from 192.168.1.146 to 204.2.229.93:55026 wouldn’t match these filter criteria.

There’s also no indication of dropped packets. The last two chunks of data sent by the client at 205.284423 and 205.296661 weren’t acknowledged by the server with an ACK most likely because it had already sent the FIN by the time it received those packets. That’s probably what caused the RST packets following the FIN, because received data can’t be acknowledged if the application already closed the socket and it thus unable to read that data.

If something along the way would drop or mangle TCP packets, the server would simply not receive those packets or ignore packets with an invalid checksum. That didn’t happen AFAICS, because the screenshot shows no retransmits and ACKs of the server for all data sent by the client (except for the last two chunks already mentioned).

Someone™ has to check the logs of the server to figure out what’s going on…

[DoomAngel]

The last screenshot only shows packets of one connection because Wireshark’s filter was “(ip.addr eq 192.168.1.146 and ip.addr eq 204.2.229.93) and (tcp.port eq 58899 and tcp.port eq 55026)”. Any additional TCP connection(s) from 192.168.1.146 to 204.2.229.93:55026 wouldn’t match these filter criteria.

There’s also no indication of dropped packets. The last two chunks of data sent by the client at 205.284423 and 205.296661 weren’t acknowledged by the server with an ACK most likely because it had already sent the FIN by the time it received those packets. That’s probably what caused the RST packets following the FIN, because received data can’t be acknowledged if the application already closed the socket and it thus unable to read that data.

If something along the way would drop or mangle TCP packets, the server would simply not receive those packets or ignore packets with an invalid checksum. That didn’t happen AFAICS, because the screenshot shows no retransmits and ACKs of the server for all data sent by the client (except for the last two chunks already mentioned).

Someone™ has to check the logs of the server to figure out what’s going on…

I just did the Wireshark conversation filter >TCP to capture all sessions between those two IP's. Those two ports are the only ones that were open for that connection.

Regarding dropped packets, while playing the game and capturing data, I've seen packet loss and the game hasn't disconnected at times, and have seen the game drop during times without packet loss. *shrug*

[Moronica]

I just did the Wireshark conversation filter >TCP to capture all sessions between those two IP's. Those two ports are the only ones that were open for that connection.

Hmm… your 3 screenshots labeled Disconnect #1 through Disconnect #3 in your original post all show two TCP connections disconnecting. Your screenshot in another thread also shows two connections.

I have no idea whether it’s normal for a FF XIV client to have two connections, or it’s only two connections for a while after you change zone/dungeon/instance/whatever.