World Login Errors and Resolutions

World Login Errors and Resolutions
Hello, this is Naoki Yoshida, Producer & Director of FINAL FANTASY XIV.

Thank you all for participating in the Early Access and official launch of Endwalker.

Since the start of Early Access, we have been seeing an unprecedented number of active players, the highest ever seen since the launch of FFXIV, as well as record numbers of simultaneous logins, but along with this, the number of connections has far surpassed the scale of our services which has resulted in long wait times when attempting to log in.
　
Even during this congestion, we’ve received words of support and encouragement, for which we’re truly grateful. Thank you very much. On the other hand, we sincerely apologize for the long waiting times and the resulting frustration experienced by players attempting to log in.

I’d like to share some information regarding the ongoing congestion, the reasons behind errors that have been appearing while waiting in queues, and the current status of our fixes. For information on error codes not listed below, please see my previous announcement on December 5 (NA: https://sqex.to/QDFMi EU: https://sqex.to/QA5Cq).

■ Lobby server maintenance to address errors

Allow me to share some information regarding specific errors. We are planning for a full maintenance on the lobby servers at the earliest timing possible, in order to resolve these errors. Although the maintenance time isn’t fully solidified, it is estimated to take about 30 minutes. This will not affect any players who are already logged into the game, but during these 30 minutes, players will not be able to make new login attempts, and the login queue will be unavailable.

We deeply apologize for this but ask for your cooperation as we address these server errors. The exact maintenance times will be shared in an upcoming announcement.

■ All Worlds Maintenance

In the early morning of Wednesday, December 8, we will be conducting maintenance across all Worlds to address server errors aside from those mentioned above, as well as a few issues that can be resolved on the server side. This requires all servers to go offline and is estimated to take about four hours to complete. Details on maintenance timing will be announced in the future. I’d like to ask for your understanding and cooperation regarding this matter as well.

Below are some details regarding errors that have been occurring as a result of congestion.

■ Disconnects by Error 4004 right before logging in

Currently, I believe the most frustrating cases are when you’ve been waiting in the queue and get disconnected by an error when your turn comes around, forcing you join the queue again from the end. These disconnects are often happening as a result of Error 4004 and 3001, and here I’d like to provide the details and statuses of countermeasures against Error 4004.

・ How the Process Works

To prevent servers from going down as a result of mass logins or overloading, the process of logging into a World that has reached the login cap involves waiting until the World can admit 100 players, then admitting the first 100 players in the login queue, a process that is being repeated constantly.

・ The Issue

We have confirmed an issue during the process of admitting these 100 players, which can occur if certain conditions coincide while the servers are experiencing extremely high loads, which will cause up to 25% of these players to receive an error. This issue is not guaranteed to occur even when the servers are experiencing extremely high loads, and the number of players who experience the issue varies. However, considering login queues can take up to several hours and this error can cause players to have to return to the back of the queue, we currently recognize this as the most frustrating issue.

* Added: Dec. 7, 06:20 a.m. PST
This fix has been implemented, but I would like to clarify some things in relation to the on-screen display. Worlds that reach the login limit will be checked for 100 available positions, and 100 new logins will be executed sequentially. However, since the maximum number of characters that can log in to a World in the space of one minute sequentially is limited to 75 in order to reduce the load, the number of people in the queue will often appear to decrease by 75 in the on-screen display. (This is how it will appear when there is a clean overlap in the update timing of the displayed number of players waiting.) This has been the way the system operates from before, and even if the processing is carried out properly with the fix this time, the maximum count will decrease by 75 each in the dialogue, making it look as if the aforementioned fix has failed. Due to the coincidental overlap in the numbers, I apologize for any confusion my choice of words may cause.

・ Resolving the Issue

We have been able to locate the cause of this error, which occurs due to extremely high server loads, and are currently fixing the source code for the lobby servers. Although the fix itself is finished, the process of checking whether the fix has actually resolved this issue requires time as it is an extremely complex process. In addition, due to the complexity of the conditions of occurrence itself, it is possible that we have not yet addressed more specific conditions, but we will implement the fixes and continue to monitor the situation. As soon as our preparations are ready, we will carry out the work through the aforementioned lobby server maintenance. We ask that you wait a little longer until the fix is made.

■ Disconnects by Error 3001 right before logging in

I’d also like to provide the details and countermeasures for Error 3001.

・ Error 3001 Cause #1

A login cap is in place on every World in order to prevent them from going down due to high loads. Despite this, a number of extremely overloaded servers have allowed players to log in beyond this cap, resulting in the number of permitted lobby server connections to fail and cause Error 3001 and disconnect players in the queue.

・ Resolution for Cause #1

We currently have a general idea of what is causing this issue with extremely overloaded Worlds, and a fix is underway. However, these are areas where complex processing is carried out and careful fixing is required for this process. As such, fixing/confirming/implementation into the lobby server will require time. This will be implemented in the maintenance scheduled for December 8, and we are putting our full effort into it, but haven’t been able to determine the full details. Further details regarding this will be announced at a later timing as well.

・ Error 3001 Cause #2

This is an issue that is occurring only on the Tonberry World on the Elemental data center. The Tonberry World has the largest number of login requests of all the game Worlds, and players are helping to mitigate this issue by using the World Visit System and logging into other Worlds. However, due to the way chat connections work when using the World Visit System, we cannot completely cut off a player’s connection with their character’s Home World, causing the total number of active connections to become enormous, and we have confirmed that as a result, a situation similar to Cause #1 above can occur.

・ Resolution for Cause #2

Although we are aware of the relative risk of the Tonberry World going down, we’ve increased the number of network connections to its absolute maximum. Your patience would be greatly appreciated as we monitor the situation based on this tentative fix for the time being.

■ Error 2002

Now I would like to explain about the details related to Error 2002 and our countermeasures.

・ Details of the Process

In order to prevent a major login server outage, FFXIV has a limit of "no more than 17,000 total players waiting to log in per logical data center". This cap is set uniformly regardless of how many Worlds exist in a logical data center.

・ The Issue

If the total number of players waiting in the logical data center exceeds 17,000, Error 2002 will be displayed and you will not be able to queue for login.

・ The Resolution for this Issue

We are now ready to deploy the backup development servers to the public lobby servers, which will improve the capacity management of the lobby server and raise the maximum number of players in the queue to around 21,000. Since this will not require any maintenance, we will err on the side of caution and proceed with the deployment outside of peak hours. It is scheduled to be completed late in the night of December 7, Japan time.

■ Measures Towards a Fundamental Resolution for Congestion

When it comes to fundamental measures to prevent this type of large-scale congestion—and this is something that was brought up in the Lodestone announcement (NA: https://sqex.to/4IkYs EU: https://sqex.to/NoWro) we made ahead of early access—the most effective method would be a “large-scale addition of new Worlds in all regions.”

On the one hand, following the explosive rise in players coming to FFXIV beginning in spring 2021, we spent over half a year scrambling to address the need to add new Worlds. However, due to policies surrounding COVID-19, which have had an effect on the entire world, even now we have been unable to procure the required hardware for adding new Worlds.

■ The Global Semiconductor Shortage (Supply Chain Issues)

When it comes to adding new Worlds, we need tens of “server machines” for every World that we add. Server machines are high-performance computers, which utilize numerous semiconductors. However, due to the COVID-19 countermeasures currently in place, many factories across the globe which produce semiconductors have halted production, or have faced labor shortages. This has ultimately led to a decrease in the number of semiconductors being produced, and has resulted in a worldwide semiconductor shortage.

Furthermore, this does not only pertain to the production of semiconductors, but from a logistics standpoint as well, COVID-19 countermeasures continue to explosively push up demand and create this turbulent situation. We have made considerable investment—even more so than usual—to secure the required hardware, but even so, a long lead time will be needed to prepare the server hardware.

If we try to look to when this semiconductor shortage may be solved, unfortunately given the state of things, we aren’t able to estimate when we might reach a resolution. The FFXIV Server and Infrastructure teams accelerated work to implement the additional high-performance machines we had secured, as well as optimizing the server source code and increasing the number of maximum logins per World. I promise that we will continue to put in this type of effort.

Given the current situation, we anticipate that it may take several months or more to add new Worlds. I want to express my sincerest apologies to all of our players for the inconvenience this is causing. I appreciate your continued understanding and support regarding this matter.