Thread: Regarding Data Center Technical Difficulties Occurring After Patch 3.5 (Jan. 27)

We would like to report the current situation regarding the issues that are occurring with the Mana, Aether, and Chaos Data Centers since Patch 3.5.

We have confirmed that the issue is with the system which controls queuing in the Duty Finder behaving abnormally under the stress of queue congestion. Using logs implemented for investigative purposes in Patch 3.51, we are reviewing data of the issues that occur, and the search for a solution is our highest priority.

Because the system which controls the Duty Finder queue is separate for each data center, an individual data center maintenance is required to perform recovery whenever this issue occurs. We apologize that this issue has been ongoing for such an extended time.

Currently, the dev team is testing fixes, and because they must carefully verify that these fixes are working properly, it looks to be several days until a solution is implemented.

In order that weekend playtime is affected as minimally as possible, data center surveillance by the dev/operator teams will be strengthened, but we apologize that stability cannot be reached by this weekend.

We sincerely apologize for the extreme inconvenience caused to our players.

We would like to report the current situation regarding the issues that are occurring with the Mana, Aether, and Chaos Data Centers since Patch 3.5.

Previously, we relayed that the issue is with the system which controls queuing in the Duty Finder behaving abnormally under the stress of queue congestion. Upon continued investigation, it has newly come to light that, under certain very rare circumstances queue management ceases entirely, causing the server to crash.

However, although we have a general idea of what is causing the issue, it is difficult to pinpoint completely in a dev environment. Therefore, during the Patch 3.51 HotFixes planned on Feb. 1, 2017 (PST) / Feb. 2, 2017 (GMT), we plan to implement a further system and investigative log in order to continue careful observation and implementation of a solution.

For this reason, more time will be needed until the issue is fundamentally resolved. In order to minimize the effect on our players, with the Patch 3.51 HotFixes we will be implementing a temporary solution, updating how queue management operates by making it so that when the queuing system becomes stuck, the problematic character or party's (single request) queue is canceled, thus removing the cause of the queue being stuck and preventing the server from crashing.

Should your queue become canceled, we ask that you please try queuing again. Although chances of a queue cancellation occurring due to this update are extremely small, we nevertheless apologize for the inconvenience.

We will continue to address this issue with the highest priority, and we ask for your continued patience.

We sincerely apologize for the extreme inconvenience caused to our players.

[Update: Feb. 8, 2017 12:42 a.m. (PST) / 8:42 (GMT)]

We have been monitoring the situation, and have found no cases of queues being canceled at this time. In addition, due to the investigative log and system we implemented with the Patch 3.51 Hotfix, we have been able to confirm the situation and general cause, and the issue is planned to be addressed in the next Patch 3.55.

[Update: Mar. 1, 2017 3:00 a.m. (PST) / 11:00 (GMT)]

This issue has been officially addressed with the Patch 3.55a maintenance which took place on Feb. 27, 2017 6:00 p.m. to Feb. 28, 2017 12:30 a.m. (PST) / Feb. 28, 2017 2:00 to 8:30 (GMT).
We apologize for the inconvenience this may have caused.