Further Server and System Improvements, Part 3

Hello,

The below is a message from Naoki Yoshida about the current server status and upcoming plans created as a follow-up to his previous post, "Further Server and System Improvements, Part 2."

Quote:

Originally Posted by Naoki_Yoshida

FFXIV: ARR Producer and Director Yoshida here.

Ever since the extended maintenance that took place on 9/12 (JST), we've been able to monitor the weekend play trends, and I would like to inform you about the current status and plans for the future.

The Status of Weekend Operations
I apologize for the inconveniences caused by the emergency maintenance that took place on September 12 for the Worlds on the Japan data center. With that maintenance, we divided the Duty Finder servers into three groups, added new Worlds, added new instance servers, enhanced the login server/lobby server and Duty Finder matching server, and managed to meet the level of performance that we had initially planned. By doing this, the North American & European peak play times could be properly split up depending on each country's time zone and we've managed to operate with constantly high concurrency. The peak time for Japan is from 9 p.m. to midnight, and despite it being the peak time, compared to what we've seen until now, it's now possible to operate with an additional 40,000 concurrent users. Also, with the automatic logout feature, we've been able to properly maintain concurrency and expand the maximum number of players for each world through the maintenance, and we've just about removed all of the limitations on each world. From here on out we will be keeping an eye on the situation along with everyone's feedback and plan on further adjusting the automatic logout timer.

There have been a couple of occasions where servers went down and maintenance occurred, but these were due to different causes, and I would like to explain them below point by point.
Resumption of Digital Download and Package Sales
By improving our infrastructure, we've completed the enhancements for new Worlds and instance servers for new players. As there have been no issues with these new Worlds and the distribution of players, we'll be resuming sales for both the Windows and PS3 version for North America and Europe today, and then for Japan tomorrow (9/18). Also, shipments of the physical version will resume this weekend since they been limited in Japan. I deeply apologize to the customers that have been waiting to play and also to our product distribution partners.

We'll continue to keep an eye on server inflation conditions and the influx of new players and optimize the amount of digital and package version available.
Population on the New Worlds
There have been some users on the forum wondering if the number of players on the new Worlds is low. These newly added Worlds were created as a destination for world transfer users once the service is available, as well as all of the new players that will enter the game with the resumption of sales. Opening these Worlds before the resumption of sales was done to confirm their stability and also to spread out the level progression of those players who arrived to the World first. With sales resuming, please rest assured that their populations will continue to increase.
Overcrowded Worlds
I discussed the situation of overcrowded Worlds and how the Worlds have been set up in my previous post, and there were questions asking exactly which Worlds were overcrowded. In response to this question, I plan on explaining the below points and providing a bit of guidance before the start of the World Transfer Service:

What is the World Transfer Service?

What are the pros and cons of transferring to a different World?

My friends and I are on different Worlds. What should I do?

Which Worlds are overcrowded? What happens when a World is crowded?

I'd like to move to a World that's easy to log into, so what should I do?

We'll be preparing answers to these questions to coincide with the introduction of the World Transfer Service, so please hang in there just a little while longer.
Character Creation Limitations
Currently, by increasing the capacity on each World, implementing the automatic logout feature, and by making it easier for everyone to log into the game, regardless of the differences on each World, we've made it easier for each World to accept the creation of new characters. After resuming sales, there will be another rush of people creating characters, and there may be a possibility that we temporarily implement restrictions again; however, this will be further eased with the World Transfer Service. We plan on addressing requests from users to be able to create two or three character under the same account on the same World while keeping an eye on the above situation.
Causes of Server Crashes
We've confirmed the following points as the main reason for server crashes.

While playing the game, and certain actions are overlapped, server memory can become corrupted which leads to the area crashing.

Actions being performed by characters on instance servers are causing memory corruption which leads to the instance servers crashing.

For #1, while the frequency is low, it is continuing to occur because of character actions taking place in the field, and we are working quickly to investigate and fix this.

The instance servers mentioned in #2 are the content-specific Worlds that players are transferred to temporarily when matched with the Duty Finder. These instance servers do not belong to any of the main Worlds and operate solely for instances. We have a few dozen instance servers, but if an instance server goes down, then the content being played on that server will be wiped out and the players will be disconnected. The cause of this is also memory corruption and we are proceeding to investigate and fix this.

We are exerting all of our power to fix both causes of server crashes and we will continue to do the same in the future. I sincerely apologize for this, but we will be fixing this as we enhance our infrastructure.

Brief Maintenance on the Worlds on the Japan Data Center
This is something that will only be implemented for Worlds on the Japan data centers.
As I mentioned previously, compared to North America and Europe's core play times, Japan's is densely concentrated to a specific time.

I believe various factors contribute to this, such as the lifestyle of players and the availability of other forms of entertainment, but the peak time is mainly from 9 p.m. to midnight.

The number of users playing during this peak time on Friday, Saturday, and Sunday are even higher, and this period of three hours has significantly higher amounts of lag, extreme login and logout rushes, and higher amounts of content that is being loaded and cleared.

The matching servers for the Duty Finder will transfer players to an open instance server after receiving the request and matching players together so that they can play the content. The instance server will then load the content that was requested from the matching server, and once the content is cleared, the content will be wiped from the server.

Of all players that were logging in between 9/13 to 9/16, the average amount of them accessing an instance was nearly 24% (meaning that if there were 100 players, 24 of them were participating in instanced content).

The emergency maintenance that took place late last night was because this cycle of content loading and clearing was happening too quickly and the instance servers were crashing due to server memory gradually becoming fragmented and a decrease in the amount of memory that could be used. On top of this, the consequent log in rush from this placed heavy stress on the lobby server and caused even further issues. This was something that had to be done to make overall fixes, but it took place during last night's emergency maintenance.

As instance servers rapidly create and clear content, it's easy for memory to be depleted. Due to this, we've decided to adopt a "rolling restart" method during afternoons when the amount of players is low to avoid periodic long maintenance, and we will be performing maintenance on instance servers one by one. The biggest benefit of this method is that overall operation will not be affected because we will be performing maintenance sequentially. However, it’s difficult to use "rolling restarts" during times when there are a lot of people, and as there are a lot of people playing during weekend afternoons, this will be limited to weekdays.

As explained above, the usage rate of instance servers is extremely high and is especially heightened during weekend peak time, and we've determined that weekday rolling restarts alone would be difficult to maintain memory conditions, so we've decided to implement the following plan for the Japan data center.

Short maintenance for roughly 2 hours every Friday from 6 a.m. to 8 a.m.

Short maintenance for roughly 2 hours every Sunday from 6 a.m. to 8 a.m.

The hours between 6 a.m. to 8 a.m. (JST) has the lowest number of players and the effects of maintenance are lowest during this time frame. With the short maintenance occurring on Friday and Sunday mornings, it will reduce the need to have emergency maintenance like last night, and increase the stability and comfort of playing during weekend nights. We appreciate your understanding and cooperation.

We will be doing our best to complete the maintenance in as short of a time as possible, and we will be sure to keep you up to date.
Moving forward, the development and operations teams will be doing their best to provide comfortable play conditions, and as always, we appreciate your continued support.

By the way -
We've managed to expand the scale of operation and resume sales, and tomorrow we'll already be staying in Makuhari for Tokyo Game Show.

We'll be doing the following at TGS 2013:

Titan battle challenge (Everyday all day long: Defeat Titan and earn an "I BEAT TITAN" T-shirt.)

Letter from the Producer LIVE in Makuhari Part 1 (9/21 12:00 JST)

Letter from the Producer LIVE in Makuhari Part 2 (9/22 12:15 JST)

Through all of these events we will be letting you all know about the current and future plans for Eorzea (in other words, patch information…), so please be sure to check it out!

Of course I will be there mingling with everyone, and I'm really looking forward to meeting players, so if you come out to TGS 2013 please sure to stop by the Square Enix booth! (Check here for details.)