Server Backups and Happy Players

[Wibbly]

So a general opinion of me and my friends during the beginning of this long down time is that we're all disappointed and saddened by the week long (or longer) downtime of FFXI.

I feel immense pity and duty to assist Japan in it's time of crisis by donating to charities and I hope their situation improves rapidly, casualties are low, and things can become normal again soon.

But it does raise an excellent point in only one fashion. SE should make backup servers for their services!

One thing I very typically notice is when they do maintenance! They shut down services from early in the morning until into the evening and disrupt services to their players in all functions while performing this maintenance.

Secondly with the crisis in Japan unfolding, we may not know when the servers will come up again. Nuclear Power plants don't rebuild themselves in less than a week after all.

So I thought to myself how can this all be solved and came to a pretty simple solution. It would solve the problems of the maintenance down time and if disaster struck either area.

Square-Enix runs an office out of California (I assume) to handle the matters of their American customers, I also think they do this for European customers.

This might take some time and doing, but considering their income off of monthly subscriptions and the fact the world stops turning when a new FF game comes out...

Why not set up a second set of world servers for PoL and FFXI in the American or Europe based offices?

Now bare with me. Why would SE want to spend what must be thousands of dollars to set up a second server network? I believe it solves several problems, and feel free to argue the point politely in response to me, but I will argue my side of things.

First: Imagine that when server maintenance is happening for the Japanese servers (general maintenance, updates, patches) our PoL accounts, and thus FFXI, sign us out of that network and switch gears to locate and connect to the secondary network that as of that time remains un-updated, but still functioning during the time of the maintenance that otherwise shuts down an entire day of the week for FFXI service. After maintenance is done, the same thing can happen with disconnecting us from PoL and switching back to the main server. This solves the problem of maintenance downtime taking away twelve hours of play.

Second: Obviously, if the main server is going to be out of commission (general hardware failure, maintenance shut down, kamikaze watermelon explodes on the motherboard etc) then primary service can be switched to the secondary server located in California or Europe and service can continue until the problem can be solved. this solves the problems facing XI players while Japan deals with it's recovery.

Optionally: We all have times in which lag can become terribly detrimental or make the game seem to play oddly. Adding in the fact constantly communicating with a server (for most players) half way across the world creates communication issues. Under the assumption that they wouldn't use the secondary server until necessary of them to continue service, and instead kept it running and communicating with the primary server, wouldn't that help the game in a global manner continue to run even as smoothly as it usually does?


Please, speak your opinion on this. If it becomes a feasible idea through our discussion we may see a vastly improved service to PoL and FFXI in the near future.

[Wibbly]

No ideas? >_>

[JagerForrester]

I'm sure they're getting on that as we speak. I don't think they were aware of the problem until it occurred. And as a smart business, they are most likely discussing ways to not let shutting the servers down become the best option.

If setting up servers in another location is what they are discussing, they are going to have to issue contracts and have the finances to do it. And before this hits wagon: Saying that SE does have the money doesn't make up they have a budget to think about too. This budget includes the funding of new products that are in development. It's not very cost-efficient to put every cent of into a product and end up with a low profit margin.

[RAIST]

While in basic theory it's a good idea, and how most corporations set up their mission critical servers, it's not practical. I put another post in some other threads about this already trying to layout the scope of it all. Basically, your client is continuously connected to multiple servers at once. When you zone, you will tap maybe 4 servers during the process. The bulk of these servers are all located in one location--in Japan. It had to be done this way simply because logisitcally it could not be done and have it be a truly global environment. If they had the game servers regionalized to different continents, then only those continents would be allowed to play on them (ie, no JP and NA players on the same servers--it would all be NA people only, and JP would have a JP only server).

The communications of the game are not designed to compensate for the lag that would be there if you tried to synchronize 30+ servers in JP with 30+ servers in the US, then another 30+ servers in Europe. And yes, it would be 30+ servers in each location if they didn't split it up--because each site would have to house a complete set of the data, as it would be a global data set. .They would HAVE to restructure the player base to put players in their own region, and isolate them from the players in another region (think Blizzard and the Diabolo series--US West players could not play with US East players...no way, no how...they were completely segregated from each other). You think there is anough of a cry-fest when 16 servers are merged into 8 now? Try splitting everyone up and rehoming them to their own continent, and not allowing them to play with eachother ever again. This would HAVE to be done simply because it would be not only impractical, but not cost effective to run a dedicated fiber-optic network between all these sites to secure the bandwidth, security (hello, credit card information), and reliability of all that data being synchronized world wide in 3 or more locations.

Best case scenario, they can have the servers replicated locally. This would allow them to take one server offline to do a full backup for bare-metal restore scenario, and then bring it back online and resynch the data. This backup could be housed offsite for security. I get the feeling this is sort of how they do it already. When I got hacked years ago, my toon was restored to a state just a few days before I got hacked, which would be in line with a backup of a replicated data set.

While this doesn't provide for quick recovery if they have a catastrophic loss, or (as in this case) are no longer able to run from the current location otherwise--it does reduce the time required to rebuild at an alternate location if they have to. The data is already offsite, in a shutdown state as if the sever had a clean shutdown. Though it may be a day to 72 hours old, it is a clean backup that can be used to bring everything back online elsewhere once the core environment is setup. So..if they have an offsite datacenter capable of handling the load, they can restore and come back up once DNS changes are propogated to point everyone to the new servers. Alternatively (like now for instance) after the game is taken down like it is now, they could do a back up, and move that backup to the new datacenter and restore and bring it up. Granted.. it's not fast, but it is doable.

Raist

[Wibbly]

Wonderful insight into the issue at hand!

I didn't think of it as splitting the world apart into so many piece and separating them permanently, but then again, I know about as much about intercontinental server communication as I do about... Well... Intercontinental server communication.

The primary reason I thought this up (like a lot of people who are spamming the forums going "wen i get XI bak?") is because I'm earnestly surprised they -didn't- have some manner of back up ready.

Secondarily because we may all be hopeful of the situation in japan, but you don't rebuild a nuclear plant in less than a week, much less two at a time. Likely we're in for the long haul on this one.

But going back to the servers, logically, how many server would it take to set up in order to restore services? I understand they have probably four mainframes for the core of the system and an additional server for roughly 4 regions of the game a piece. Maybe it wouldn't even be far fetched that someone who enjoys the game would donate a temporary super server for their uses. You never know until you poke around, right?

[Cenz]

I like many others have a very uneducated perspective on the current situation. Outside of all the crying that i have read about and self centered posts this is the first one that Ive seen that made sense. I may be wrong but i am pretty sure its solely a power issue and until the power is stabilized it wouldn't matter where in the country the servers were. The government itself requested the companies to conserve. I highly doubt SE is happy about not getting the months subscription and i believe they are gonna do everything they possibly can to get the game running asap. Does anyone know for a fact if the actual facility suffered any damage?

[Mirage]

There are server backups.

That's not the same as redundant servers in several physical locations, though. How many online games do you think have this?

[RAIST]

well, it's roughly 8 different servers I've tracked my system hittting so far when having issues with connectivity in the past. but that was only when troubleshooting for a couple of zones. Granted, some of those hits were to one or more of their 4 DNS servers as well. I think I have 6 of their servers whitelisted in my firewall atm, as they were actively pinging my system and getting blocked (cause for some of the connectivity issues). There were 32 active server worlds (plus one test server) prior to the last merge, so they currently have 25 game worlds in production now. So, with their own in house servers for payroll, internal messaging, development and such, regional Lobby servers, Updates server, Accounting, DNS, 25 game worlds, and the POL messaging server itself...I would hazard a guess that they have at LEAST 40+ servers that have to be setup somewhere for this division to fully function. It is not something you just pick up and move somewhere. It took us a full weekend just to physically move a 5 server office across town and wire everything up and get DNS propogation to flip over to the new ISP (they switched from a T1 to Cable to save money with the move--perfect time for the flip as services were down for 48 hours anyway).

So yeah...relocating would be a big project, and setting up an offsite replication forest would be a big budget problem, provided they could get the bandwidth required to do it properly.

Raist

[Rezeak]

Really not worth it the only time that a second set of servers would be needed is after a massive disaster like now and u rly can't except a company to rent a second space and train staff to maintian them for a situation that really is once in a 40-100 years situation where a disaster cripples the entire country.

As for it fixing
lag :nope servers would still need to commucate between each other + lag is more a isp issue anyway.
backup: As the reason above it's just better to have backup on site
To stop Maintance : if they could do this they would of done so already at the same site they have the normal servers there is definty more too it than i know anyway.

[kitykat]

I think the big picture is being missed somehow..the server(s) are fine. Japan does not the energy to run the game. While trying to figure out how to make the game run smoother is a commendable topic, it can not address the problem at hand. Sadly... (and here is the hardest truth of the matter) this game is dieing and does not have much time. This is obviously known my the developers, who continue to downsize servers and upgrade gaming content in the hopes of extending the games life further. With this said, knowing the game life is ending, it does not ring true that SE would spend time nor money in updating servers or server locations.