Server Backups and Happy Players

[Wibbly]

So a general opinion of me and my friends during the beginning of this long down time is that we're all disappointed and saddened by the week long (or longer) downtime of FFXI.

I feel immense pity and duty to assist Japan in it's time of crisis by donating to charities and I hope their situation improves rapidly, casualties are low, and things can become normal again soon.

But it does raise an excellent point in only one fashion. SE should make backup servers for their services!

One thing I very typically notice is when they do maintenance! They shut down services from early in the morning until into the evening and disrupt services to their players in all functions while performing this maintenance.

Secondly with the crisis in Japan unfolding, we may not know when the servers will come up again. Nuclear Power plants don't rebuild themselves in less than a week after all.

So I thought to myself how can this all be solved and came to a pretty simple solution. It would solve the problems of the maintenance down time and if disaster struck either area.

Square-Enix runs an office out of California (I assume) to handle the matters of their American customers, I also think they do this for European customers.

This might take some time and doing, but considering their income off of monthly subscriptions and the fact the world stops turning when a new FF game comes out...

Why not set up a second set of world servers for PoL and FFXI in the American or Europe based offices?

Now bare with me. Why would SE want to spend what must be thousands of dollars to set up a second server network? I believe it solves several problems, and feel free to argue the point politely in response to me, but I will argue my side of things.

First: Imagine that when server maintenance is happening for the Japanese servers (general maintenance, updates, patches) our PoL accounts, and thus FFXI, sign us out of that network and switch gears to locate and connect to the secondary network that as of that time remains un-updated, but still functioning during the time of the maintenance that otherwise shuts down an entire day of the week for FFXI service. After maintenance is done, the same thing can happen with disconnecting us from PoL and switching back to the main server. This solves the problem of maintenance downtime taking away twelve hours of play.

Second: Obviously, if the main server is going to be out of commission (general hardware failure, maintenance shut down, kamikaze watermelon explodes on the motherboard etc) then primary service can be switched to the secondary server located in California or Europe and service can continue until the problem can be solved. this solves the problems facing XI players while Japan deals with it's recovery.

Optionally: We all have times in which lag can become terribly detrimental or make the game seem to play oddly. Adding in the fact constantly communicating with a server (for most players) half way across the world creates communication issues. Under the assumption that they wouldn't use the secondary server until necessary of them to continue service, and instead kept it running and communicating with the primary server, wouldn't that help the game in a global manner continue to run even as smoothly as it usually does?


Please, speak your opinion on this. If it becomes a feasible idea through our discussion we may see a vastly improved service to PoL and FFXI in the near future.

[Wibbly]

No ideas? >_>

[JagerForrester]

I'm sure they're getting on that as we speak. I don't think they were aware of the problem until it occurred. And as a smart business, they are most likely discussing ways to not let shutting the servers down become the best option.

If setting up servers in another location is what they are discussing, they are going to have to issue contracts and have the finances to do it. And before this hits wagon: Saying that SE does have the money doesn't make up they have a budget to think about too. This budget includes the funding of new products that are in development. It's not very cost-efficient to put every cent of into a product and end up with a low profit margin.

[RAIST]

While in basic theory it's a good idea, and how most corporations set up their mission critical servers, it's not practical. I put another post in some other threads about this already trying to layout the scope of it all. Basically, your client is continuously connected to multiple servers at once. When you zone, you will tap maybe 4 servers during the process. The bulk of these servers are all located in one location--in Japan. It had to be done this way simply because logisitcally it could not be done and have it be a truly global environment. If they had the game servers regionalized to different continents, then only those continents would be allowed to play on them (ie, no JP and NA players on the same servers--it would all be NA people only, and JP would have a JP only server).

The communications of the game are not designed to compensate for the lag that would be there if you tried to synchronize 30+ servers in JP with 30+ servers in the US, then another 30+ servers in Europe. And yes, it would be 30+ servers in each location if they didn't split it up--because each site would have to house a complete set of the data, as it would be a global data set. .They would HAVE to restructure the player base to put players in their own region, and isolate them from the players in another region (think Blizzard and the Diabolo series--US West players could not play with US East players...no way, no how...they were completely segregated from each other). You think there is anough of a cry-fest when 16 servers are merged into 8 now? Try splitting everyone up and rehoming them to their own continent, and not allowing them to play with eachother ever again. This would HAVE to be done simply because it would be not only impractical, but not cost effective to run a dedicated fiber-optic network between all these sites to secure the bandwidth, security (hello, credit card information), and reliability of all that data being synchronized world wide in 3 or more locations.

Best case scenario, they can have the servers replicated locally. This would allow them to take one server offline to do a full backup for bare-metal restore scenario, and then bring it back online and resynch the data. This backup could be housed offsite for security. I get the feeling this is sort of how they do it already. When I got hacked years ago, my toon was restored to a state just a few days before I got hacked, which would be in line with a backup of a replicated data set.

While this doesn't provide for quick recovery if they have a catastrophic loss, or (as in this case) are no longer able to run from the current location otherwise--it does reduce the time required to rebuild at an alternate location if they have to. The data is already offsite, in a shutdown state as if the sever had a clean shutdown. Though it may be a day to 72 hours old, it is a clean backup that can be used to bring everything back online elsewhere once the core environment is setup. So..if they have an offsite datacenter capable of handling the load, they can restore and come back up once DNS changes are propogated to point everyone to the new servers. Alternatively (like now for instance) after the game is taken down like it is now, they could do a back up, and move that backup to the new datacenter and restore and bring it up. Granted.. it's not fast, but it is doable.

Raist

[Wibbly]

Wonderful insight into the issue at hand!

I didn't think of it as splitting the world apart into so many piece and separating them permanently, but then again, I know about as much about intercontinental server communication as I do about... Well... Intercontinental server communication.

The primary reason I thought this up (like a lot of people who are spamming the forums going "wen i get XI bak?") is because I'm earnestly surprised they -didn't- have some manner of back up ready.

Secondarily because we may all be hopeful of the situation in japan, but you don't rebuild a nuclear plant in less than a week, much less two at a time. Likely we're in for the long haul on this one.

But going back to the servers, logically, how many server would it take to set up in order to restore services? I understand they have probably four mainframes for the core of the system and an additional server for roughly 4 regions of the game a piece. Maybe it wouldn't even be far fetched that someone who enjoys the game would donate a temporary super server for their uses. You never know until you poke around, right?

[Cenz]

I like many others have a very uneducated perspective on the current situation. Outside of all the crying that i have read about and self centered posts this is the first one that Ive seen that made sense. I may be wrong but i am pretty sure its solely a power issue and until the power is stabilized it wouldn't matter where in the country the servers were. The government itself requested the companies to conserve. I highly doubt SE is happy about not getting the months subscription and i believe they are gonna do everything they possibly can to get the game running asap. Does anyone know for a fact if the actual facility suffered any damage?

[Mirage]

There are server backups.

That's not the same as redundant servers in several physical locations, though. How many online games do you think have this?

[RAIST]

well, it's roughly 8 different servers I've tracked my system hittting so far when having issues with connectivity in the past. but that was only when troubleshooting for a couple of zones. Granted, some of those hits were to one or more of their 4 DNS servers as well. I think I have 6 of their servers whitelisted in my firewall atm, as they were actively pinging my system and getting blocked (cause for some of the connectivity issues). There were 32 active server worlds (plus one test server) prior to the last merge, so they currently have 25 game worlds in production now. So, with their own in house servers for payroll, internal messaging, development and such, regional Lobby servers, Updates server, Accounting, DNS, 25 game worlds, and the POL messaging server itself...I would hazard a guess that they have at LEAST 40+ servers that have to be setup somewhere for this division to fully function. It is not something you just pick up and move somewhere. It took us a full weekend just to physically move a 5 server office across town and wire everything up and get DNS propogation to flip over to the new ISP (they switched from a T1 to Cable to save money with the move--perfect time for the flip as services were down for 48 hours anyway).

So yeah...relocating would be a big project, and setting up an offsite replication forest would be a big budget problem, provided they could get the bandwidth required to do it properly.

Raist

[Rezeak]

Really not worth it the only time that a second set of servers would be needed is after a massive disaster like now and u rly can't except a company to rent a second space and train staff to maintian them for a situation that really is once in a 40-100 years situation where a disaster cripples the entire country.

As for it fixing
lag :nope servers would still need to commucate between each other + lag is more a isp issue anyway.
backup: As the reason above it's just better to have backup on site
To stop Maintance : if they could do this they would of done so already at the same site they have the normal servers there is definty more too it than i know anyway.

[kitykat]

I think the big picture is being missed somehow..the server(s) are fine. Japan does not the energy to run the game. While trying to figure out how to make the game run smoother is a commendable topic, it can not address the problem at hand. Sadly... (and here is the hardest truth of the matter) this game is dieing and does not have much time. This is obviously known my the developers, who continue to downsize servers and upgrade gaming content in the hopes of extending the games life further. With this said, knowing the game life is ending, it does not ring true that SE would spend time nor money in updating servers or server locations.

[Mirage]

I agree. I doubt the game is going to get a sudden surge of new subscribers, while I also think the game is good for another 5 years or more before all servers are shut down. I do believe that they'll keep it up even with just a single game world all the way until they're directly losing money each month on doing it.

[Rakam]

FFXI and FFIV are the only major game everyone in the world play on same server. Im sure other game have server in different location cause there server are dedicate to a region only. They already have a back up of all server and the game, but think of this for a second: having 3 servers in 3 different area for the same world and scynchro em in real time? or having 3 different locations ready at any time when this is the first time in almost 9 years that SE shut down FFXI. This would made explode the base cost for SE. Also this is the the bigest tsunami in about 150 years in Japan. I personnaly dont think we will have FFXI and FFIV back for at less 3 weeks and probably more. Im glad SE decide to shut down server to save our data from a power shut down and make their part in helping all there country to get back on the track faster.

[Rakam]

I agree. I doubt the game is going to get a sudden surge of new subscribers, while I also think the game is good for another 5 years or more before all servers are shut down. I do believe that they'll keep it up even with just a single game world all the way until they're directly losing money each month on doing it.

FFXI will never die.

[Mirage]

let's hope so

[Wibbly]

I think the big picture is being missed somehow..the server(s) are fine. Japan does not the energy to run the game. While trying to figure out how to make the game run smoother is a commendable topic, it can not address the problem at hand. Sadly... (and here is the hardest truth of the matter) this game is dieing and does not have much time. This is obviously known my the developers, who continue to downsize servers and upgrade gaming content in the hopes of extending the games life further. With this said, knowing the game life is ending, it does not ring true that SE would spend time nor money in updating servers or server locations.

SHAME ON YOU.

"this game is dieing and does not have much time."


Look carefully at what you're saying and realize the only reason this game would be dying is because people express it to be dying, or giving up their support for it.

I don't see World of Warcraft dying. It's a very old game now.

I don't see Ragnarok Online dying. It's at least twice as old as XI.

The reason for this is expansion of content.

What I see is a deep community of players who know what they like, and what they want to play. the only reason a game like this would be dying is because people kill it.

I know times are tough and it's difficult to look on the bright side of things, but SE is going to look to us for support of their games, and say that it's still worth running if we still pay, and still think it's worth playing.

Corny Speech: HAVE HOPE MY FELLOW VANADITES. The world is not ending! Give your faith and strength and it shall be returned to you!

[Mirage]

Ragnarok online isn't anywhere near twice as old as FF11. WoW isn't as old as FF11, but it doesn't matter because that game is in an entirely different situation. Blizzard is making so much money on it that they can invest a *lot* more money into the continued development of it, and the fact that it is the biggest MMO in the world makes a lot of people wnat to try it out rather than other MMOs.

[Kingofgeeks]

Having redundant servers would def minimize, if not eliminiate the lack of availablilty. The only issue is cost. FFXIV is free to play at the moment which means SE is losing A LOT of money every month. FFXI is not as big as it used to be, so funding for the game is as low as the revenue it produces. So while there's a lot of things that can be improved when it comes to game content and services, i think the real issue is just the serious lack of money required to make all this possible.

(i'm pretty sure someone else already said this, but oh well! i don't feel like reading every post in this thread :-P)

[Wibbly]

Thank you all for your insight. <3

[Shadowkitty_Carby]

I'd like to add that from what I understand there is actually enough power to run FFXI servers, but they are respecting the disaster going on and not using vital power that is needed to run the cities and save lives. The death toll is now 13,000 and still going. As a NA FFXI Player, I, among other addicts, usually enjoy countless hours of fun and drama. And I'm already having withdrawal as I'm sure many others are experiencing lol. But we have to look at the big picture, there is not enough money nor is it even practical to develop servers at other countries. Most likely such an action would result in communication problems and even more maintenance, and that is assuming such an idea is even feasible which I feel really isn't with the resources SE has atm.

And truthfully it is not even important right now, we need to take our eyes off FF and look towards the tragedy and see how we can help. It is not normal for the servers to be down like this, but it is a unique situation and sometimes we have to realize that there cannot be preparation for something of this unexpected magnitude.

It is surprising to see how little charities are actually building to help Japan compared to other recent world disasters. Use all the free time that the servers are down to help out! :) And I actually don't think it will be that long until servers are back up again. The main issue is saving lives, once things stabilize it'll be safer to use power on more frivolous things like FF. We'll just have to wait and see!

[kitykat]

I'd like to add that from what I understand there is actually enough power to run FFXI servers, but they are respecting the disaster going on and not using vital power that is needed to run the cities and save lives. The death toll is now 13,000 and still going. As a NA FFXI Player, I, among other addicts, usually enjoy countless hours of fun and drama. And I'm already having withdrawal as I'm sure many others are experiencing lol. But we have to look at the big picture, there is not enough money nor is it even practical to develop servers at other countries. Most likely such an action would result in communication problems and even more maintenance, and that is assuming such an idea is even feasible which I feel really isn't with the resources SE has atm.

And truthfully it is not even important right now, we need to take our eyes off FF and look towards the tragedy and see how we can help. It is not normal for the servers to be down like this, but it is a unique situation and sometimes we have to realize that there cannot be preparation for something of this unexpected magnitude.

It is surprising to see how little charities are actually building to help Japan compared to other recent world disasters. Use all the free time that the servers are down to help out! :) And I actually don't think it will be that long until servers are back up again. The main issue is saving lives, once things stabilize it'll be safer to use power on more frivolous things like FF. We'll just have to wait and see!

Ok here's the part where i say something totally unpopular and ppl throw things at me.. According to one news casters political view, Japan hasn't received a lot of financial support because its not needed (according to Japan). They have requested large heavy construction equipment (back hoes etc) and fresh food and water, and experts to help with reactors. Japan is wealthy and their government (military) takes great pride in providing for themselves. Plz don't think im not in support of helping them, I am. But i don't think cash assistance is going to help them as much as actual in person experienced individuals to provide there ability's and knowledge to this particular natural catastrophe. (such as the swift water rescuers that deployed immediately out of V.A.) As well as cadaver canine unites and how the U.S navy set up shop off the coast and provided off shore electricity to small hospitals etc..

And just for the record, i HOPE the game nvr ends.. i love playing and will continue right up till the end!! Honestly.. i miss you guys im addicted to the chat and company as much as i am to crafting =P

[Flunklesnarkin]

I think ffxi as a whole is a good experiment

It shows why regional servers work better than "everybody plays in same server" type system

regional servers don't eliminate rmt but they do make it a hell of a lot harder for rmt farms in china/korea/india/etc to function

they don't knock out the game for players from another region so you wont feel guilty about rushing another country after a disaster like this lol

and people don't have to worry about all the jp onry bull.. or american's harassing the french or w/e else might pop up

I can understand the jp onry stuff.. people want to play a game.. not struggle to understand other's language.. regional servers would eliminate that.

[Celedh]

Personally, this reminds me of the issue that was brought up recently.

My brother's house burned down, taking away with it most of his things, including the computer and hard drives he had. Thing is, on those there was some pictures and videos of my niece. A portable, external hard drive is a great idea, but if the only backup of it is inside the same house, it's not useful. Having backups is great, but they need to be kept somewhere else.

So I recently got myself a USB key of 16gb, just for the purpose of storing my personal data outside of the house. I keep it at work. I figure that if both my apartment and work were to go down in fire in the same day, there's some higher power at work that wanted me to lose that data no matter what <_<