Servers in Danger of Crashing RIGHT NOW!

[Archulak]

Which means pretty much the opposite..

It means that someone is going to be super wise and say "Its inconvenient to move items across alts therefore you can completely discount it". To which my counterpoint in that same sentence is you can't say that no one will do it because I know people who do and it must be given consideration.

So yes you did miss understand hope that clears that up.

Also if you're building a system you don't say "Oh this elevator can carry about 500 lbs but i don't have to have safeguards in place, in case someone goes over that, cause most people won't", you need to account for possibilities that exceed your expectation or you can come across massive failures and at that point your only option is to say "Hmm didn't think anyone would do that."

[Twilite]

I guess most of you don't understand how databases work.

[Raist]

The point of their statement was about transmitting the data. And yes, it isn't 400k users on one world, but we knew over a year ago that we had crossed the 5000 simultaneous login threshold per world. We also know that they increased capacity past 7000 shortly after that, with a goal to eventually reach as high as 15,000. That is simultaneous connections per world.

Even just mathing it out at the point of 7000 capacity, 24 slots at your simplified example of 32bits per ID would increase the query count potential to an additional 168,000 queries--and requiring north of 5Mbit/second bandwidth to support it (7000*32*24=5,376,000). That is per world and it doesn't account for any handshaking, overhead, etc...that is just the expected data stream for transferring the raw data in one direction. Keep in mind also that it has to flow both ways (client to server, and server to client), so it could easily become 11Mbits combined just for one world to move only the raw item ID back and forth.

Now, add in the extra information that may be bound to the ID as well (materia, Spirit Bond and Repair levels, etc.)---it wouldn't be too far a stretch to see the potential to have a spike in traffic that could consume up to the entire bandwidth of a single 10 Gigabit card across the entire playerbase. Remember, we have over 1million subs--have to prepare for the possibility that concurrent connections may ramp up at a moment's notice. Just using the simple math of 32 bits and 400k users---that is north of 3Gbits in one direction, more than 6Gbits in full duplex at the current active player levels.

There are currently 60 some-odd worlds, split up in two continents...NA has just over 30 worlds. Soon there will be another data center in Europe as well. There is much potential and expectation for future growth in play here. Keep in mind also that this isn't just internal data transfers across the backend either--all this traffic is also relayed to the clients over the internet as well.

Now consider the real-life conundrum that is at the heart of North America's internet latency/stability woes. There is a known, documented issue with overselling last mile bandwidth against the established upstream bandwidth at the exchange points to transit/peering partners throughout North America. Multiple filings with the FCC by some top tier ISP's (specifically TATA and later Level3) going back as far as 2011 complained/warned about increased internet consumption in the US and how the last mile ISP's were neglecting to upgrade their upstream bandwidth capacity at the exchanges, and the potential for it to bottleneck the internet in general. Couple that with filings over the next few years trying to get them to operate under fair practices (eventually led to exposing the Comcast forced throttling of Netflix traffic). Testimonies were cited where top tier ISP's offered to share the cost to install 10 Gigabit ports, even offering to install and wire everything up at their expense...and those last mile ISP's STILL refused to upgrade.

Just this past year, the same accounts were cited online and before the FCC and Congress (Net Neutrality, Time Warner mergers, Comcast/Verizon scuffles...etc.). MAJOR INTERNET PROVIDERS were publicly shamed for over-utilized, overly congested, and flat out failing upstream capacity--and the simple fixes being proposed were to install additional blocks of 10Gbit bandwidth. In some cases, just even ONE card would have provided 25% more bandwidth at exchange points that were pegging no less than 90% utilization and would have been a HUGE improvement in staving off the congestive failures. In that one specific example, there was space for up to 8 cards, and only 4 were installed. And still... those ISP's refused to install just one single 10 Gigabit card, and the bottleneck continued to spiral towards congestive failure.

So...we have an established problem where the ISP's in North America want to dicker for nearly 5 years over installing 10Gigabit cards to support the increased bandwidth they have been selling to us....and you expect there to be no impact whatsoever if SE makes a "simple" change to provide 24 more slots that could have the potential to throw 10Gigabit/second worth of traffic on the internet....yes it is not per world, but combined it will have an impact. Remember, there are 30 some-odd worlds hosted in Canada at the moment, and we have seen documentation where 10Gbit upgrades are needed just to properly support the existing traffic flow we currently have.

There is far more to this problem than just having enough hard drive space to store the data.

[Kaedan]

Maybe ask them what's going to happen when that returning players campaign kicks off and thousands of inventories SLAM the servers?

Maybe ask how they plan to handle the game getting any new players, adding 450 massive inventory slots each.

Apparently we just can't spare the space!

It's useful to think before you post. Let's look at a full server like Balmung. Now, if they suddenly increased the data for EVERYONE on the server, that could potentially cause server problems. But if it's only a handful of people buying extra retainers, that's not an issue.

You can't create new characters on Balmung for the most part, so they don't need to worry about new players or those coming back.

On low pop servers, increasing inventory wouldn't be an issue. It's the high pop servers they have to worry about. I'm not sure how many people there are per server, but let's say 3,000 for argument's sake. You give 3,000 another 24 slots and that's 48,000 new slots. To equal that, people would have to buy 500 Retainers.

Also, maybe Yoshi-P wasn't thinking just 24 slots per person when he said the servers might have issues, maybe he was thinking another Retainer's worth. And then the gap gets that much bigger.

[Wade]

ItemID 24 bits
GlamourID 24 bits
Color: 8 bits (256 color options, conservative but high - unless they do R,G,B,A,M (M would be a bit, metallic, 1 or 0)
Durability: 15 bits (it's decimal, hidden, 8 bits for 0-200 + 7 bits for comfortable decimal up to 1/127th)
Spiritbond: 15 bits (as above)
Creator: 32 bits (it'd be an ID, not their name, however for transfer the name would be sent - say 1024 bits)
Materia: 120 bits (at most 5 item ID's)

238 bits for storage (< 30 bytes)

95,200 bits for a character inventory (11,900 bytes - 11.62kb) that might be openable at any time so is the worst case scenario.

11.62kb x 15,000 players = 170mb written to disk every 15 seconds for inventory


Position/Direction: 128 bytes
Animation: 1 byte
Selected Target: 4 bytes
Selected Target Type: 1/2 byte (player, NPC, object etc - point of reference for facial expressions)
Character ID: 4 bytes
Character Name: 128 bytes
Character Visible Gear: 27 bytes
Job: 3/4 byte (56 possibilities)

TOTAL: 293.25 bytes

64 players on screen: 18,768 bytes - and some people get lag at this point showing their bandwidth is saturated.
When it comes to storing characters however, you only need the ID and the position/direction.

Characters are there for reference to show you how much data they store about things that are not inventory stuff.

tl;dr: Inventory is a tiny part of the data they store and send out to clients - so small in fact that even at capacity of a server (if everyone was online at once) any modern SSD's would store the data far quicker than they currently write it out.

[Ilitsa]

WOW: The biggest bag you can have is 30 slots. They support 136 slots of inventory with the backpack.
Open the bank? 210 slots max.

On WoW:

the reagent bank: 98 slots where all items that can go into there stack to 200 a piece
Void Storage: another 80 items (Equipment only)
Guild storage: 98 slots per tab, 8 tabs total, equaling another 784 slots

and your personal bank starts with 28 slots, which can be expanded by 30x7 currently, which totals 238

that gives you 1200 (I didn't count for on-hand inventory either, only storage that's held off-hand) on the dot for storage, and I'm not even counting alts at all. Also you can craft from your bank, you don't need the items on hand. There's not a lot that WoW does great, but inventory is superbly handled imo.

[Eul]

Just turn the augmented AF into armoury already. I had much gear to put on retainer's inventory right now (And stuff.)

[papichulo123]

How would you guys suggest they fix the issue with data???? How can they improve the data center? Can a data center be upgraded???

[Waliel]

I did some quick maths about inventory space once. If you increased inventory stack limits by one byte for every slot to allow bigger stacks for 8 million characters (4 million registered accounts and two characters per account), it would increase the amount of storage needed by just over 7GB. Increasing stack sizes from 100 to 200 would not even require any extra space at all, unless Squenix is using signed integers for items for god knows what reason.

Inventory space is nothing compared to everything else they have to store and send about our characters.

[DragonFlyy]

So compared to other MMOs, either our servers are crap (which is ridiculous when we have so many players paying monthly mind you) or our inventory is so horribly innefficient that even flavor of the month korean F2P games do it better than FFXIV.

Let me break out some mumbo jumbo to put into context the amount of data we're talking about in an ideal inventory:
Every item in your bag is just an item ID and a quantity, that's about all the server needs to know and tell your client. It knows what item it is by looking up the id. It knows how much you have by the quantity.
All the items info, graphics etc are stored client side. Using the item ID it looks up the graphics, descriptions, etc from the data on your computer.

We know that items are stored client side since sites like XIVDB regularly datamine it from the client.
So thats all an inventory stored in a database should be - the item id of each item in each slot, and a number signifying it's quantity.

Do you want to know how big an item ID and a number would be? They can both be numbers, and if we're going to be extravagant they're full on integers. 32bits each. So each slot would take up two integers.
How big is an integer? 32 bits.
How many slots do we have in our bag? 100. 175 per retainer.
2 numbers for each slot, that's 64 bits.
100 slots that's 6400 bits.
Each retainer by the way would be 11200.
Bear with me now: there's 8 bits in a byte. Our bag is 800 bytes. Our retainer is 1400 bytes.
Let me put that in terms of a quantity you may know: kilobytes. A KB is 1000 bytes, so our bag is .8KB, Each retainer is 1.4KB.
Our bags plus two retainers is 3.6 kilobytes.

To put that into perspective, a 1TB hard drive holds around 1,000,000,000 kilobytes. So that's what, 50 bucks to accommodate 277,777,777.778 players. I don't think we have 278 million players right now.

Want to know something else? I used waaaaaaaaaaay too much data for the quantity, since the most you can have is 99 of any item. We can just use a byte (8 bits) for the quantity. The item id being an integer by the way, gives us enough space to identify 4,294,967,296 unique items. We could make the item ID even smaller to accommodate say, 2 billionish instead.

Simply put, you could fit the inventory into an even smaller number of bits than what I gave above if you were to optimize it.

"Hey Phenidate, whats all this mean to me?"

Well it means that if the people who engineered the inventory system were trying to be good engineers and not waste space, the amount of data of everything you own could fit onto a grain of rice.

It's not about the storage space, it's about the transfer. I agree that it is something that can be addressed but it was already stated by a dev that it is a problem with how often the server is queried to save our inventory. It saves ALL of our active inventory somewhere like every 15 seconds. For every character on every server. Retainers are not included in that save because they are not active at all times. They are saved when an inventory change is requested.