A overly technical view on the Glamour System

[DaringDan]

Hi there.

So, I had this idea roaming through my mind, and wanted to get this out of my head so I could start returning to a proper sleep schedule. It concerns the glamour system, and a way it could be improved. Now, I could talk about ideals, but I'd like to instead take a more technical approach, applying my knowledge as a recently graduated Software Engineer. I'll try to TL:DR those parts so that anyone could read this, but I tend to ramble inconsistently and struggle with English as a second language, so I apologize in advance.

[DaringDan]

Part 1: The current system

First off, I want to talk about how this current system works. Most of this will be guesswork, but it will include a certain amount of leeway from best to worst case scenarios. Also, I will be talking about this using bits, a simple data format that’s either 0 or 1, off or on. I just ask that you allow me this small indulgence as we move forward, since I need some setup for the second part.

The current glamour system allows you to store items in the Glamour Dresser. To make a system like this, you need a reference to the item. In the case of an API like Universalis, it shows that those identifiers are simple numbers. What kind of numbers? This is where the guesswork comes in.

If you're reading this part without programming knowledge, then here's a small piece of info you'll need for the next part: numbers in code are stored as a data type called INTEGER. There are a lot of terms I'll skip over for clarity's sake, like unsigned, but there are multiple types of these integers to handle different sizes, depending on server capacity.

• The smallest size that could work would be a SMALLINT (small integer), which can go from 0 to 65 535.
• A MEDIUMINT (medium integer) offers a bit more leeway, and can go from 0 to 16 777 215.
• The usual format some people go for is an unoptimized use of the INTEGER itself, which can go from 0 to 4 294 967 295.

Now, this was assuming a number was being used. Why is there doubt? That's because if you go look at an item on the Lodestone, you'll see a set of numbers and letters in the URL next to /db/item/[identifier here]. Letters are handled a bit differently in code. Each letter saved in ASCII (a format for defining letters in code) takes up space as a CHAR (character), so having a series of 11 letters ends up taking more space than the previous options.

Below, you'll see a small table comparing the sizes. As of patch 6.2, there are 800 glamour slots. As such, the total size is the data size multiplied by the number of slots. For clarity's sake, I'll be ignoring the Armoire.

+----------------------+----------+-----------+---------+----------+
| Data type            | SMALLINT | MEDIUMINT | INTEGER | CHAR(11) |
+----------------------+----------+-----------+---------+----------+
| Data Size (in bits)  |    16    |     24    |    32   |    88    |
+----------------------+----------+-----------+---------+----------+
| Total Size (in bits) |   12800  |   19200   |  25600  |   70400  |
+----------------------+----------+-----------+---------+----------+

TL:DR - This table above is the important part. These are my estimates for how much space (in bits) the current glamour system takes on the server per player. The reason it’s in bits will be explained in the next section.

[Jeeqbit]

numbers in code are stored as a data type called INTEGER.

Well, they could be floats/decimal numbers.

Now, this was assuming a number was being used. Why is there doubt?

They use a number. The ASCII is just a lodestone thing. Problem solved.

The reason it’s in bits will be explained in the next section.

Yes, bit tables are sometimes used for things like this. What makes you believe they aren't doing it as efficiently as they can?

[DaringDan]

Well, they could be floats/decimal numbers.

That's my bad. I should have specified I meant numbers for identifiers.

They use a number. The ASCII is just a lodestone thing. Problem solved.

Good to know. It was more of a passing thought when I looked at the Lodestone, but I should have looked into it.

Yes, bit tables are sometimes used for things like this. What makes you believe they aren't doing it as efficiently as they can?

I guess I just looked at the different systems in the game, and wondered why the glamour system works the way it does. If you take into account that ARR was made while half the team was maintaining 1.0, it makes sense that a N-to-N solution would be taken, which would be easier to implement, but come with the limitations of the current glamour dresser. This is also why I added that last section, talking about how the solution makes sense on paper, but maybe not in practice, and that it may require some important changes. But, more than anything, I just wanted to try and get the ball rolling on a conversation about potential solutions.

[Jeeqbit]

I guess I just looked at the different systems in the game, and wondered why the glamour system works the way it does.

1. They use physical machines rather than cloud servers. That means it's not really easily scalable like a cloud server. They decided against cloud servers because they are slower, despite the advantage of them being able to scale.
2. Having to transfer inventories to different physical servers seems a real challenge for them, so for example you don't carry your chocobo saddlebag into a duty server or your retainers to other worlds.
3. They are most likely taking into account network or database transfers over a large number of players at a given time as well.

I don't fully understand their reasons, because as an earlier poster stated, modern hardware is extremely good and it was even extremely good when I was doing tasks like this a decade ago. But they deal with a lot of users and seem to have their reasons.

As far as UI goes, we have experienced evidence of a disconnect between the UI team and... anyone else that knows how to code, in the past. They said it's "impossible" to show on tooltips when you have obtained items already, but there was definitely a way to do it if you really wanted to, even if the client got the information from the lodestone instead of directly from the server. And shortly after, they did it, proving their own argument wrong.

Maybe it is the UI team not working closely enough with programmers, but this really dented my confidence in how the UI team was being managed.

If you take into account that ARR was made while half the team was maintaining 1.0

The glamour dresser itself was added after a few expansions, I recall, so it wasn't something we had in ARR. But we did have glamour prisms for as long as I can remember (although there used to be different types).

Honestly, I wasn't aware of that. If you don't mind, could you link those?

Nobody is going to link that sort of thing on the official forums. It's against the rules.

[AmiableApkallu]

Now, this was assuming a number was being used. Why is there doubt? That's because if you go look at an item on the Lodestone, you'll see a set of numbers and letters in the URL next to /db/item/[identifier here].

If those numbers and letters fall in the range [0-9A-F], then what you're looking at is most likely the hexadecimal representation of an N-byte value. (See also: Base 64.)

[DaringDan]

Part 2 - The Solution: Flags

Flags are what you call a bit whose job is to inform you of a single thing. For example, did I unlock <insert glam name here>? 0 means no, 1 means yes. And as much as I could just say this is how it’s done, the solution is a bit more nuanced than that. Why? Well, you can find the answer below.

+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Item Amounts | A Realm Reborn | Heavensward | Stormblood | Shadowbringers | Endwalker | Mog Station | Total |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Head         |       552      |     535     |     522    |       501      |    383    |     126     |  2619 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Body         |       546      |     536     |     507    |       493      |    395    |     145     |  2622 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Hands        |       409      |     475     |     460    |       445      |    350    |      61     |  2200 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Legs         |       453      |     495     |     460    |       459      |    368    |     101     |  2336 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Feet         |       413      |     499     |     470    |       468      |    362    |      97     |  2308 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Earrings     |       203      |     247     |     199    |       216      |    197    |      11     |  1073 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Necklace     |       198      |     234     |     197    |       193      |    202    |      4      |  1028 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Bracelets    |       205      |     234     |     194    |       191      |    182    |      3      |  1010 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Ring         |       261      |     234     |     194    |       194      |    183    |      4      |  1070 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Total        |      3240      |     3489    |    3203    |      3160      |    2622   |     522     | 16266 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+

[DaringDan]

Making a single flag for every single item in the game would be a fool’s errand. Or an unpaid intern’s hellish and unfulfilling job. In other words, something I wouldn’t wish upon my worst enemy.

In other words, to take every item into account, you’d need to make 16266 flags, excluding weapons and off-hands. That is not something that is humanly possible or sustainable. There is, however, another way to get that same effect. When you understand that everything is made of bits, you can then make your own data types. Or rather, make a collection of bits as large as you want. For example, splitting them up by type and expansion as shown above.

I’ll demonstrate with an easy one: The number of Rings on the Mog Station. At the moment, there are 4. Let’s say you got the 1st and 4th one. What would be saved is the number 9. Why 9? Because bits can be represented by a number when using powers of 2.

Side note: In programming, there’s also a rule on how lists start at 0 instead of 1. This is why the 1st and 4th are identified 0 and 3 down below.
As such, you can do:

1001 => 2^0+2^3 = 1 + 8 = 9

This same technique works for any category, but a simpler option would be easier to demonstrate.

TL:DR - You can use a list of bits saved as a number to identify what glamours you unlock. Splitting them by expansion to make loading faster is possible.

[DaringDan]

Part 3 - It’s not (that) easy

Now, I talked about how using those bits can identify the items in order, and that sounds great. But there’s a catch: we haven’t defined an order persay. You could say that we go chronologically through the list of items, and sort them like that, but that ends up adding extra load on… well… load times. So, what if we could give our system a bit of a push, and a way to easily access individual items quickly?

To do that, we have to give them a number as an identifier (or ID). From there, these items would be easy to find, since you wouldn’t need to look up the list of all items to find just one. Also, since there is now the possibility to retroactively add items to a previous expansion, like what we saw recently with the Fae King’s weapons, splitting them by expansion could be as simple as looking at the level, or as tedious as adding another ID for the expansion (0 for Mog Station, 1 for ARR, 2 for HW, etc).

In addition to that first part comes the UI. You would have the grid as it is, but it would show darkened versions of those icons for the items you haven’t unlocked yet. If we’re shooting for the moon, I’d personally like to have it in a new collection menu, with information on how to obtain a certain piece (aka, if it’s from a dungeon, trial, shopkeeper, crafting, etc).

Finally, there’s also the question of what to do with the current systems. Honestly, there isn’t much to do. The Glamour Dresser could contain the aforementioned UI while also continuing to support the Plates system. Items would still be registered by “offering” them to the system. Although, this time, there would be no need to remove them, so keeping a distinction between NQ and HQ would not be required (that would potentially double the amount of data required to save this on a server).