Hi there.
So, I had this idea roaming through my mind, and wanted to get this out of my head so I could start returning to a proper sleep schedule. It concerns the glamour system, and a way it could be improved. Now, I could talk about ideals, but I'd like to instead take a more technical approach, applying my knowledge as a recently graduated Software Engineer. I'll try to TL:DR those parts so that anyone could read this, but I tend to ramble inconsistently and struggle with English as a second language, so I apologize in advance.
A overly technical view on the Glamour System
Printable View
-
09-05-2023, 01:41 AMDaringDanA overly technical view on the Glamour System
-
09-05-2023, 01:44 AMDaringDan
Part 1: The current system
First off, I want to talk about how this current system works. Most of this will be guesswork, but it will include a certain amount of leeway from best to worst case scenarios. Also, I will be talking about this using bits, a simple data format that’s either 0 or 1, off or on. I just ask that you allow me this small indulgence as we move forward, since I need some setup for the second part.
The current glamour system allows you to store items in the Glamour Dresser. To make a system like this, you need a reference to the item. In the case of an API like Universalis, it shows that those identifiers are simple numbers. What kind of numbers? This is where the guesswork comes in.
If you're reading this part without programming knowledge, then here's a small piece of info you'll need for the next part: numbers in code are stored as a data type called INTEGER. There are a lot of terms I'll skip over for clarity's sake, like unsigned, but there are multiple types of these integers to handle different sizes, depending on server capacity.
- The smallest size that could work would be a SMALLINT (small integer), which can go from 0 to 65 535.
- A MEDIUMINT (medium integer) offers a bit more leeway, and can go from 0 to 16 777 215.
- The usual format some people go for is an unoptimized use of the INTEGER itself, which can go from 0 to 4 294 967 295.
Now, this was assuming a number was being used. Why is there doubt? That's because if you go look at an item on the Lodestone, you'll see a set of numbers and letters in the URL next to /db/item/[identifier here]. Letters are handled a bit differently in code. Each letter saved in ASCII (a format for defining letters in code) takes up space as a CHAR (character), so having a series of 11 letters ends up taking more space than the previous options.
Below, you'll see a small table comparing the sizes. As of patch 6.2, there are 800 glamour slots. As such, the total size is the data size multiplied by the number of slots. For clarity's sake, I'll be ignoring the Armoire.
TL:DR - This table above is the important part. These are my estimates for how much space (in bits) the current glamour system takes on the server per player. The reason it’s in bits will be explained in the next section.Code:+----------------------+----------+-----------+---------+----------+
| Data type | SMALLINT | MEDIUMINT | INTEGER | CHAR(11) |
+----------------------+----------+-----------+---------+----------+
| Data Size (in bits) | 16 | 24 | 32 | 88 |
+----------------------+----------+-----------+---------+----------+
| Total Size (in bits) | 12800 | 19200 | 25600 | 70400 |
+----------------------+----------+-----------+---------+----------+
-
09-05-2023, 01:45 AMDaringDan
Part 2 - The Solution: Flags
Flags are what you call a bit whose job is to inform you of a single thing. For example, did I unlock <insert glam name here>? 0 means no, 1 means yes. And as much as I could just say this is how it’s done, the solution is a bit more nuanced than that. Why? Well, you can find the answer below.
Code:+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Item Amounts | A Realm Reborn | Heavensward | Stormblood | Shadowbringers | Endwalker | Mog Station | Total |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Head | 552 | 535 | 522 | 501 | 383 | 126 | 2619 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Body | 546 | 536 | 507 | 493 | 395 | 145 | 2622 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Hands | 409 | 475 | 460 | 445 | 350 | 61 | 2200 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Legs | 453 | 495 | 460 | 459 | 368 | 101 | 2336 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Feet | 413 | 499 | 470 | 468 | 362 | 97 | 2308 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Earrings | 203 | 247 | 199 | 216 | 197 | 11 | 1073 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Necklace | 198 | 234 | 197 | 193 | 202 | 4 | 1028 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Bracelets | 205 | 234 | 194 | 191 | 182 | 3 | 1010 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Ring | 261 | 234 | 194 | 194 | 183 | 4 | 1070 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
| Total | 3240 | 3489 | 3203 | 3160 | 2622 | 522 | 16266 |
+--------------+----------------+-------------+------------+----------------+-----------+-------------+-------+
-
09-05-2023, 01:45 AMDaringDan
Making a single flag for every single item in the game would be a fool’s errand. Or an unpaid intern’s hellish and unfulfilling job. In other words, something I wouldn’t wish upon my worst enemy.
In other words, to take every item into account, you’d need to make 16266 flags, excluding weapons and off-hands. That is not something that is humanly possible or sustainable. There is, however, another way to get that same effect. When you understand that everything is made of bits, you can then make your own data types. Or rather, make a collection of bits as large as you want. For example, splitting them up by type and expansion as shown above.
I’ll demonstrate with an easy one: The number of Rings on the Mog Station. At the moment, there are 4. Let’s say you got the 1st and 4th one. What would be saved is the number 9. Why 9? Because bits can be represented by a number when using powers of 2.
Side note: In programming, there’s also a rule on how lists start at 0 instead of 1. This is why the 1st and 4th are identified 0 and 3 down below.
As such, you can do:
This same technique works for any category, but a simpler option would be easier to demonstrate.Quote:
1001 => 2^0+2^3 = 1 + 8 = 9
TL:DR - You can use a list of bits saved as a number to identify what glamours you unlock. Splitting them by expansion to make loading faster is possible. -
09-05-2023, 01:46 AMDaringDan
Part 3 - It’s not (that) easy
Now, I talked about how using those bits can identify the items in order, and that sounds great. But there’s a catch: we haven’t defined an order persay. You could say that we go chronologically through the list of items, and sort them like that, but that ends up adding extra load on… well… load times. So, what if we could give our system a bit of a push, and a way to easily access individual items quickly?
To do that, we have to give them a number as an identifier (or ID). From there, these items would be easy to find, since you wouldn’t need to look up the list of all items to find just one. Also, since there is now the possibility to retroactively add items to a previous expansion, like what we saw recently with the Fae King’s weapons, splitting them by expansion could be as simple as looking at the level, or as tedious as adding another ID for the expansion (0 for Mog Station, 1 for ARR, 2 for HW, etc).
In addition to that first part comes the UI. You would have the grid as it is, but it would show darkened versions of those icons for the items you haven’t unlocked yet. If we’re shooting for the moon, I’d personally like to have it in a new collection menu, with information on how to obtain a certain piece (aka, if it’s from a dungeon, trial, shopkeeper, crafting, etc).
Finally, there’s also the question of what to do with the current systems. Honestly, there isn’t much to do. The Glamour Dresser could contain the aforementioned UI while also continuing to support the Plates system. Items would still be registered by “offering” them to the system. Although, this time, there would be no need to remove them, so keeping a distinction between NQ and HQ would not be required (that would potentially double the amount of data required to save this on a server). -
09-05-2023, 02:39 AMJeeqbitWell, they could be floats/decimal numbers.Quote:
Originally Posted by DaringDan
They use a number. The ASCII is just a lodestone thing. Problem solved.Quote:
Now, this was assuming a number was being used. Why is there doubt?
Yes, bit tables are sometimes used for things like this. What makes you believe they aren't doing it as efficiently as they can?Quote:
The reason it’s in bits will be explained in the next section.
-
09-05-2023, 02:53 AMAmiableApkalluIf those numbers and letters fall in the range [0-9A-F], then what you're looking at is most likely the hexadecimal representation of an N-byte value. (See also: Base 64.)Quote:
Originally Posted by DaringDan
-
09-05-2023, 03:44 AMJeeqbit
That is a good point. That looks like hexadecimal.
-
09-05-2023, 04:39 AMKacho_Nacho
Daring Dan, I want to applaud you for presenting your idea in such a clear and concise way. I have not touched software development or coding for decades; yet, it was easy to follow your thoughts.
-
09-05-2023, 05:12 AMSindele
I don't mean this to be discouraging or insulting, but have you spent any time looking into the resources that exist for data mining and understanding both how the client operates and the data exchanged between client and server? There's an absolutely ridiculous amount of work that has already been done with regards to reverse engineering the client, and you could dispel some misconceptions very quickly with it.
That aside, from a design standpoint, flagging on a per-item basis is wholly undesirable for one big reason (and one immediately apparent if you're familiar with similar systems like WoW transmog!): model overlap. You don't need to save a colossal bitmask for 16000 items when thousands of them are just duplicate models, nor would you want to.
Hexadecimal is always going to be more compact than the same value in base 10. As far as I know, lodestone IDs are just short hashes used to obfuscate the 'true' item IDs. Universalis don't care - Universalis pulls the ID straight from the game's item table.Quote:
Originally Posted by AmiableApkallu
-
09-05-2023, 05:22 AMFenyxRisingSorry but none of what you said is hard, unless making a list of tuples is now hard ?Quote:
You could say that we go chronologically through the list of items, and sort them like that, but that ends up adding extra load on… well… load times.
[(2054, TRUE),
(2055, FALSE),
(2056, FALSE),
...]
A list of tuples of length 16266 is not something that is hard to send and receive. Retrieving the boolean from that list is at most O(n), and O(1) if you use the appropriate data type. This is not something that stresses modern hardware. Not even if you do it a million times.
The reason they don't want to do it is because they are too lazy to code up the UI, that's it, that's all there is to it.
Please go actually benchmark what you're saying, go do it, even with the worst languages that isn't going to take longer than a microsecond.
Of course, the fact you have to wait a billion years between depositing two items in a company chest without the game yelling at you that multiple people are using the company chest (which is semi-bypassable if you don't use the UI) probably tells us that the real issue doesn't lie with what you're saying at all, a lot of it is with how bad the UI - and how that UI interacts with the server - in this game is. -
09-05-2023, 05:43 AMEraden
Mao thinkings that might be good idea find out how GW2 does glams and then see if doables in FFXIV.
-
09-05-2023, 09:40 PMDaringDanThat's my bad. I should have specified I meant numbers for identifiers.Quote:
Originally Posted by Jeeqbit
Good to know. It was more of a passing thought when I looked at the Lodestone, but I should have looked into it.Quote:
Originally Posted by Jeeqbit
I guess I just looked at the different systems in the game, and wondered why the glamour system works the way it does. If you take into account that ARR was made while half the team was maintaining 1.0, it makes sense that a N-to-N solution would be taken, which would be easier to implement, but come with the limitations of the current glamour dresser. This is also why I added that last section, talking about how the solution makes sense on paper, but maybe not in practice, and that it may require some important changes. But, more than anything, I just wanted to try and get the ball rolling on a conversation about potential solutions.Quote:
Originally Posted by Jeeqbit
-
09-05-2023, 09:47 PMDaringDanHonestly, I wasn't aware of that. If you don't mind, could you link those?Quote:
Originally Posted by Sindele
To be fair, the transmog system was the first thing I thought of, since I already maxed out my dresser multiple times, and had to split hairs on what I'd keep and discard.Quote:
Originally Posted by Sindele
So Universalis is using the real ID, while the hex is a fake to hide the real IDs. Got it.Quote:
Originally Posted by Sindele
-
09-05-2023, 10:01 PMDaringDanI suppose for me, it's more about the time it would take to load the items, UI frame, images for the items, building the tooltips, and most of the front end stuff. Splitting it by tabs was what I was trying to get to. Similar to what they do with the marketplace.Quote:
Originally Posted by FenyxRising
They've been giving us a few QoL changes though. But with the scope of the game, I can't imagine that's an easy task for any team. Also, since UI is the first thing we see, they're always the first ones to get flak for things that might be out of their control, due to lacking data from the backend or other constraints.Quote:
Originally Posted by FenyxRising
To be fair, this was more a post to get a conversation started. And, seeing the response here, I'm glad it did just that. As for the time it takes, I should have mentioned that it wasn't just server times that I was looking into.Quote:
Originally Posted by FenyxRising
The double user interaction is perhaps the reason that we can't have a glamour dresser as a housing item (I think I remember YoshiP mentioning that subject), but the amount of items stored should be a separate issue.Quote:
Originally Posted by FenyxRising
-
09-05-2023, 10:04 PMDaringDanGW2 has a nice system, and I prefer their UI. That's what I had in mind when I made this post.Quote:
Originally Posted by Eraden
-
09-05-2023, 11:28 PMJeeqbitQuote:
Originally Posted by DaringDan
- They use physical machines rather than cloud servers. That means it's not really easily scalable like a cloud server. They decided against cloud servers because they are slower, despite the advantage of them being able to scale.
- Having to transfer inventories to different physical servers seems a real challenge for them, so for example you don't carry your chocobo saddlebag into a duty server or your retainers to other worlds.
- They are most likely taking into account network or database transfers over a large number of players at a given time as well.
As far as UI goes, we have experienced evidence of a disconnect between the UI team and... anyone else that knows how to code, in the past. They said it's "impossible" to show on tooltips when you have obtained items already, but there was definitely a way to do it if you really wanted to, even if the client got the information from the lodestone instead of directly from the server. And shortly after, they did it, proving their own argument wrong.
Maybe it is the UI team not working closely enough with programmers, but this really dented my confidence in how the UI team was being managed.
The glamour dresser itself was added after a few expansions, I recall, so it wasn't something we had in ARR. But we did have glamour prisms for as long as I can remember (although there used to be different types).Quote:
If you take into account that ARR was made while half the team was maintaining 1.0
Nobody is going to link that sort of thing on the official forums. It's against the rules.Quote:
Originally Posted by DaringDan
-
09-06-2023, 01:22 AMSindele
Glamours weren't implemented until 2.2 - early, but not at launch. I still have some old graded goldsmithing glamour prisms on an alt somewhere...
And yeah, won't be linking anything here. It's not exactly under wraps, though - poke around online and you'll find what you need. -
09-06-2023, 04:40 AMAlexandre_Noireau
Glamours are atrocious, were worse, but still is a terrible implemented system.
Specially when compared to the wardrobe in GW2.
We celebrating 2 color code-lines for clothes next expansion. GW2 since lunch has had 4.
The fact you can change any piece at any time but combat, stop and dye, without any hassle is just amazing.
In FFXIV glamouring is SUCH a process.