It's because it's an online game. In order to obtain information about what is in your inventory, it must send a request to the server and wait for it to return a list of items. If there is a lot of traffic on your connection, it could affect the flow of traffic into FFXIV, although this is less likely if your connection is really good.

As time has gone on, more and more objects in the house use the internet. This includes phones and iPads for each family member, computers for each family member, let's throw in a digital watch, television devices to watch Amazon and Netflix with, and sometimes objects connect to the internet as well such as security alarm systems, or even fridges and washing machines if you're in the minority that bothered buying ones that connect to the internet. All these devices download updates (Windows, iOS, Android, app autoupdates).

I've personally found that this has been grinding on my internet capacity as time has gone on. It reached a point where there was an opportunity to upgrade my internet speed, and it was perfectly the right time for it: my connection was getting crushed by how many items were connected to the internet. But now I'm at a point where I've upgraded far beyond what I will ever need and can probably download FFXIV in moments - the entire thing.

It's further affected by the distance to the server. So if you are not close to the server (ie. in the US or OCE regions), your ping could be really high. Personally I don't notice this much, but I do a bit. What cut down retainer loading speed more for me was actually increasing my FPS cap (it was a night and day difference).

Another factor that can affect loading is whether or not you have an SSD. This is strongly recommended now, because your game is constantly loading/unloading assets, especially from the players wandering around and being rendered/unrendered, and that takes ages for traditional hard drives.

Beside all these factors that can contribute it, nobody really believes that SE has optimized their network and server processing code effectively and it's not appeared to change much in 10 years despite advances in technology, so that likely shows up more when a roommate is hogging your connection. It could help for now if they try to set a speed limit somehow, such as in the Quality of Service (QoS) or Bandwidth Control pages of the router, or on the device itself.