Not going to lie, this seems not well thought out. It might seem like a good idea at first, but take it to the wider spectrum of how you manage everything and it falls apart completely.

The first thing I will address is the enmity value itself. You would need a dynamic gauge that told you what your enmity is and at the very least how far behind the second in the list is. This will cause a fluctuating gauge that changes constantly, which makes it difficult to plan ahead, it makes it difficult to know when you should use something and when not to. There is a reason monks hate the RNG on chakra stacks, it just is not consistent.

Next is gearing. What you suggest is a lower geared tank has to use more enmity tools to keep hate over higher geared DPS. So the tank is going to do less damage just because they might not have the highest gear for the dungeon, but to top it off, since they have to use more enmity skills to keep the hate, they will do even less damage. You essentially punish the player for not being a high ilevel, despite the fact they might be running the dungeon to get to the higher ilevels. Add to that that the higher DPS might also have to slow down their DPS so that they don't out enmity the tank, causing even slower runs. With the current system, these issues are non existent. You are not punished just because the tank happens to have a lower gear ilevel.

Next is skills and how they manipulate enmity. How do you determine what each one costs in terms of enmity, how do you manage how much you gain from a skill. If you make enmity generation proportional to damage, then better geared tanks will generate more enmity and by extension, have more enmity to work with. This then ties into what do the enmity skills cost? Make it a static value based on level or change it based on gear? But, changing it based on gear or even level is not going to be fun. Having to potentially rethink how you do a fight just because you changed a piece of gear won't be fun. There is a reason they normalised MP across all levels after all.

then, as an addition to the above, how do you deal with crits and direct hits? Do they give more enmity, or are they ignored. Again, noone like RNG on the gauge, it makes things too messy to plan out, plus, the same would be true for the DPS. When they crit/direct hit, you will suddenly lose a chunk of your enmity lead. This is just a messy situation. Even if you ignore crits and direct hits, just individual potencies on DPS abilities will change how the fights go. Dropping your enmity and then having a summoner suddenly use a 650 potency Revelation from Demi Phoenix, or a BLM using a 380 (before AF3) Despair and then an insta cast 750 Xenoglossy straight after. That would be a nightmare depending on where you are in your rotation and it is something that is completely out of your control and you cannot plan for. You should not be expected to keep track of where a DPS is in their rotation just so you can keep track of your own enmity better.

I haven't even touched on your suggested abilities but as an example, Bloodbath. Why would you want to convert enmity to HP when your entire goal is to build enmity. Use it too much, you won't have enough for your fell cleave spam and you could even lose your lead in the process. 15 seconds is a long time in your system for someone to not be building enmity.

Now, lets break down the points and try and find a system which isn't riddled with faults and failings. The long and short of it is this, you want a gauge that you increase with abilities and you can spend it on others. Now, with all the inconsistencies that come with having it tied to damage and gear, lets normalise that and base it on potency. But this can still come with issues of massive DPS bursts happening at the wrong time and you losing it. The only way I can think to fix it would be to have the DPS not affect your gauge at all. This obviously goes against what was envisioned, but it is the only way I can try and get rid of DPS burst inconsistencies. So, what we are left with is a gauge that, at this point, might as well be separate from enmity, so we will normalise the gauge increase and cost. However, if you still want the tank to have to focus on enmity as well, then you can have your 2 combos to deal with it, an enmity one and a DPS one. Going on from that, it depends on how the 2 combos are balanced as to how they will be used. In the end, if the enmity combo, with all the enmity skills to go with it, produces less damage than that DPS combo, people will try and use it as little as possible, which defeats the point of building up enmity, however if the enmity combo does more, then what is the point of the DPS combo, even the OT would use the enmity combo because more damage, if they are equal then what's the point. The first case where DPS > Enmity is basically what SB tanking was, noone wants that. Enmity > DPS just cuts out the DPS combo from ever being used, so what's the point. Might as well have 1 combo that does it all.....Wait, that sounds familiar. 1 combo for damage, a gauge that fills and empties based on what the tank does and is independent of others, that sounds alot like the current system in a nutshell.

I am interested to see if you have any counter points, but there is a reason the system has ended up as it is, not only in FFXIV but I believe in other trinity MMOs as well (though I have no personal experience and this is based on what I have heard). The current system was inevitable and is the natural progression of following the logical steps based on the fact that the ability to do your job at a fundamental level should not be impacted by the rest of your team.