Currently there is Cumulative Enmity (CE) and Volatile Enmity (VE). CE typically builds up and stays with you, and is pretty much only lost by taking damage (or using JAs that effect enmity, THF has some... and Enmity Douse). VE rapidly decays, and generally serves as a hate spike. Kaeko has done a lot of testing on this.
My knee-jerk suggestion is to change the way enmity is accumulated and deteriorated across the board for all jobs, actions, spells and abilities.
The first step would be to define a list of tanking actions in battle (what is the behavior of a tank?). These do not necessarily need to be current tanking actions. A quick, incomplete list of my suggestions would be Provoke, Flash, Shield Bash, Ninjutsu Enfeebles, Chi Blast, Sneak Attack and Animated Flourish. These abilities should have a significantly slower VE deterioration rate compared to a melee attack. In conjunction with this, I would make the following adjustments to these abilities:
Shield Bash: Recast time significantly lowered to 60 seconds. Creates a huge spike of VE, deteriorates very slowly. Merits will lower the recast by three seconds each down to 45 seconds.
Chi Blast: Recast time significantly lowered to 60 seconds. Creates a moderately large spike of VE, deteriorates somewhat slowly. Penance merits will now effect the potency of the TP down effect, and lower recast by three seconds each down to 45 seconds.
Sneak Attack: Recast stays at 60 seconds, but merits adjusted to possibly lower recast to 30 seconds (each merit would shave 6 seconds off the timer). VE generated by a critical hit under Sneak Attack will be multiplied by 5. If used in conjunction with Trick Attack, the 5x hate multiplier will apply to the person the THF chose to Trick Attack off of.
Counterattacks: These will generate extra enmity.
The second step would be to determine how enmity deteriorates. I think as a general rule, enmity for non-tanking actions (melee damage) should deteriorate somewhat faster - but in tiers. The higher your VE, the slower it deteriorates. If a melee player is somewhat reserved and only attacks an enemy occasionally, their VE should be somewhat minimal. If a MNK has been going full blast on a NM for 15 minutes, he shouldn't lose hate so easily. If we break it down into four tiers, it could work as follows.
Tier 1: Low VE - VE is rapidly gained and lost over time.
Tier 2: Moderate VE - VE is rapidly gained, and steadily lost over time.
Tier 3: High VE - VE is steadily gained and steadily lost over time.
Tier 4: Extremely High VE - VE is slowly gained, and somewhat steadily lost over time.
This keeps the pressure on tanks to keep using their hate tools, and gives melee jobs the option to either tank, or rotate with another.
Finally, CE serves as a somewhat balancing factor. I would adjust CE so it serves as a floor for VE. Sure, those cures might not spike hate so easily, but each one slightly pushes up your CE, thus meaning your VE will never deteriorate below that level (until the mob knocks you silly). This would add some necessity towards using reliable tanks, competent mages and effective damage dealers. If a tank requires too much healing, or the fight drags on too long, then the fight will get somewhat chaotic.
This is by no means a complete system - but just a few ideas to change up the dynamic of battles and tanking.