A call for help system is an obvious fix for "help! I'm getting my ass kicked here!"

The least messy option for claiming is the first attacker (and his/her party) get claim, and nobody else gets to attack it unless the claimant calls for help. At which point the monster gives no XP or loot.

I wouldn't mind seeing roaming mobs divided into groups like levequest mobs, and have the pathing and arrangement of enemies set such that you don't have to worry about aggroing a group other than what you're fighting.

Alternatively, the moral of the story could be "don't suck" and if you get aggroed by a mob while fighting something else, oh well. Sucks to be you.