The way I see it the 2 core problems are the absurdly high accuracy requirements and the AoE ability spam that results from hitting the thing.

So as an starting point, rather than adjusting the formulas why not just add an Accuracy bonus status effect while fighting these monsters and add a cooldown for their abilities so spamming them back to back isn't such a huge issue. They can be as powerful as you want, even one-shot you if you aren't paying attention and don't get away, but to get spammed with abilities faster than auto attacks is just ridiculous.