The system OP wants already exists.

If you are incapable of performing the task that the savage content asks of you, and are unable to master it (including the dps checks) then you get no loot. However, if you can perform at a lesser level; say, what normal raid mode asks of you, you can get that loot. If you can't, then perhaps you can get what loot is available at a lower level than that, say, dungeons.

And so on.

There's already tiered gating based on mastery of the material, and the pass-fail mercy kill of a failed DPS check is the game telling you directly that you need to adjust to it to attain that next tier of reward.

It's feedback; and the feedback is 'Get better, and you can get rewarded.'