I believe that removing potency is the better solution, but we could improve the situation by adding more potency or bonus effects to skills.

For stuns:

When the target is stuned during using an ability the potency could be trippled.

For silence:

Debuff that explodes for additonal doubled potency if the target's spell is interrupted.