Potencies are calculated at the time of application. So if you do BB while in defiance and switch, it was calculated when Defiance was on so it will have the penalty. Similarly, if you use a weaponskill RIGHT before storm's eye wears off, the potency will still be boosted by 20% from storm's eye even though it fell off during the animation.