Could be the same enmity thing as Stealth, if you have any enmity (your enmity is shared with party, btw) then you can't activate Stealth and if it's already applied it will be force dropped.