There is nothing stopping you, other than the party who will immediately vote kick the person not wearing their soul crystal.