You have to contact SE support for them to remove the physical token.