I was able to fix this, you have to delete the associated gear set to the job in the portrait. That is entirely unclear in game and 99% of players will not be able to figure that out.