That is not right either, since Azem very much definitely lost their title as Azem when they left the Convocation.
Likening the crystal to a badge or a banner is not working as well as you think it does, as the crystal has a purpose and is not just an object to be stuck on your lapel as a symbol. The Convocation crystals served as vessels for memories. And the Azem crystal has a copy of Azem's personal magic pasted into it as a personal gift from Emet-Selch to them. It was never meant as a symbolic object like a badge or a banner, it has a function.
Also the game never talks about the bearer of the title, it very much says the bearer of the crystal. As in, the object. The little tide-pod with the invocation in it. The forbidden magic gummy.
The title is absolutely inconsequential for the purpose of the crystal. The wording of what Hydaelyn says there is not.
And now before the argument "but they are always referred to as Azem!" pops up - it is the game's way of avoiding the hassle that would be giving a name to a character that is influenced by what you play as. Just like we do not get to see their face.
If you want to go into the "badge" direction, sure, be my guest - Emet-Selch really created that thing as a badge of honor to remember his cool friend.