BOOOM!
Let's be realistic please.
They need to make no restrictions. By specializing a CON to BLM you can simply get a bonus to nuking and a malus to healing/buffing. The current armory system can work very well with a specialization system, as many proposals that went that way clearly explained.
Taking away what people rightfully earned in game, as yoshida himself said, is not even on the radar.