The Azem crystall give the WoL the ability to summon allways other beeings to him (i think it was never stated, who or what they are, at last, do i not remember that). So, from story telling is he allways outnumbering the enemie. Whe dont even know, what strenght they have. In theorie could he summon them for every encounter. Even, again one of the bugs at the start.

Removing the stone would make that stuff impossible. But, at the same time would it be again a pain in the ... for the writer, to explain, where allies are coming. Or, if he is solo again (like fight again Ifrit: cut scene for and after the fight shows, that WoL is solo, in the fight are other people there, Or Titan: you was senden first and it was stated, that the others are trying to follow as fast as possible, but arive only after the fight, and yet, are other people there in the fight).

Removing the light stones would probably be a better nerf.
Midgardsorm allready sealed it once, to test, if whe are worthy. But removing the stones would at last reduce the power a little.

By other hand:
If the WoL even truly that strong? When he needed allways help from others, to do the big fights.^^