I think silence works in more ways than just preventing one from talking. I dont think every magic spell requires words, if so I would like all spells voiced too to show this ^^ I never see my characters mouth move when casting either, always the 'concentrating' face. I think the magic debuff also blocks the flow of magic in your body. Or it could be like Amnesia, and you just forget how to cast magic.