My take on it is that it has to do with route memorization and concentration. Most things used to cast magic (spell incantations, focuses, hand gestures, etc.) are used as shortcuts so that the magic user doesn't have to be fully focusing on the fine points of casting magic in a fight. Silence interrupts the incantation "shortcut" which is something almost all magic spells use. That makes it take more concentration to cast magic and in a fight that causes problems.

You can think of casting spells normally like using a program shortcut on the computer desktop to launch a program. Silence would make the shortcut not work and it takes time to open up File Explorer and hunt down the program's actual location in the computer. And doing it while fighting someone else at the same time...