Just look at the attack animation.
If it's a fist to your face, it's physical. If it goes pew pew at you it's probably magical.

98% of the time this will hold up.