I think I'd just as soon see them instead remove the distinction altogether. I mean, don't get me wrong, I understand the hesitation in constantly reducing systems such as this to some dead simple state. It's not far off from the argument that every "main stat" could just be combined/reduced to "Power" and nothing would change. But this. Part of how I feel is definitely this exact problem, of not being able to easily tell (supposedly chat log text can be used to distinguish, but, who's reading that constantly? Mine's definitely not set to anything that could remotely be followed during combat).
But a big part is just how, there's not really any consideration to put in either way. No player is going "Oh, that's a physical attack, I can take that just fine," or "Ooh, casting a magic, eh? Time to use my magical shield." Of course not. All the player sees is "They're going to do damage, and I don't want to get hit." Obviously there are special mitigation tools, and to an extent some players now do need to make that consideration. But it just seems like a needless complication. Exactly as you say, without being able to tell the difference, it's sorta just a punishment for not "just knowing."