As someone who had absolutely no knowledge of the actor and his accomplishments outside of his cameo appearance, I went through the entire episode blithely unaware of what the fuss was all about.
In fact, I actually liked the scene, because of how it humanised the common soldier in the Lannister army. And I thought the actor actually did quite well, bringing home the fact that, in the end, the soldier on the opposite end of the battlefield was really not that much different from his counterpart on the other end.
But many of those who went into the episode fully aware of who the actor is, were unable to dis-associate the character from the person in real-life. And, to these viewers, the cameo was distracting, and ruined the episode for them.
What can I say? It wasn't a problem for me, but it was a problem for others. My life experience is different from theirs, and that affected the way all of us approached the story.
Chalk it up, then, to subjective opinion.
But in that case, to what extent is the criticism fair? To me, it's completely unfair, especially not when an actor is trying his best to play a role. We could criticise his competency as an actor. That's fair game. But, to me, it seems grossly unfair to say he's a mistake simply because you, the viewer, can't ignore who he is in real-life, and are therefore unable to judge his acting on its own merits.
The failure, to me, is with the viewer, not in the acting, not in the casting, and certainly not with the script.