This.
Emet had (tied with Hythlodaeus; he was just a much better mage than Hyth, period) the strongest aetheric sight of all the ancients (part of the reason Azem and Hyth pushed for him to take the seat of Emet-Selch), and could perceive the unique colour of a persons' soul. Early on in ShB when he's spying on our activities during the quest The Oracle of Light, during his observation he says "then again... with a soul such as that...". Logically, there's only one person he could be speaking about, and given his strength of sight, there's zero way he could be mistaken. He knew we were the sundered Azem and was hopeful we could be convinced to side with the him and the other Ascians and tests us, but we're found wanting, yes. At that point it's more that he's unwilling to accept that we're his old friend. A "Azem could have done it, but since she/he clearly can't, this must not be Azem" sort of thing. Strikes me more as rationalizing his disappointment/grief.
I feel it also bears reminding that Emet had our soul stone prepared at some point before our final battle with him, per Hythlodaeus' shade; that quest is Etched in the Stars where we're picking up the soul stones of the lost Convocation members, the final of which is ours. Hyths' shade tells us straight out that even though the fourteenth seat (Azem) was considered disgraced due to us disagreeing with the whole Zodiark plan and defecting, Emet ("a friend", though again, logically it could be no other) had the stone prepared a long time ago, in secret. No, he couldn't accept that we were Azem before we best him; however it seemed he also really wanted to be wrong and had Hyth's shade leave those stones for us. The shade was more aware than the rest (no different from the real life Hyth, honestly), but still clearly influenced by Emet's hopes.
Also, uh... Emet is just Emet. He is indeed a grump-grump but clearly cares more for his friends/in general than he'd ever admit. Not sure why some people (not quoted here) think there's a weird disconnect between Elpis Emet, ShB Emet, and Endwalker finale Emet. One is just him before his myriad burdens, the other is him after shouldering them and going to extreme lengths to rectify what he saw as a great wrong, with the final being him after he remembered and accepted everything else. In Elpis he was actually horrified to hear that he ends up being the cause of great suffering and (understandably) vehemently denies it and us as a result. I mean... if someone told you that you would go on to cause devastation yet ultimately, despite your adherence to duty, give up your goals ("forsake my brethren", as he put it)... You'd likely react in the same way; you'd refuse to accept any of it.
Eh, this is a bit of a ramble as I've been low on sleep so I'm not sure how well I conveyed my thoughts... hopefully this was clear enough.