You cant go by that logic that they heard Ilberd's voice. We recognized his voice because we were conditioned to look for it, or we thought to look for it, since this is a work of fiction with limited voices. In real life people have different voice registers and depending on what is said, and in what tone, voices can sound similar to one another. There is a reason why someone might confuse you for your mom or dad over the phone, because you might sound similar without a basis to go off of. In the case of the game, while our character might think its Ilberd, it cannot be verified until he removes his mask or performs certain manorisms, specific to him. This is one of those instances where reality is stranger than fiction, our minds can be tricked rather easily.