Alright, so as I mentioned in the title, spoilers ahead, and this is also just a theory that I thought about while in the shower. (Yay shower thoughts!) Sorry for the wall of text!
So, I will start off by saying that yes, they could have gone with something different than "Hey, go back in time and figure this out OH WAIT EVERYONE BUT YOU AND ONE OTHER PERSON HAS THEIR MEMORY WIPED!"
However, within the confines of what the story is I think it all fits together rather nicely.
While I was playing through the first time, I always wondered why Emet-Selch decided to help us so much. At the time I thought "Oh, he just wants to use me as a new light warden" and to an extent that was true until he said that he was disappointed in our strength and was hoping that we would be able to handle it (Or something to that extent) and again, questions were there. Then the stuff with Elidibus started happening, and soon I realized that there was some hidden connection. I started thinking that I was an Ascian myself! Why would they recognize me as Azem? Did I come from that time and just...forgot my entire existence up to that point where I started adventuring?
After the two were gone, I thought that was the end of it, but NOPE! Hythlodeus came in in his created shade form. Calling us his "New old friend." Again "I have to be an ascian" entered into my mind (And honestly it would have been an amazing twist in hindsight) but for those that finished the game, we know this isnt true.
Before Elidibus and Emet die, they both recognize our character, thinking that he is Azem. In Elidibus's case, I think it was simply the time that passed and his obsession with his job that made him forget, but anyone who played through the game knows why Emet forgot. So how did he recognize us?
Well, thats the next part. When Hermes uses Kairos, I believe that it is a flawed system, which is actually alluded to early on while on Elpis. We figure out that some of the creations continue to exhibit the same instincts even after Kairos is used on them and erasing all of their memories. I believe that this is why at the end (Before he dies) Emet remembers us as being Azem. In his subconscious, he still harbors the same memories, and thats why he helps us the way he does. This is why Hythlodeus is different from the other shades in Amaurot. Kairos couldnt get rid of the subconscious memories and instincts that Emet had from all those ages ago.