I always assumed everyone forgot who exactly they were because they had been plucked out of the timeline. People remembered the feats, but drew a blank as to who exactly did them, because they technically didn't exist anymore. Maybe.

Dropping them back in would restore the memory, although apparently with a bit of lag, presumably it takes a bit of time for them to mesh back in properly?

I dunno, sounds plausible enough to me.