The bit about not leaving your place in the present certainly fits, but if it just a daydream there wouldn't be other consequences in the people around you and how they treat you.

If you aren't doing something to history, why would O-App-Pesi trust you as a friend of Fye?