Stop stressing over getting it done as fast as possible. Do occasional Fates when passing by.