The fatigue that you accumulated, which is what made your XP go yellow, will slowly decrease as you don't play that class. It decreases the same whether you're logged out or on a different class or sitting around on the fatigued class--it doesn't matter as long as you're not gaining SP on the fatigued one. Note that once fatigue fades enough to let you get back to white SP, you will probably have to kill a mob before the color will actually change; it's kind of a bug or something.

One week after gaining the first SP on that class, fatigue resets to 0. Once you start gaining SP on that class again, the counter restarts, and then resets to 0 one week from gaining THAT first SP. It's a bit confusing to describe, but there's not really much to it.

Hope that helped!