Rested XP only diminishes when your current class is receiving XP. Level 50 classes do not gain any XP, since they've reached the current level cap. But, from what understood from your post, the rested XP appears to be cleared when you switch back to the class you've entered the queue with.

If that's the case, then I would seem that you've found a bug. You should file a bug report so that SE can investigate the case of the problem and fix it.