Love this idea. Gets my vote!

As long as you limit the control of the spectator, shouldn't have any major issues (that couldn't be resolved). I remember back when I used to play half-life lonnnnng time ago using spectator mode all the time whenever I stopped playing to eat or do some homework. It was way better than cable tv!!