Make storyline roulette require the "new person" that actually chose that duty be required to hit a "ready button" to start the CS for a boss. This will at least make sure you could not trigger a boss fight and lock them out