At the very least they could add in an option to invite people from your friends list or FC if someone leaves and you need a replacement...
I'm sure they only reason they haven't added something like that in would be the fact that it would encourage undue vote dismisses in order to try to sneak friends in at the very end of a duty :/