Next time you enter an empty duty (whether it be Steps of Faith or a dungeon), kindly click the Vote Abandon button instead of leaving. It will put up a Vote Abandon vote but since you're the only one who's voting, it's guaranteed to pass.