Heck with it, when you think there are enough people there just start the fight. I was in the middle of a levequest one time and the mark was right where I needed to mine. My timer was running out and people just kept showing up for the fight but no one was starting it. With my timer about to expire, I said heck with it, switch to PLD lobbed my shield at it and got things going. Finished my leve with a few seconds left, no one even said anything.

Its just going to appear again anyways, so who cares.