You don't have to be the party that spawned them, and sometimes they pop on their own anyway. What ends up happening is the first party to land a hit on them "claims" them. In order to then also gain the gold chest (since more than one party can) other parties have to meet a DPS requirement, which can be pretty steep for stars. This is probably the biggest problem with Diadem since it basically penalizes teams that either try to pick up ads or have a gatherer among them off doing something somewhere else.