Only way I can see "open world" raiding working, is via non instanced dungeon. DAoC had some of the best designed ones i've seen, with trash mobs more powerful than most bosses in the game, and mechanics in place where if you tried to engage a boss before trigger to get to it was handled, then everyone in dungeon would be automatically killed (not to mention, final bosses being completely immune to dmg till certain mechanics of their fight were taken care of).

Course these were still just glorified zergs, since to make any progress you'd still need to bring at least 40-60 people with you, and to beat the entire raid would take 2-6 hours.