Sounds like it could be a pretty fun idea, especially the no fixed party part.
Some of my favorite dungeon runs have been when we do something weird like 3 dragoons and a scholar.