I'll run a Caravan run regardless if I get a party invite or not. Much as I dislike the "Free For All" concept it's right now the the only tactic being used. However I always make certain to contribute be it holding the Caravan at certain points to spawn the extra enemies, healing/buffing, or trying to get in some sort of attacks. If you can't contribute in some form or manner then I strongly agree there should be some kind of penalty for it. Honestly what I'm seeing is possibly there being something like a timer implemented that will disallow immediate reentry from the Caravan runs, something similar to how they do it with Behest. A successful Caravan run will bar you from entering until say the last minute or two before the Caravan departs. This lets those waiting get a chance but still allows those who joined on the last run to enlist should there be room.