It would be ideal, but also a little convoluted.

I'd be perfectly happy with a toggle in the party nembers menu for the party leader, that simply had 'Synch party to lowest members level'
That alone would solve almost all issues people have, except for your scenario of gimping yourself for a challenge, but really you can do that for FATEs and instances already.
The 'challenge' aspect is optional, but being unable to match party members level is what really impacts play.