I'm not against that idea. but really, they made the diadem so folks could do both. If you dont like it then make a PF group and SPECIFY what you are after (ie. Combat or gathering)
Most of the people on DoL that I have seen were trying to complete objectives to get ESO before swapping back to a combat class.