It's not really against any rule, just poor etiquette.

Rude? Absolutely..
Obligated to wait for stragglers? Nah.. It's an open world mob. People are going to pull.

What is "on time" for some people is still going to be "early" for others.