There is a notable quest you will miss out on if you don't have a tank/physical/magical/healer.

But on the other hand you will be dealing with a lot less inventory issues by not needing stuff for every role.