Well, the key thing for me is on screen visibility, and 32 is the maximum number of actions that can be seen at the same time using only the cross hotbar. Even actions that are not used in combat are still useful in between combats, especially when doing hunts and treasure maps.

But yes, in the end, it's all about priorities, and it would be different for each job's hotbar selection.