There is some penalty, but usually it's so small cross-class that people just ignore it. That's why you see everyone wearing Dodore's Doublet even if it's not optimal: the reduced stat is still far better than anything else available to any class.