I'm not sure if you can set visibility per class, but if you're seeing irrelevant skills, it sounds like some of your hotbars are set to be shared between classes when you're using them as though they're class specific. In character options, you can specify for each hotbar whether it's shared or not. Ones that are not shared should only have skills and actions you've set for that particular class. They should be empty for low-level classes without enough skills to need them yet.

It's just common skills that don't depend on class, like sprint or teleport or calling your mount, etc., that should go on a shared hotbar that would be available to all classes.