There are 19 super-important skills between SCH and Arcanist, 6 cross-class skills with Conjurer, and 3 with Thaumaturge. That leaves 4 for Fairies. For me, it'd be more like 7 for fairies because I won't go for Thaumaturge until later. So, without Heavensward and only having Conjurer and Arcanist skills, I think it's perfectly possible to play with 32.

Unless, of course, those Thaumaturge skills are absolutely essential for proper SCH play. In that case, I'm not sure 4 slots would cut it for commanding the Fairies...