It's a good question.
WHM and SGE are entirely doable on controller. You basically drop Repose and everything else fits just fine. After that, it's just putting buttons to taste where you think they're easiest for what you want. Each has 30 total abilities, I think, so adding Sprint, Limit Break (you WANT these), and a slot for Potion/Tincture (or Mount when not in combat) works just fine, you just remove either Rescue or Repose.
SCH is a little trickier, but you can kind of cram most everything in to make it fit. You probably have to remove Physic from your bar, but...we don't cast that anyway. When I do low level content, I just put Physic in the place I have Chain Strategem for higher end stuff and then swap back once I'm out. You have to manually trigger Potion/Tincture because there's no room for it like WHM/SGE, but that's not TERRIBLE. Alternatively you could always fine something you never ever use and swap that for Tincture when in high end fights. Lustrate maybe, since between Soil, Excogitation, Indominability, and Energy Drain, and even Aetherpact, you don't REALLY need it in most cases. Many fights you could probably drop Esuna (fights that have no debuffs to dispell, which is most right now). But basically as long as you can find 1-2 buttons to sacrifice, SCH can work on only two crossbars. You won't have room for all that and Rescue or Repose.
AST, on the other hand...is a mess. SCH has a lot of buttons and they're all useful, but a couple like Physic, and Lustrate or Esuna situationally, you can live without. AST kind of needs most of the ones it has. And then there's the targeting. I've gotten REALLY good at pressing up/down on the D-pad for quick party selection, but it's always going to be a bit slower than F-keys.
But I agree with you that the press and hold stuff is annoying. I have a similar issue with PLD and most melee just having too many buttons while WAR and SMN can fit everything they need easily.
Definitely a challenge. But for the Jobs it works, I definitely prefer controller.
EDIT: AST is just the perfect storm where you need lots of abilities but ALSO to individually target party members (the other healers don't need to do this much) where macros could be handy, but then you have even MORE buttons!