The 3.4 update coming up soon will also give controllers some additional hotbar slots with its new W cross hotbar (activated by double tapping the L2 or R2 triggers). While most jobs are ok with the in-combat cross hotbar + out-of-combat cross hotbar + expanded cross hotbar, adding the new W cross hotbar may be helpful for summoners/scholars who have pet commands in addition to their own actions to deal with (or for players who just like to duplicate some of the more common menu options on their hotbars rather than going to the menu for them).

That should ease the concerns over the availability of easily accessible hotbar slots (before needing to swap hotbars, which gives access to all 128 slots, but which some players find awkward). The biggest remaining concern is targeting, but as Dzian said, setting up multiple targeting filters, along with party list / enemy list targeting, eases most of that.

(Window management is easier with a mouse, but I don't do it often enough to count it as a very significant difference. The only reason I don't like selecting things on the map with controller is that the targeting cross-hairs are white on a pale yellow background, so rather hard to see.)