You can put your fairy actions on a regular hotbar and use the mouse for those. That's what I recently started doing. Unfortunately, because the controller is the primary input method, the controller placement interface is activated when you click the "Place" action with the mouse.