I don't have this problem with my gamepad but I use targetting mode B where I use the right and left arrows to target members and up and down to change modes. It's not as quick but works just fine if you get used to it. Would be faster to target them going up and down the party list though I agree.