Targeting in this game sucks, pure and simple.

I play keyboard only so I can't help you much but I have to tweak things a bit to even get close to FFXI compact keyboard settings (which are WONDERFUL). I also use a combination of Target Nearest NPC, Target Nearest Enemy, and Shift+Num4/Num6 to target. I use an ALL setting.