As a player with FFXI experience, I never expected much from the mouse controls. As Ferth says, playing with the keyboard only tends to work much better, though right now the two main reasons for this are: 1) The fixed key setup is unsuitable for one-handed use (with the other hand operating the mouse), and 2) the camera stutters a lot when controlled with the mouse.

As far as your problem situations go, the "Nearest NPC" button really compliments TAB targeting a lot. My own setup mimics FFXI with F1 being target self, F2-F8 are for party members and F9 is closest NPC. Unfortunately, unlike in FFXI, the closest NPC command only targets enemies your character is facing, so you can't target an enemy that jumped your from behind without turning around first.