Theres an option to make it based on camera direction, and nearest mob.
Theres another shortcut for "nearest mob"
But imo, you're better off using the original targeting system, and not their half hearted attempt at mimicing typical MMO targeting. (Which was introduced at a later date, and defaulted for mouse/keyboard users unfortunately)
The ideal way to target is to use the "target the nearest mob" shortcut, or to use the "confirm" button (which is defaulted to num pad 0) I suggest "confirm", since it can do a few other useful things for ya in combat.
This is based on your characters direction, NOT your camera direction! BUT is ALSO based on your cameras line of sight, mixed with your characters line of sight (leaving u with the smallest window in which to target things) partial obstruction is considered full obstruction. (such as being too tall, or being on stairs)
Next, there is an option to tab through monsters in 2 directions. "Left of your target", or "Right of your target".
(If you dont target the nearest enemy 1st, it will pick the enemy on the farthest side of the screen instead. (which sucks)
So in this case, lets say you see 3 enemies on screen. (Left to right, marked 1-3.)
1 2 3
You want to target 1. You can either face the enemy perfectly, and press confirm.
or
You can press confirm, and after u see which target is picked (usually 2) you press left, and it will tab over to the left by one monster. (in this case landing on 1)
With this method you ALWAYS know how many times u have to press the tab key, to target the exact mob you want.
IMO this is much more effective than the typical WoW tab targeting, which leaves u with a generalized guess of how many presses. I personally rather know how many, press it, and not even bother to check if its on the right mob, because i KNOW its on the right mob.
(As a side note, I do wish they would implement the FFXI system of targeting, as it would both save you a button press, and help speed up the process of cycling through monsters with its take on the "Left/Right" cycling. I absolutely loved this system, and was disappointed FFXIV was loosely based on it, but with far less polish)