I think right now it targets the lowest number/letter mob and cycles in numerical/alpha order. At least this is my experience. So if you tab target it starts with A then moves to B regardless of distance. If you have C targeted it moves to D and so on.

I may be wrong but this is the feel I get from the system. Have not done extensive testing.