If you're talking about tab (or controller equivalent) targeting, it only really works by cycling through enemies on screen from left to right, regardless of distance.

Personally, I find myself using the "target nearest" key binds far more. It's not great if I need to find a particular one to hit, but it's enough in most situations.