I agree that there is an issue with tab-targeting in this game.

I often find myself switching the camera to a top down perspective to prepare for using tab so that it won't select a far target like the OP says, and it is annoying.

It'd be really useful to be able to switch targets that you're close to rapidly because many times tab will select an enemy that is completely out of my range, or even out of combat with anyone in a dungeon (which leads to accidental pulls of enemies).

I've tried using the target filter settings and the various targeting keybinds to set up my tab key so that it won't pick non "aggroing enemies" which is one of the options, or to "Cycle through Enemies (Nearest to Farthest)", which, again as the OP says, doesn't always function in the way that it describes, instead picking something strange. I think that it might have to do with the order at which the enemies appeared on your screen and which one(s) you've already targeted.

At the very least, adding an option to limit the distance in yalms (like a slider) at which tab can select a target would make it a more useful keybind for keyboard/mouse users, since I suspect that one of the reasons it works as it does is for gamepad users.