This target system is very good...just poorly made ^^:

I'd improve it this way:

Split it into up/down target mode and left/right target mode. You can set them both the way you want (like friendly at up/down and enemy at left/right) and have it locked this way (the main problem of current one is targeting self and mistakenly changing mode during battle, when trying to change monster).

To change the modes, i'd use the same idea used for camera zoom, making player hold the auto-run button and then pressing up/down to change up/down mode and left/right to change left/right mode. This way thee would be no risk of mistakenly changing modes midfight and would allow players that don't like this system to just leave them both as All (default) so it would work exactly as it did before the change.