The target filtering only works for controlling how your targeting works. In your case, you're just hitting X without targeting anything, so the targeting filter never comes into play. With nothing targeted, X will select a nearby default. Normally, in combat you'd select a target. The X default is more for talking to quest givers in town, which people still want to be able to do even if they don't normally target NPCs.

It's not very intuitive, I'd agree. But you just need to get used to targeting an enemy before attacking it. Use the left/right directional buttons to select the mob you want (making it a soft target) before hitting X (to make it a persistent target). The targeting filters you've configured will control what mobs/players/items/NPCs your directional buttons will cycle through.