Might be targeting issues. I tell game to only target enemies, but X keeps targeting interactable objects during fights.