You can disable targeting minions, more annoying would be if you get hit by a boss mecanic because you were staring at the minion lol.