You can target yourself with Holmgang to avoid this.