I don't see it as something that will initiate Harassment as you can just decline the offer to fight anyway. Harassment, under my view, is someone that just continues to bother and pester you for various reasons despite your requests to stop.