I've had that happen, but I can always finish the quest anyway - just need to pacify the rest.