Exactly. just don't join their party, but there's no need to insult them over it.
Calling someone unreasonable (particularly if it's the truth) isn't an insult.