I only get annoyed when they immediately leave after a single disagreement or argument with someone rather than try and work things out. Now granted it's not always the case, but sometimes sitting down and talking for a few minutes can fix a lot of things.