Bring up your duty finder (keybind is U by default) and just select leave.