This is the caravan escort from camp tranquil to quarrymill

I have spoken to the guy to quit the escort choosing the "Aye, bugger this" option but it does not do anything, the timer and quest status are still there, the music still plays and I am stuck returning to camp tranquil.

I waited for the timer to run out but it was a few minutes after that when the quest actually decided to end, music was still playing and the yellow quest circles were still there for a good 2-3 minutes after the quest objective tracker disappeared.