I don't have an answer for you as to whether you can decline helping them since I didn't try saying no. However, I'm pretty sure you need to choose to aid them in order to move the story forward.

As far as the story itself is concerned, the characters are going to go on the mission no matter what. You make the "choice" to tag along, but that's all. The characters in the story are committed to their course of action whether you join them or not.