I used to tell myself I'd "go back for them for EXP on secondary jobs" but in the end it was 100% downsides. The pace of EXP for secondaries was much slower than non-quest options, the story felt like I was seeing it at the wrong time and I was less invested, my rewards were a mess because I only got bits and pieces of things at any given time. It was tedious as all hell. Since Stormblood I clear the map of all sidequests before moving on with the MSQ. No regrets, so far.