Seeing how this game offers a chance to level all classes on one character, and you even benefit from it in form of cross class skills, I say try them both to see which fits you better. Even if you are going to pursue the career of summoner, getting the swiftcast from level 26 black mage certainly wont hurt. Neither of them is overlooked in high level content so there is no one right answer to which one is better.