What exactly is the downside?

Also everything that people have said here but also maps, having your dex/int jobs on an elezen and str/mnd jobs on a roegadyn, that kind of thing. Not a huge boost but there's not exactly a huge detriment to running 2 characters.