think of summoner as an arcanist who has learned to summon bigger and badder summons. You still get all of your arcanist skills plus all of the summoner ones, as long as you keep the item for summoner equip. From what I can tell, there is no benefit to NOT being summoner after you unlock it.