Again, this is why the triggering of a move with battlefield aoe of either damage or nasty effects would be tied to specific monster phases. You know, like how Bahamut swaps between Megaflare, Gigaflare, and Teraflare based on how much HP he has remaining. Or like Spike Flail back in the day which was triggered almost exclusively when one took agro in certian locations in relation to the mob. In essence during a specific phase of a monster, it getting hit with massive damage on the scale multiple SMNs can produce causes the mob to do something nasty, or during said phase being highly resistant or evasive to certain abilities. If it is tied to specific phases, it doesn't make any class unplayable, and instead encourages bringing more variety to fights thanks to mechanics.
As to being able to tell the difference between pet types... I am pretty sure they can differentiate between pet magical and physical damage, as after all they have gear that can give bonuses to specific ones. Also don't forget certain avatars are more specialized towards physical damage as well.
The biggest problem with SMN's damage is how safe it is to use it in the first place. I am not saying make every summoner action generate a lot of agro, but some do need to generate more of it. Grabbing a random person to aggro the mobs in a way that keeps them off the hate list so a group of smns can pop Astral Flow/Conduit and blast away and clear them all in perfect safety does not work for long term health of the game. Should Smn be able to put out ridiculous damage while using those abilities, yes. Should Smn be able to use those abilities and the mob never really bat an eye towards the Smn? This is where the problem lies, and one of the reasons I propose adding emnity components to their 1hrs, and lesser amounts that decays fairly quickly for damaging blood pacts(enough so that Conduit+multiple BP's at the very least pull agro from a recently summoned avatar). The right amount is hard to say, but should be enough where unless the tank has had time to build significant enmity the 1hrs run a very real risk of pulling agro in a hurry.
Again, with having phases to the monsters where using smn 1hrs is bad, either due to counter attacks or them being ineffective, helps even things out as well and adds a little bit of skill requirement to smn by having to pay attention and know when to use their abilities, instead of defaulting to pop 1hrs and just obliterate.