Pets indeed have different 'scaling' factors that come into play when looking at their potencies. This has been known for quite a long time, but I'll leave the technical explanation to someone who knows it better than me. But as you've found out for yourself, it's quite easy to see just comparing the numbers of something like ifrit's '100 potency' egi assaults to your own 100 potency ruin 3's (with no dots).
So yes, suffice to say it means the potency number you see on most pet related tooltips is a lie, and it's not real potency. In practice, garuda potencies are roughly 80% of 'player potency', and ifrit's are more like 48%. Demi summons use 'garuda potencies' so that 650 akh morn/revelation is actually 520, and wyrmwaves are 120 not 150. Those wonderful ifrit egi assaults that bloat up the rotation? They actually do 48 potency for your trouble making them even more pathetic than they look, heh heh.
This was the case even in Stormblood when summoner damage was just fine. So it's probably not like they're balancing the job around the wrong damage numbers or our pets are bugged without them knowing or something. It's just a very unclear game mechanic that is not explained anywhere.
Ifrit actually lost its autoattack going into Shadowbringers for some reason. It's true that ifrit used to have AA + burning strike spam, while garuda as a caster pet only has wind blade spam but no AA. That was one reason why (contribution from contagion aside in Stormblood), ifrit did more than damage than garuda even with a worse potency multiplier. Now however, you will see ifrit only does burning strike and nothing else.
Several people have reported the loss of ifrit AAs as a potential bug since the launch of the expansion, and I'm given to believe it has been moved into the 'accepted bugs' category. If ifrit's autoattacks are returned to him, that might indeed be one source of missing damage that would improve summoner numbers.