I think a lot of low level arcanists don't realize Topaz will steal aggro from the tank. They just know it is safer for them. They also don't realize their pet doesn't appear on the party list for the healer - which is important to point out to them.
You may have gotten people running alts who knew the dungeon or people who knew nothing about aggro. Low level dps spamming aoe's or conjurer's spamming medica is a common problem.
I don't consider myself a great tank but I don't normally have any trouble holding aggro.