Were there other minions around him? If you send a swarm of your minions at the brick while it's at an aetheryte with other enemy minions, your minions will just attack whatever is closest to them, the Demon Brick usually being left out if there's a horde there. Try positioning your minions so their range of attack has the brick right in front of them and the other enemy minions on the other side. I had this trouble both with the brick and other bosses and had to re-position my minions to "persuade" them to attack the right target.