I'd start replying with "disability is no excuse to be an asshole" if they continue.

Because it isn't. You are not an object for someone else's ADHD hyperfixation.