If that's their only reason for kicking you, they're assholes. My raid team has a member that's colorblind and I never even knew it until we got to T9 and he had trouble with the golems. As there was no way we were going to kick a friend (and a very good black mage) for something like that, we adjusted our strategy. The adjustment? Mark the green golem. It's simple and quick and solves the problem. Why your team couldn't be bothered to tell you what color of tether you had is beyond comprehension. It sounds more like an excuse to me.

As far as quitting, don't let something like that sour you to the entire game. As long as that was the only issue, I'm sure you can find a static that's perfectly willing to work with you.