That depends entirely on how you define it. Fundamentally and originally, gender is not biological, but linguistic. It is arbitrarily categorical and used solely for convenience of lingual sets, normally following a euphonic ruleset based on a terminal or elisioned component such as an ending vowel or lack thereof -- e.g. -a ending nouns and -o ending nouns for feminine and masculine, respectively, in Spanish. Since then, however, gender has been either or both a stand-in for one's sex itself or for those things attributed to or packaged with it. True, what we choose to ascribe to a gender beyond the physiological consistencies is, by definition, artificial and constructed. But there's still a large portion worth talking about which is not just the binary distinction itself (0|1) that is very much real and proceeds our perceiving it. There are real differences. Not as many as we may like to think, but also perhaps more than many others may like to.
If two sets hold a one-to-one correspondence between them, does it matter that, say, one's immediate association to "red" may be a different shade than your own? The sole purpose is communication, and parameters, lenient to irrelevant measures of nuance, are essential to communication.It's funny that's your example because we can't actually prove that two people see the same color. They just both know to call the ball they're seeing the color on by that name because it's a social agreement that that color, whatever it looks like to them, is "red". Regardless if what I see it as is closer to what you see as blue.
Another thing that's amusing in your example is that words by definition are context-driven. There are Eskimo tribes that have fifty words for snow. Because in their society, the nuances and contexts of snow matter much more than it does to people who live in other places and speak other languages. So you saying 'there is only male and female' isn't so much fact as it is a testament to the society and political climate you live in.
Such lingual distinctions are created only when there is consistency and utility in those distinctions. Can such exist for genders? Can you generate words of consistent and distinct object that are worth that further differentiation? And would those words actually be spectral? I could take the modern use of "gamine" for instance, with a very distinct and unique sense of something alike to gender or gendered-ness, but that's not a position between male and female any more than those fifty forms of snow are measured by their progression between solid and liquid. It becomes something not quite describing gender, because it doesn't quite have anything to do with gender. And it's not merely because there are too few words for the existent sexes or how they may be perceived; it's solely because the things worth describing beyond that, outside of intersex or dualsex persons, clear-cut biological distinction don't really have to do with gender itself. And we have those words already. And if it actually becomes useful enough to make and use them, we'll eventually have more or new ones, just as others have faded from use. Having just a word each for solid, liquid, or gas, for electron, quark, or neutrino, isn't evidence of an empty system. It's just evidence of a technical division where the described is technically divided by the given factor. Other words for factors not thus divided are still free to surround and augment them.