I gonna play semantics here (and like Canadane, I not gonna speak for all men). Yes there is a fetish for Maid outfits, but I feel that it's more due to the cuteness factor rather than straight sex appeal. I feel that the tren either started in either Japan or France, the the former codifying it as a fetish. Because of the internet and Anime imports/dubs, the fetish spread further to the Americas (Not sure if Europe had gotten it form France earlier than that or not).
The Butler on the other hand, has a different appeal, while the Maid outfit plays towards an appeal of cuteness, the Butler's have an air of shall we say, class. They would be the people you see if 5 star restaurants, mansions and fancy parties that look like something outta Victorian London. In other words, the rich and aristocratic (don't help that business suits also resemble butler outfits, though that might just be due to how suits generally look)
