I think its really just a cultural stigma.

most people see purses are for Women, no one would suggest a man wear make-up or Paint their nails. Somethings just simply aren't for men. While of course it is just simply a bag you carry stuff in, Its designed and marketed with women in mind.

Therefor its considered inmasculine for men to wear a purse.

Do i agree with the social stigma? thats for another time