It's actually a significant fashion style in Japan, to the point that people have made references to a "Lolita subculture". Since you're all playing a Japanese game (as in made by Japanese), it's perhaps no harm to know that in Japan, they often adopt foreign words and quickly evolve it into something that had little or no relevance to its original foreign meaning. Lolita fashion being one.
Yes, it's not big if you consider all of the fashion world, but in the Anime/Comic/Game market, it is considered part of common knowledge by now.
But I suppose there's no way to avoid people feeling creepy; not everyone can adjust their first impression when they go from ignorance to being informed. It's why discrimination exists in human society, after all. There are still people who feel a darker shade of skin is dirty and think gays are disgusting. Same thing here. People can be quick to judge and they won't bother learning more about it because it doesn't matter to them either way.
