Can't you change them at the calmity salvager?
I might be wrong about that as I've never changed the gender of my character, but I vaguely remember hearing about it.