I go on a face by face basis.

Though when i was a keeper with this face, i went for large, but when i went seeker with the same face the large were bugging me in the character creation screen so i switched to small.