Technically Shirogane housing isn't just for adventurers. It's moreso for general Ijin housing. If you go to the district itself you can find other people in the markets, some of them looking like nobles, rich merchants, or politicians. I think the Ijin district was described as a more recent phenomenon, a result of the increased presence of foreign diplomats and merchants in Kugane. They needed their own housing to store themselves and their families, and they figured they might as well open it up to other foreigners as well.
