They should put a summoning bell in the barracks. Or have your retainers appear there in person when they are not on a venture.