I could be mis-remembering, but I think pressing the "go to armoury chest" button swaps to the saddlebags if you press it again (i.e., it cycles from inventory to armoury to saddlebags and then back).