Be sure you're not engaged in a menu like talking to an NPC, crafting, or using the marketboard. That'll block you from switching.