I think this needs to be tweaked for gamepad players.
If a teleport dialog pops up when I try to target something with A button, I will be selecting the dialog confirm button instead and I am back to town.
There should be an option to not show this kind of dialogs when in battle or an option to not pop them up immediately.