Yes, it's annoying to have to confirm that you really do want to progress through a dungeon when the only alternative would be to stand in front of the door until you get booted out. Obviously you want to progress. It would be a small QoL change that I'd welcome, personally.

As to Haauke Manor, they could change the tiny keys needed for progression to 'Large keys' or something so that it's clear what's optional to pick up and what is not. I can't think of anywhere else where the key/lever etc could be used to open optional areas as well.