I mulled this a bit, does anybody actually know where this idea was first brought up?

I personally remember it from the novel Märchenmond by Wolfgang and Heike Hohlbein, so that's in 1982 (it also had the amazing conversation about having windows on the inside but not on the outside). But the idea has to be older than that in some fairy tales and such, no?