They don't even need physical access either, if the password is saved on your computer, then if you get some spyware it could potentially reveal/send that file to whomever. Even encrypted probably wouldn't work as the program itself would decrypt it (it would need to so that it could use it).

I guess from SE point of view this way they can't be blamed for someone getting your password, they're only responsible for the password security on their servers.