Let me put it this way so you are satisfied. There are already games out there doing this, and it's working out very well.

Your very existence on this forum contradicts your statement because you are vulnerable to ID theft, even if you have Token/Token App, what's to stop your mom, siblings, GF/BF/Married partner, or cat, dog, cockroaches from picking up that token, log on to your account, and sabotage your stuff? What you are describing as a concern, is like your city wants to build you a better cross city transportation, and you are saying no to it because it can potentially run someone over.