I could understand trying to justify the system if there wasn't instances with restricted loot drops but there is.

Not rolling on the loot should default to pass, not just toss it in your inventory for not rolling. It doesn't make any sense.