You can summon the XMB with the PS button. That's about the closest you can get to alt+tabbing on the PS3.

It switches the controller priority though, so you wouldn't be able to do anything in game until you tabbed back. E.g. if you were mashing X in game when you opened the XMB, you'd just end up selecting the PS3's menu items (settings, achievements, etc.).