If it breaks your immersion then don't do it?

I'm confused. Is one's "immersion" in a fictional world really that important and fragile?