Easily solved:
1. Add a config option to automatically deny duel requests (have yet to play a PvP game without this)
2. "Open World" PvP is usually not open world. It usually refers to a few zones that PvP is allowed in. Make those zones only reward PvP gear (i.e., gear that is useless for PvE) that way people like you can play without being expected to PvP.
You have to remember, a lot of us really want what you're so vehemently against; however, there's no reason we can't be bloodily maiming each other while you play with ponies in PvE land.
Edit: This isn't an untested idea either. It's been implemented with great results in other games.