I'm all for PvP but last time I checked, Ul'dah wasn't at war with Gridania and Gridania wasn't at war with Limsa. So no thanks on factions just yet.
Adventurer vs Adventurer outside of cities (DoH/DoL excluded obviously) would be fine and would not effect immersion and would not really do any harm to the community (people who gonna hate, gonna hate anyway). It could be just like trades, accept it or don't accept it.