1.) Latency -- In PvE, latency is a huge issue with certain fights, but you can mitigate the effects of latency using memorizaion. By memorizing the boss attacks and utilizing AOE markers to see where attacks are directed, you can get out of an attack way before it goes off. In PvP, this is not the case. People will be using AOEs that have no marker or indication, and when you're constantly under attack, you'll throw a heal or a potion on someone that is already dead server-side, you just haven't seen it yet. Attacks will whiff, and just like the Beta PvP, it may just be icerink the game.
2.) Gearcheck -- This game is heavily gear based. A full set of armor grants you about 2000hp and 100s of damage. In PvP, Morale has a high chance of being the main stat, as it has the exact opposite effect of Determination. So whoever has the best gear wins, it's not really a skillcheck. For this reason, I highly believe that Level 40 pvp will be the "go-to" pvp since level 50 has too much variance with the item levels.
3.) Forced party composition -- 2 dps, 1 healer, 1 tank is forced in PvP unless you have a full team. So if you're going in as a lone wolf or a random, chances are you will be at a disadvantage. Team Customization will belong solely to those who are going in with friends.
4.) FRESH MEAT -- When you are in PvP, you become an NPC. Literally. Others see you as an npc called Fresh Meat. Character expression has been thrown out the window. I would love some form of private match option that removes the fresh meat restriction, there's no need for it among friends. Don't award tokens or anything for private matches, just let us have fun.
Each of these concerns is pretty high on their own for a hardcore pvper, but all of them together? It's really not looking so hot at launch. I would love for Square Enix to comment on some of these issues though.