If you play on JP prime time I think it'll be better to stay. You might find NA servers a little bit empty at that time.
Valefor is one of the lowest pop server so if you want see more people in the open world you can always try servers like Chocobo, Shinryu, Bahamut, Tonberry. Though it's much harder to buy a house on those servers.

You can do a ping test to see how it is.

(JP Elemental)
ping neolobby01.ffxiv.com

(NA Aether)
ping neolobby02.ffxiv.com

(JP Gaia)
ping neolobby03.ffxiv.com

(NA Primal)
ping neolobby04.ffxiv.com

(JP Mana)
ping neolobby05.ffxiv.com

(EU Chaos)
ping neolobby06.ffxiv.com
Good luck and have fun! ^_^