As far as I can tell, travel time is affected by the total players using data centertravel at the same time for the entire region and not the population of the destination world/data center. If I'm making a very brief visit, traveling time is usually the same both coming and going even though I'm usually taking my Marilith characters to congested Crystal (since that's where my friends are) then returning.

Population on the destination world is going to impact the log in queue time instead.

Patch has brought more players back to the game so there are more using data center travel. Everyone ends up with longer travel times.

There's nothing for them to be ashamed about.