Here are my issues with it:
1) Slow loading, like OP said. Thanks to DC-travel and the buddy system's bugginess, half the time I can't even tell if certain friends are online because it loads them as offline or greyed out until I take that second step and click the little circle icon next to their name. This also makes cleaning out my friends list difficult; every time I delete a person, the entire list refreshes. It takes ages to remove multiple people.
2) No simple way to communicate a message to everyone on my friends list at once. Maplestory had a buddy chat channel. Anything you said in this channel was pushed to everyone on your friends list, and vice versa. It was crude but effective at what it needed to do. XIV has nothing like this. If you want to ask everyone on your friends list if they're interested in doing content, there's no way to quickly do this outside of making a crossworld linkshell (multiple, really) and begging all of your friends to join.
3) If I add someone to my friends list but regret it later, removing them on my end does nothing on their end. They can still see when I'm online, where I'm at in game, etc. This shouldn't be a one sided thing. If I remove you from my friends list, I should vanish on your end, too.