you can solo POTD (at least up to 100), you can do all of the pomanders, treasures yourself. You don't need to jump into a mindless party on speed unless you're only there for exp.

sadly, the exp for fates (and sidequests) is pathetic these days. That's why people aren't doing them.

It doesn't need to be removed (in fact, it's probably the best additional content I've seen, and it's longevity speaks to that)

If you don't like POTD, don't do it. Problem solved.