Older game systems are abandoned because utilization statistics indicate people are not interested.

If that weren't the case, the original (or new and improved, after complaints) version of Diadem would still be a thing.

When Deep Dungeons first came out in Heavensward, it was heavily utilized. Perhaps when Heaven on High came out they saw a distinct drop in the number of players who actually ran through it. Enough of a drop to make its inclusion in the current expansion something to ponder. Since the Bozja front was being designed, it might have been easier to place 'leveling alt jobs' as an option, unlike its predecessor, Eureka. They have statistics on Heaven on High and Deep Dungeons from both SB and ShB, and on Eureka from SB and Bozja from ShB. The number of players who participate is as important as "we do this because we've always done this".

Overall storyline also has a place in what does, or does not, continue to be developed. When we were still in Eorzea, Squadron Missions made sense. Once we take off for ShB, we aren't really in Eorzea from a story perspective, and continuing General Company duties didn't make sense. That, and the number of players who've reached the rank of "Captain" may be an indicator of its [lack of] popularity.

Similarly to the Centurio Hunts not being a thing on the ShB, so they got replaced with a ShB-specific set of Hunts instead. (People apparently enjoy doing the hunts for loot and gear for alts, so it continued from HW through ShB).

For the argument "number of Dungeons dropped", I found this Reddit posting to be illuminating, when you consider the number of Dungeon and Raid Bosses from a "there's more than just dungeons" perspective: https://www.reddit.com/r/ffxiv/comme...tory_timeline/ . The ARR timeline at the top has a typo, which is corrected later; the relevant charts I'm referring to are further down.