VIII's problems are myriad.
It came out on the heels of VII, which to this day is one of the most popular entries in the series. Expectations were unbelievably high.
Unfortunately, its themes and battle system were drastically different from its predecessor. Leveling up did nothing for your stats; instead you had to draw magic out of enemies and junction it to your stats to actually improve your combat efficiency. Leveling up did, however, increase the stats of enemies, as well as change their item drop and steal tables. Limit Breaks were present only as Desperation Attacks, limiting their tactical uses (well, unless you have Aura); getting new ones also requires a fair bit of know-how depending on who you're using (as opposed to just using them / killing enemies and using a manual for the final one). A fair amount of VIII's easily accessible combat augmentations are only available through Chocobo World, which runs on a Pocket Station... a Pokémon Pikachu-like peripheral that was never released outside of Japan. Weapons aren't bought and sold like normal, but upgraded at special shops with item drops from monsters. You only get a SeeD paycheck as opposed to earning money from battles (which makes logical sense, but gets vexing if you don't know how to raise that). Compounding it is that little to none of this is explained in-game, or otherwise not given the weight it needs.
The story, meanwhile, is centered around Squall and Rinoa's romance and Squall's abandonment issues. VII had more universal themes of self-acceptance and moving on after failure that resonate with a wider audience (and is part of why I find VII:R's ending so contentious... but I digress).
tl;dr: VIII had too much to live up to, an esoteric combat system, and a narrower audience compared to its predecessor.
My main complaints with IX are the inability to save Trance for when you really need it, the lack of leaked EXP leaving certain characters in the dust (Freya gets it worst, but Quina and Steiner also suffer for it), and the story focusing so heavily on Garnet and Vivi it becomes detrimental to the rest of the cast (Freya's character arc is completed early in Disc 2, Amarant's is pretty much self-contained in Ipsen's Castle, etc).
Still very good, both are, but the reasons they're not as popular as some other entries is pretty clear.