The challenge perhaps is what puts some people off but for many it is the rewards.
If you look through all the raiding threads that have come up over the years you will soon see a hell of a lot of people saying they simply don't raid exactly because the rewards suck. It's a very common opinion and has been for a long time. I'm not saying it's the only reason raid participation is so low but it's certainly a big one. there's even a few people that have said the same kind of thing in this thread.
So i'd say it's probably a much bigger reason than you might think it is. and is an important factor for more people than you might think.
on a more personal note it's also the main reason of people I know or knew stopped playing the game. because the gear just isn't rewarding or satisfying to get. and never actually feels valuable..
10 million players or so in this game. but looking at the ops data maybe 800k are active. makes me wonder how many of the other 9.2 million stopped playing for the same reasons as many of my friends. I'd guess it to be quite a few though. a lot of people have said this game has a real problem with player retention. hell some of these census threads have had the active playerbase at less than 300k.....