I don't consider this a issue at all. Like many players are referring, for experience I only see people being vote kicked after a long afk.
Other situation that can happen, is that you go into a new dungeon/trial/raid etc., if you say you don't know the fight and your team vote kick you, to be honest, is because they are not comprehensive and don't want explain the fight, they just want to finish the duty quickly.
If SE remove this feature more people would be afk during the Duty. You think it would be fair for the rest of the players to do everything while other person is doing nothing? Well this is just my opinion hehe.
Cheers Fellas.