The duty finder doesn't pull people just on your server. It pulls from all servers. You know how you can select all languages to queue up for? Other people in other countries have the same option. Likely there are fewer FR and GR players.. so using some logic here... if there are fewer FR and GR players then they could and likely do have longer queue times if they want to try to get a only FR or GR group. So say you have 3 FR players and they want a quick dungeon run. Would you wait for your language to pop or select JP and EN so that you can get a faster pop on the dungeon you want?

Nothing in the game says they NEED to know English and because you don't understand the Duty Finder or relative population issues for specific languages then YOU are upset that they don't speak english. The only problem here is YOU.