I really do feel that, as of right now, there is no great solution for this:
1) You can remove XP entirely. This would keep it filled mostly with active PvPers. The issue is, there is not much incentive outside the Season rewards and the limited currency rewards. This solution would make queues longer, but lead to better quality matches. It also wouldn't really work well unless PvP rewards were greatly expanded.
2) You can tier XP based on performance. Make it like... (based on current XP yield for a win) 70% for 3rd, 140% for 2nd, and 210% for 1st. While this might incentivize people to participate more, you will still get people who join and do the bare minimum.
The game also needs to make Vote Kicking a bit more accessible. It's kinda hidden in the UI. There's plenty of matches where I'd vote to kick very inactive players if it wasn't a waste of time to try and find where it is (since I don't do it very often anyways). It should be an option under right clicking a player in your party.
I do hope this gets fixed. I'd rather wait 15 minutes for a good and competitive FL match than 2-3 minutes and risk getting on extremely passive teams.