I'm sorry to hear that people are kicking you out due to playing Blue Mage. But at the same time I feel the limitations that Blue Mage had is why they kicked you out. You are stuck at level 50 while they are probably level 70 and want to just clear the content fast and then leave. Blue Mage in this scenario acts similar to a leech in that regard.

I think people have posted some advice about forming your own party and being specific about what you want to do. You can't expect everyone to welcome you with open arms as you are playing a job that isn't exactly beneficial to a group of level 70. (Even though I'm sure they should be able to solo most of the EX primals by now)