This is why I leave my elements at base. I have yet to see elements make a difference in defense.
The only effect I've seen is casting the right or wrong spells on mobs.
Like Aero II on a wisp for more damage.

What's really frustrating is weapon skills with an element only being negatively impacted.
If you were to use Iron Tempest(Wind) on a Sundrake(Fire) with your Wind at 174 it would do the same damage if you used it on a goblin with your wind at 15.
But if you were to use Red Lotus(Fire) on a Sundrake(Fire) your damage would be reduced.

Hopefully once Physical Levels are removed and stats are fixed, it will fix elements as well.