A character with a maxed elemental stat and low Int will still do less damage to an enemy weak against said enemy then a character with high Int and a balanced elemental stat (as you seem to imply that you are). The only situation where you'd be weaker would be if the person high in the element that the enemy is weak too also had a high Int, however they'd be weaker then you against any other element that they're lower in.
And as for the fire dungeon bit, you could still use anything 'except' Fire to the same, basic effect: no more, no less. You just wouldn't be able to use water 'as' strongly against them. Wind, Lightning, Earth, and Ice would all do the same amount as your proposed non-elemental spells, because they aren't directly strong against or weak against Fire.