Essential for making brass, but yeah, I feel it too.

I just wait till I hit a stack of 99 zinc ore then sell to NPC for gil. +2/+3 goes in the retainer for brass production or eventual bazaar sale.

Think of it this way - if you mined mythril easier and more frequently, it would be worth far less. Every zinc brings you closer to a mythril and makes that mythril more precious.