First, a quick comparison of things my character has:
NINJA:
Kiss of the Wasp. Level 6 toggle. Increase damage by 20%
Level 1 attack Spinning Edge has potency 150, (or 180 with KotW) Level 130 Weapon damage: 57
BARD:
Wanderer's Minuet. Level 52 toggle (with a 3 second cast). Increase damage by 20% (and turns off auto-attack).
Level 1 attack Heavy Shot has potency 150 (180 with 1.45s cast time with WM) Level 130 Weapon damage: 51
I think a very, simple fix to the issue would be to simply remove that bolded bit, (the casting time) and adjust the underlined parts, (Weaponskill Potency and/or weapon physical damage) to be where they need to be for job balance.
I'm a Lvl 57 Bard, Lvl 50 everything else pre-Heavensward. I really enjoy all the jobs a lot, but I have always mained bard because it was a job I could shoot bad-guys with while running. I would like for it to go back to being that. Casting time is not needed in this job for balance. Other things can be tweaked for that.