I always use it as a Bard if there are no melee DPS in the party or if the melee is one of those people who refuse to use anything until the last boss, which is I'd say 95% of my runs anyways. Ideally I will save it for a pull that will have more enemies than usual but if not then even three will do. The point is that I always leave it so that we definitely have a full bar for the final boss unless it's a low level run where you will never fill the second bar during the fight anyways. Generally speaking, you don't want to use it any later than at the beginning of the second boss fight, but at times even that can be cutting it close, especially in high level runs. I pretty much never use a one bar Limit Break on bosses since it seems much more beneficial to save it for a group pull.

As for having anyone hassle me over it.... no, no-one ever says anything.