I'm going to be a bit contrary and say that if everyone is topped off, and you're standing there idling, I don't really consider it playing to the best of your ability.

You're being lazy.

It will annoy me. It's the same as if the tank spams overpower until he has threat and then just goes AFK for half the pull. Or the summoner spreads dots and then just idles, because it's 'a dot job'. Or if you're playing an archer rather than a bard because 'it's my character concept'. Dungeons are easy enough that everyone can do this, but it's terribly dysfunctional.

Now, it's your game and your sub and your money. That it annoys me doesn't give me the right to do anything about it to you. But maybe playing a team-based game while not putting in a team effort is the wrong mentality to bring into things.