This is the invisibility quest for the sylphs where you retrieve artifacts.

Im trying to do this quest at level 47 and I can get from flower to flower just fine without running out of time, but certain flowers are surrounded by level 41 or so mobs. If I get too close to them, like 5 yalms, they aggro me and pull me out of invisibility making me have to start all over.

Is this something that happens to you if your level 50, or is this just an oversight by the dev team concerning sub 50 characters doing dailies?

It's not a big deal as I will level up soon enough but this quest is all but impossible since several of the flowers are surrounded by mobs and its not like i can wait around till they leave or clear them all before hand.