If you're still having issues while zoomed in, then you have too many objects around the garden plot that you can interact with. I suggest moving these objects, or the plot itself to a location that is more user friendly.

Minions and pets also make things annoying so just dismiss them while tending.