Put your own buffs somewhere where you can see them better. I have my own buffs, enemy debuffs and my own debuffs fairly close together, one row for each and close to other important elements so I don't need to constantly look up, down, back up again etc.

Open the HUD menu and click on the element for your buffs. You can split it into three groups, which I would recommend because it displays less interesting buffs like buff food in a seperate element. Group 1 (Status Info Enhancements) is the one that displays your own buffs. I increased it to 180% iirc and put it somewhere near my hotbars where my minimap, HP/MP and cast bar is. So whenever I look slightly down I see my cooldowns, my buffs, my position on the minimap, my HP/ MP and cast progress at a glance.
Buff icons are tiny on 100% so I recommend setting them at least to 160% so you can easily see the number of the timer through effects.