Fester damage is only affected by Bio 2/Miasma/Bio, your main three DoTs. When Fester is used when you have one of those DoTs on an enemy it will deal 100 potency. Two DoTs will lead to 200 potency and three DoTs will lead to 300 potency. If none of those DoTs are on an enemy and Fester is used it will deal absolutely nothing. It's crucial that all three of those effects are on an enemy at all times. Fester is your best burst damage tool.

As far as Ruin 2 goes it's a filler spell. You never want to spam this spell. Typically you will want to use this just before you're about to use Fester for some extra damage. The blind effect from Ruin 2 isn't crucial so don't worry about diminishing returns.

I'd recommend fooling around on the dummies found throughout the game. Practice rotations on those.