The colors assignments are randomly determined at the start of the fight. The first set of totems will be three of the four colors. Remember which color is missing. That's the Toad totem. Pick up the other totems one at a time until you find the Damage Up totem. Once you know which one is Damage Up, ignore the others. The colors will remain fixed for the rest of the fight, so if cyan is the Damage Up totem the first round, cyan will always be the Damage Up totem.
If your DPS is "too high" you'll skip the 3/4 set and go straight to a 4/4 set. In this case, you'll have to figure out which color is the Toad totem the old fashioned way.
When he turns you into a Toad, grab a Toad totem to return to normal. Grab all the Damage Up totems each round. Ignore the other two colors.