It's multiplicative as was stated above, meaning that the second example you gave is correct, where Rampart being used while in Shield Oath would do 20% damage reduction based on the 80% damage being taken due to ShO, resulting in (80x0.2)+20=36% damage reduction.

A little annoying that because PLD's tank stance mitigation is % damage reduction based, it reduces the effectiveness of the job's main defensive CD's, Rampart and Sentinel.