Would the spell speed on normal gear affect PoTD casting speed, though? I've never personally seen that. Could be that the WHM's that OP is referring to make liberal use of PoM and Swiftcast. Or that they have the same spell speed, just that the full animation doesn't show up on OP's screen due to latency.