So after doing some more testing, I've been seeing positive results by sacrificing even more direct hit for determination. So as of now I am prioritizing Crit > Det > DH.

I'd imagine we'd want DH to at least be a certain value, but I think stacking it at the expense of crit and maybe det is a detriment overall to DPS. As my DH lowers I'll see how it effects my DPS, there's only so much I can do with my current gear. But yeah, stacking crit has been the single best thing for my DPS so far.