Damage wise yes, but the HP you'd lose would be quite substantial,, and as our gear gets stronger and stronger, the amount of HP you'd be missing out on will only get larger to the point where it's a huge detriment to not have it.
Plus there's enough variety in Vit accessories that it's perfectly possible to avoid parry.
