But what's the point in a vulnerability debuff that only buffs yourself or jobs that don't need it? you may as well just remove the debuff and increase potencies at that point.

The point of those debuffs is to empower other jobs other than yourself. Not to empower yourself. It's to allow you to bring something useful to the group table as a DPS. Adding this to BRD removes part of DRG's utility attractiveness. BRD offers other group benefits besides this debuff.

Same reason BLM could come with a magic vulnerability debuff (in exchange for lower potencies), in order to be a more attractive group member and have some utility that is different to RDM or SMN.

Glass half empty.