Why do I have to re-set the portrait literally every time I play pvp or else I risk having the default portrait that isn't even in my portrait collection anymore?

Am I doing something wrong?

I set the portrait to my gear set even and it seems to have done nothing. The portrait just doesn't work unless I click 'use as instant portrait' every time I change so much as a ring on my gear.

Admittedly, a very minor issue, but boy is it annoying me.