I believe there's no penalty for swapping allegiance, the only thing it affects is where you turn in items for daily GC seals. Also if you have any GC restricted item(ie: Maelstrom armour/weapons) you will not be able to use anymore if you switch to another GC.