Just ruling out some possible oversights:

You have the Goliath Worm actually equipped in your Ammo slot.
You don't have a macro or gearswap that is automatically de-equipping your Goliath Worm...