So I'm using the spearfishing gig in the Ruby Sea. Trying to catch the swordtip for some crafting. I start and catch a Little Dragonfish. I then use the Veteran Trade skill and get the message "little dragonfish are aware of your presence and have left the area". Next catch is a wentlefish but the next one is a little dragonfish.

Is this skill broken or am I doing something wrong?