It's good they didnt fight right away IMO. Battles are better when there's a build up towards it. We know the battle is going to happen and the players want it to happen, so now there needs to be a reason for your character to want to face them. Then the payoff will be worth the hype (assuming it's done right).