A limit would be bad. Killing stronger mobs wouldn't be rewarding then anymore. Instead they should reduce the party bonus a bit and split the experience between parties depending on damage dealt for each enemy.