Honestly the best fix for this is just to restart the server every couple of hours that will clear out a big amount of afkers.