I don't think it will matter, nope.
It might decrease the waiting time a little but you're never going to permanently remove that because it is to my knowledge a permanent buffer that exists regardless of the actual population already on the world. Happy to be wrong but I am fairly certain it's always going to exist.