It's hard to say with certainty what your problem is without knowing your start color and the rest of your feeding sequence, and I'm basing this answer just on seeing four plums fed in a row.

Each fruit gives or subtracts five points from each primary color. Pineapples, for example, give red+5, green+5, blue-5; while plums give red-5, green+5, blue+5. From the last color change, four plums would give red-20, green+20, blue+20. There are two possibilities that would explain why you don't see another color change message by then.

A) You might have reached the max value of green, the max value of blue, or the minimum value of red. Reaching the extremes will lead to subsequent fruits having a smaller effect, since only one or two of the color values will change with each feeding.

B) You might not have changed the values enough to move into a new color category.

Note that the two possibilities are not mutually exclusive, and A would increase the likelihood of B. To avoid this, and get the full value out of each fruit, you should alternate fruits to keep values away from the extremes. The popular calculator has a button, "View feeding order", which will determine an order that prevents the RGB values from bottoming/topping out.