That UI you made looks super clean, I love it! If anything, I'd do it so hovering over each color tells you how many of it you have, and if you don't have any it shows up as an empty circle to prevent confusion.