I tried toggle myself briefly and for problems like this stopped using it. IMO it's toggle that's broken.

Instead I have it set so the hotbar only appears when pressing L2 or R2. I only press buttons when I need to and it doesn't clutter my screen.