As you've said youre not a raider, I wouldnt be concerned with clipping while double weaving.
Raiding is about optimising every second you have, and losing a GCD has a significant effect. Outside of raiding however, youre much more likely to reap the benefits of having quicker cast times, giving you the freedom to react to things and make decisions on the fly quicker.