It's specialization, and yes, if you really care to compare it to real life, someone that spends time on one skill 'will' slowly lose skill at other things. A carpenter who switches careers to construction work and later goes back to carpentry isn't going to be as good at it as they were when they left.

Further, you're exaggerating the loss. You 'occasionally' lose .01 points in a random other desynth craft while you're raising one and under 300 total. After 300, I imagine it's more severe, as it has to stay balanced, but it may well depend on how much you're raising one by.