Actually, some of the German army were conscripts of lands they took over (yay IRL parallels). How do you feel about say the Czech who were forced into the army against their will (do it or get shot)? Funny enough, if you watch Saving Private Ryan... it subtly touches on that point. After the Americans take over the beach on D-Day, there are two surrendering "Germans", with their hands up and are pleading to surrender in "German". Except they weren't, they were speaking Czech: "Please don't shoot me, I am not German, I am Czech, I didn't kill anyone!". The US soldiers didn't understand, and shot them on sight making jokes.
https://www.youtube.com/watch?v=VCbW7Gkgu2U (Scene)
So let's bring it back to the Garleans; It's been established that most of the soldiers (especially the magic users since Garleans can't use magic) are from captured lands, specifically Ala Mahigo. A survivor of an ambushed battalion even says so in La Noscea, saying they weren't even Garlean and were forced into the war. He just wanted to go home and see their family again. I'd say a lot of these people are victims from both sides.