Wanderer's Palace is unnecessary if you can clear Amdapor Keep. However, you get the materials to craft some relic weapons and pre-Ifrit weapons from Wanderer's Palace.

I'm also having serious trouble with Garuda because of the lack of item level requirement. I'm getting groups full of DPS in AF and white items who can't pull enough DPS to kill the sisters before they shriek.