That's correct. He isn't to blame for what happened to him, though it wouldn't have been easy for either Rosa or Cecil to deal with the aftermath - and he felt extremely guilty about it all.
Elidibus isn't a friend, though that doesn't mean that the story couldn't play out in a way to have him become an ally in the future. Perhaps there will be revelations regarding Zodiark that will shake even his faith. Perhaps there will be revelations about Hydaelyn that will shake the faith of the protagonists. There's a lot that could happen and I don't see it as an impossibility for Elidibus to change his tune later on.