Keycards and tank being afk is a problem, the other 2 are entirely on your group. The colossus is intended to bug out when you focus too much on it rather than killing the adds, the adds are a mechanic of the fight that are meant to be killed and trying to bypass them to just save your group those extra 5-10secs that you oh so need is just asking for the various things in the encounter to bug out because you tried bypassing an intended mechanic.