He does seem to look away, but what is the Japanese version of that part of the script?

And in any case, if it is accurate that he's ashamed to look you in the eye while saying that, how would that suggest "brothers" any more than "close friends"?