The thing is, after you defeat your shadow and accept it, the soldier at Whitebrim mentions giving "your friend" a proper burial. From that line and the fact that people do seem to at least acknowledge Fray's presence when s/he speaks to them, I presumed "Fray" was your shadow possessing the body of Fray the dead Dark Knight to have a corporeal form. Sidurgu, the Auri Dark Knight who offers to train you after the 30-50 questline, will talk about him if you ask, and though he hasn't much to say he confirms that Fray was a real person.
It's kind of screwy, but that's what I patched together.
Ehh, I felt it was something closer to a Persona 4 story arc. "Fray" is you - or rather, your shadow: the repressed parts of your psyche, especially you anger and resentment about having to play the hero all the time and getting nothing but praise (at best) in return. In the end you have to accept that "Fray" is a part of you, much like the characters of P4, even though the entire population of Whitebrim saw your dark and ugly side.