He knew where your house is and got there that way. Everything you said about the FL is true but people you remove from your FL can't teleport to your estate or FC the options don't show. Thats an easy way to tell if someone removed your from their FL if you can't teleport to either their house or FC.

That being said the FL is flawed and ancient. It needs a huge rework.