Other than yourself, this sort of thing is hardest on those who have known you longest. It is so easy to associate a person with arbitrary features, and when those features seem to change, it can seem as if you never knew them at all.
this is a mistake, of course, but there is a fine line between surprise, fear, and rage.