Matt Lauer Remains 'Withdrawn' 5 Years After 'Today' Exit: 'Talking to People from His Past Is Painful'

A source tells PEOPLE the former morning show anchor's "level of trust has diminished with a lot of people he considered friends" since he was fired over multiple allegations of sexual misconduct