"This is one of those unfortunate decisions that had to be made and we move on," Lionel Richie told PEOPLE after Sunday's semi-final episode
