Speaking at Vanity Fair's New Establishment Summit, the legendary architect said his "final conclusion" surprised him, too.