The former president’s second son said the Duke of Sussex had ‘gone off the deep end and it’s sad to watch’