Sources close to Prince Andrew have revealed he knows there is 'no way' for him to curry favour with the public