A friend of the Prince has warned that he will "look like an idiot" if he comes back to the UK.