Harry is "annoyed" ahead of his return to the UK return despite reportedly refusing an offer to stay in Buckingham Palace.