Prince Andrew will no longer be attending tomorrow's pre-Christmas lunch at Buckingham Palace.