The US president is said to have remarked that his own royal invitation now feels ‘less special’