The late Queen Elizabeth II made a harsh remark about a former Prime Minister at a family gathering in Balmoral before her death.