The King’s brother had already withdrawn from festive celebrations at Sandringham