They will skip the annual event hosted by King Charles