Prince Andrew and his ex-wife Sarah Ferguson have opted not to attend to avoid further embarrassment to the royal family