A watchdog criticised the health board for failures in its management of a "fragile" and "unsustainable" medical service at Dr Gray's Hospital in Elgin.