A gardening support group set up to support survivors of abuse and illness has been “destroyed” by Storm Bert.