A hidden gem that is believed to have once drawn pilgrims, the sick and the curious