Joel said it was the “darkest hours of our lives”