It was not always plain sailing for the band in the early years