Exclusive: ‘I thought it was going to be a major part of the tour,’ E Street Band musician said