The drummer looked back at how the band had to make swift exits from gigs