‘There were times when we were shooting where I was like, ‘I hate doing this,’’ actor said