‘I was kinda mad at him,’ said the actor