‘It wasn’t the right way to engage in that conversation,’ actor said