‘I thought that was a safe space to do that, weirdly,’ actor said of controversial moment