He tried to claim that the phrase may have innocently been used while his brother was rapping in the street