There is no question of forgiving Karnataka Minister on derogatory remarks