Zameer’s remarks drove away some voters from Congress: Yogeshwar