An ex-council leader said he was verbally abused after announcing his defection from the Conservatives to Reform UK.