Rajugouda made some insulting remarks against the party at the meeting, he says