Never spoke against people: Madhya Pradesh minister