Mallikarjun Kharge said PM Modi is afraid that if he allows a caste census then all sections of society will demand their share.