Party workers say Opposition is conspiring against Chief Minister