The former minister was sacked over claims he said he wished a constituent would die