The TV gardener and presenter was asked which plant he would happily never see again