The comedian said he is not going to do another series