The comedian shared the reason for his decision