‘It’s been unfortunate as it’s been so much fun,’ said the comedian