‘Physically and mentally it takes its toll,’ the comedian said