A man has admitted to killing a baby by neglect in "extremely distressing" circumstances, with the boy left unattended for at least 21 hours.