‘I don’t talk about it, but I do. I have to worry about family. I have to worry about everybody,’ former president says