‘We have a big family now. We have 22 grandchildren and great-grandchildren, 38 of us in all,’ Carter told CNN in 2015