The former Gardeners' World presenter opened up about his family life