The One show star has opened up about his relationship with his famous dad