‘Bake Off’ judge has forged an unlikely friendship with Lively, who he says is a big fan of the show