Lilli and Scott missed a few Christmases together when she was growing up