Their relationship began showing cracks a few years ago