This allegedly "wasn't the first time" the pair considered calling it quits.