The couple said they feel as though they have been 'left'