A few remnants of the complex can be seen today after the site was sold hundreds of years ago