The now empty town centre site was supposed to become home to a new market and homes, but these plans were scrapped