Plans for the site of the former market hall would also see the town's library relocated