People have concerns about the libraries moving to different buildings