The building was described as "underutilised" amid a lack of demand for office space in the town