The structure has been deemed "at risk" by Historic England