Planning chiefs said the additions would "undermine the grandeur, detail, consistent classical design" of the A-listed building.