Even Historic England has said the building lacks 'fun and frivolity'