Senior Conservative admits scrapping the quango was "probably" something his party should have done