Conservatives said closing the three museums would be "incredibly daft"