Financial regulator says it will scrap new public interest test after backlash from the City and government officials