The annual fee faced years of scrutiny under the Conservative government