The duty routinely tops surveys of the most hated tax despite the limited number of people it affects – just 4 per cent