Thousands of road users are at risk of having their driving licence taken away on safety grounds.