The lack of specialist teachers is having a 'huge impact' on pupils' attainment.