SNP ministers urged to take ‘immediate action’ to reverse ‘worrying’ downward trend