A former Scottish Government cabinet secretary has branded the refusal to speed up work on the dualling of the A9 road as “deeply disappointing”.