The prime minister isn't ready to take a walk in the leaves