Frustrated locals in the UK's coldest village have told the chancellor and the prime minister to "get out of their ivory towers".