The link between poor quality homes and worklessness might be stronger than the government has bargained for.