The warning comes on the back of a new survey from Halifax