Britain cannot rely on overseas construction workers to deliver the Government’s target of building 1.5 million homes, the housing minister has said.