The Business Secretary's comments have prompted a backlash from Brexiteers.