The Republican congressman has urged the Harris campaign to ‘stick to the facts’