Locals were left "shocked" when work started on the site last week.