Residents are pushing back against the use of a former horse riding school as a scaffolding yard.