Residents in an historic UK market town with a rich history say they're terrified by what's going on.