One resident said their boundary wall has been knocked over 28 times in 40 years by speeding vehicles