Travellers have set up camp near an iconic Swindon site.