One of the walkers 'felt scared' at times during their four-day trip due to 'anti-English' attitude