It's just three miles from a much more famous, busier town - but has all of the charm without the crowds