This little town draws tens of thousands of visitors every summer but it is a gorgeous place for a weekend away whatever the season