The town boasts fewer than 3,000 residents but sees people travel from afar to enjoy