The stunning Victorian seaside town is worth a visit whatever the season