A popular New Year's Day swim has been cancelled ahead of bad weather forecast to hit the area.