Many turned up to brave the cold waters, in trunks, shorts or fancy dress, watched by a huge crowd