"What happened in that building is historic" – Campaigners and councillors have voiced dismay over Eastney Swimming Pool demolition.