The closure of a leisure centre has been described as a “huge loss” to the community.