School grounds should be opened up for public use after hours and on weekends, Victoria’s peak infrastructure body says.