The former Victorian CFMEU boss declared he would never vote for Anthony Albanese but praised state Labor.