The billionaire made the gloomy comments about the economy after Harvey Norman's annual shareholder meeting on Wednesday.