Residents have been warned it’s “not safe to return or leave” amid a bushfire in Victoria’s south.