Impact of levies could be worse than expected, says former Bank of Canada governor