Poilievre said Carney had acted as an 'unreliable boyfriend' while leading the central bank from 2013 to 2020