Prime Minister Mark Carney made a quick stop in London on Wednesday but didn't take any questions from reporters