Many Canadians agree with the prime minister that he should not be in the office