'The place of work is not important to me; it’s about the result,' Poilievre said when asked if he would order federal workers back to the office five days a week