‘I’m not against work from home. I’m against where it doesn't work’ he said while justifying his in-office mandate