Peter Dutton’s call to banish work-from-home rights for Australia’s 185,000-strong public servant workforce has been lashed as “lazy”.