Savers who switch to part-time work could find themselves over £58,000 poorer when they retire.