Shadow home secretary Chris Philp said millions of adults not in jobs ‘need to make a contribution’