Lord Peter Mandelson suffered an early exit in the race for Oxford University chancellor.