The former Prime Minister refused to 'stop banging on about his book'