Building society has handed out millions of pounds to customers over the last two years