A fifth (21%) of savers refuse to put their funds in accounts with specialist or challenger banks.