Specialist bank has set aside £165mn after sector-wide regulatory probe and court ruling on historic commissions