The trading firm is pushing frontiers in markets once dominated by banks — but is not yet regulated like one