From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0C192C016F for ; Sat, 30 May 2020 14:16:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id EBED587E95 for ; Sat, 30 May 2020 14:16:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ry+hTVp5H71j for ; Sat, 30 May 2020 14:16:28 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id B2BEE87DA2 for ; Sat, 30 May 2020 14:16:27 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id m18so2737391ljo.5 for ; Sat, 30 May 2020 07:16:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shesek.info; s=shesek; h=mime-version:from:date:message-id:subject:to; bh=6b5G/F3ua9mC6M6u7c+zYrpTzTsB1/GM5lKIa28POec=; b=vyzsZu8IrKCEMWo/NPUOSzHUHj/KLQ0Y5GsK3nhS1C3qXP+GdZPUlAkP8fkC60upNk uYyj18731uwKFVLATQwfKVsfKx7C7IjYHkM8tyf8Io5K4TIIp9Y/wWThuL+prljqL7VW IH/VqOHpasafSzw4FyWZWoxJ2hXiPA3iLIFL0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=6b5G/F3ua9mC6M6u7c+zYrpTzTsB1/GM5lKIa28POec=; b=gjVGi///yL44BUwz/oIf0O4JdlELX2QrA3GtskxfZb7dcDZ7O6/+wTQe/5G0gbU1Hn md8cc5uvu06pGsYI/E2hTXlAfISUBXFMHg+6Xzm1jkM3IrZFn2+JMVK0f5Q7gWAfNGI6 OOSb+R9fBsgIDhtyWTh0IdJFxMs+KTmVD3fBT4IFaicKsH0TopcaMucoeH1p8N2SVDXw eFf9dbtZqceHW3/zSDUouIQSfNosdGFRDD0hUyNG67ATvAD7oRvNYVdG7Swwau+HJAsw yRrhiai758wWI2l2RPFx8tyDO9ksTw6kuPt75rfLWcVKFzUoPT8AsIjAnBrdQU4XTbgi E+Jg== X-Gm-Message-State: AOAM532Iff/3Nd1vF11Esmq4P0oj5CAAjq0AwpO1kur34Ry9kjtq2baA tmsODTDng4VmU4BdDI1Q6VD6M4dX78R9GxV81vhc57Nby5Y= X-Google-Smtp-Source: ABdhPJxhc0jNTCkF1ahV5NDxE8XlG+i5rvouCOa7ANIv/X3K28O20Zwk6wgjS3tK2Hgjchgyq4Bhov5qxOlRv1vFFC0= X-Received: by 2002:a2e:92d6:: with SMTP id k22mr3276380ljh.89.1590848185423; Sat, 30 May 2020 07:16:25 -0700 (PDT) MIME-Version: 1.0 From: Nadav Ivgi Date: Sat, 30 May 2020 17:16:14 +0300 Message-ID: To: bitcoin-dev@lists.linuxfoundation.org Content-Type: multipart/alternative; boundary="000000000000fbef4205a6de345a" X-Mailman-Approved-At: Sat, 30 May 2020 14:19:55 +0000 Subject: [bitcoin-dev] Announcing Bitcoin Wallet Tracker X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 May 2020 14:16:29 -0000 --000000000000fbef4205a6de345a Content-Type: text/plain; charset="UTF-8" Hi all, I recently released bwt [0], an HD wallet indexer implemented in Rust, using a model similar to that of Electrum Personal Server. It uses the bitcoind wallet functionality to do the heavy lifting and builds additional indexes on top of that, which can be queried using the Electrum RPC protocol, as well as a more modern, developer-friendly HTTP REST API. The electrum server can also be used as an electrum plugin [1], which integrates the server straight into the electrum client. From the user's perspective, this allows connecting electrum directly to a full node. The HTTP API is my take on a modern design for a wallet tracking API aimed at app developers. Some use-cases include using it as a backend for wallets (similarly to Samuari's Dojo) or to track deposits to a watch-only xpub (similarly to BTCPay's NBXplorer). Compared to using the bitcoind RPC directly, bwt provides: - The ability to track an xpub and automatically have new addresses derived and imported as needed, according to the gap limit. - Two additional indexes, one for looking up the transaction history of addresses, and another one for looking up txo spends (a map of funding_txid:vout => spending_txid:vin). - Real-time updates using Server-Sent Events [2] (a long-lived streaming HTTP connection) or Web Hooks [3] (an HTTP request sent to a configured URL). The updates being sent [4] directly provide information about the funded and spent wallet txos, instead of the client figuring it out from the tx. - Some API conveniences and simplifications, like including key origin information directly alongside inputs/outputs [5], the ability to specify key origins in place of addresses (eg. GET /hd/15cb9edc/8/utxos), a compact history format [6], and an easy way to catch-up with missed events [7]. Unless explicitly asked for, the API omits information about non-wallet inputs/outputs and protocol-level details like scriptsig and witnesses, which are typically not needed for higher-level app development. The indexer is designed in a way that minimizes RPC requests to bitcoind. By using labels to store key origin information, it is able to index incoming transactions using the information available from `listtransactions` alone (plus 3 extra rpc requests that don't grow with the number of transactions), but requires 1 additional rpc call per outgoing transaction (to learn which prevouts were spent). It can index 10k incoming txs in under a second, or a mixture of 5k/5k in under 5 seconds. The index is currently entirely in- memory and does not get persisted. The indexer logic can be seen in [8]. One major item on the roadmap that I'm hoping to tackle soon is support for output script descriptors. If anyone is interested in contributing, the README has some useful developer resources [9] and a handy script for setting up a development environment. This is an early alpha release, recommended for use with testnet/regtest. All feedback welcome! Cheers, Nadav [0] https://github.com/shesek/bwt [1] https://github.com/shesek/bwt#electrum-plugin [2] https://github.com/shesek/bwt#server-sent-events [3] https://github.com/shesek/bwt#web-hooks [4] https://github.com/shesek/bwt#event-categories [5] https://github.com/shesek/bwt#wallet-transaction-format [6] https://github.com/shesek/bwt#get-txssinceblock-heightcompact [7] https://github.com/shesek/bwt#catching-up-with-missed-events--re-org-detection [8] https://github.com/shesek/bwt/blob/master/src/indexer.rs (sync_transactions and load_transactions_since) [9] https://github.com/shesek/bwt#developing --000000000000fbef4205a6de345a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi all,

I recently released bwt [0], an HD wallet i= ndexer implemented in Rust, using
a model similar to that of Electrum Pe= rsonal Server.

It uses the bitcoind wallet functionality to do the h= eavy lifting and builds
additional indexes on top of that, which can be = queried using the Electrum
RPC protocol, as well as a more modern, devel= oper-friendly HTTP REST API.

The electrum server can also be used as= an electrum plugin [1], which
integrates the server straight into the e= lectrum client. From the user's
perspective, this allows connecting = electrum directly to a full node.

The HTTP API is my take on a moder= n design for a wallet tracking API aimed
at app developers. Some use-cas= es include using it as a backend for wallets
(similarly to Samuari's= Dojo) or to track deposits to a watch-only xpub
(similarly to BTCPay= 9;s NBXplorer).

Compared to using the bitcoind RPC directly, bwt pro= vides:

- The ability to track an xpub and automatically have new add= resses derived
=C2=A0 and imported as needed, according to the gap limit= .

- Two additional indexes, one for looking up the transaction histo= ry of
=C2=A0 addresses, and another one for looking up txo spends (a map= of
=C2=A0 funding_txid:vout =3D> spending_txid:vin).

- Real-t= ime updates using Server-Sent Events [2] (a long-lived streaming HTTP
= =C2=A0 connection) or Web Hooks [3] (an HTTP request sent to a configured U= RL).
=C2=A0 The updates being sent [4] directly provide information abou= t the funded
=C2=A0 and spent wallet txos, instead of the client figurin= g it out from the tx.

- Some API conveniences and simplifications, l= ike including key origin
=C2=A0 information directly alongside inputs/ou= tputs [5], the ability to specify
=C2=A0 key origins in place of address= es (eg. GET /hd/15cb9edc/8/utxos), a compact
=C2=A0 history format [6], = and an easy way to catch-up with missed events [7].
=C2=A0 Unless explic= itly asked for, the API omits information about non-wallet
=C2=A0 inputs= /outputs and protocol-level details like scriptsig and witnesses,
=C2=A0= which are typically not needed for higher-level app development.

Th= e indexer is designed in a way that minimizes RPC requests to bitcoind. By<= br>using labels to store key origin information, it is able to index incomi= ng
transactions using the information available from `listtransactions` = alone
(plus 3 extra rpc requests that don't grow with the number of = transactions),
but requires 1 additional rpc call per outgoing transacti= on (to learn which
prevouts were spent). It can index 10k incoming txs i= n under a second, or a
mixture of 5k/5k in under 5 seconds. The index is= currently entirely in-
memory and does not get persisted. The indexer l= ogic can be seen in [8].

One major item on the roadmap that I'm = hoping to tackle soon is support for
output script descriptors.

I= f anyone is interested in contributing, the README has some useful develope= r
resources [9] and a handy script for setting up a development environm= ent.

This is an early alpha release, recommended for use with testne= t/regtest.

All feedback welcome!

Cheers,
Nadav

[0] = https://github.com/shesek/bwt=
[1] https://g= ithub.com/shesek/bwt#electrum-plugin
[2] https://github.com/shesek/bwt#server-sent= -events
[3] http= s://github.com/shesek/bwt#web-hooks
[4] https://github.com/shesek/bwt#event-categori= es
[5] https://github.com/shesek/bwt#wallet-transaction-format
[6] <= a href=3D"https://github.com/shesek/bwt#get-txssinceblock-heightcompact">ht= tps://github.com/shesek/bwt#get-txssinceblock-heightcompact
[7] https://github.com/shesek/bwt#catching-up-with-missed-events--r= e-org-detection
[8] https://github.com/shesek/bwt/blob/master/src/indexer.= rs (sync_transactions and load_transactions_since)
[9] https://github.com/shesek/bwt#devel= oping
--000000000000fbef4205a6de345a--