From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 93D61C000E for ; Thu, 4 Nov 2021 22:01:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7BC8F4014B for ; Thu, 4 Nov 2021 22:01:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: 1.527 X-Spam-Level: * X-Spam-Status: No, score=1.527 tagged_above=-999 required=5 tests=[BAYES_05=-0.5, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, PDS_OTHER_BAD_TLD=1.975, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=no autolearn_force=no Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m2fmfla-0Qce for ; Thu, 4 Nov 2021 22:01:26 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) by smtp2.osuosl.org (Postfix) with ESMTPS id DF65540114 for ; Thu, 4 Nov 2021 22:01:25 +0000 (UTC) Received: by mail-yb1-xb2f.google.com with SMTP id q74so17890264ybq.11 for ; Thu, 04 Nov 2021 15:01:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=WUN/q9027srP+Vv1BvPwzWs4wm1VZ7aY3X40i+c/rGY=; b=hIA1bVEhWw0QPPci4h7KYlhpp+cjk+BSUAxaZG2Vo6aXz09x2XTRV+ogJ2GBZ6Fayo OZqu4FoOZRYrbKcs3klsoCOio76bXUMTeTBrvaxl/FKsN1ArkipxSo1dIVSPCfilweHf PyjNaQgGe7StPvJ9RxDWyhnyF0zKeSdXp4RsiJKR6CVBrJn1V5k8kHDcICjUq28aRo0J xOQoyiNeU42bRO0AdZwiA46IO8tz/wTpN5PSzOYPgfYGuUbNYaFHeC9BP/GtNjIcj0Wc uvC16ycOCHh8Ns5+ySrUl8jWqCp3GEE4yHZLgSNWlzM3rn74TLe1pwYGTL5l2UMxJDtY g1lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=WUN/q9027srP+Vv1BvPwzWs4wm1VZ7aY3X40i+c/rGY=; b=mBAGQGQ5dlpJACFnqbGzVp0N/JCJA/q+CnXigsF48Ktz6u12bQOgtHA4dNw2Iywsp4 SL9yVCM+kMfE5cSSCUQjDhp8RFbJMrdT6Rkqtz5Jln0QaHha1E1cUtlaL2D5RLl6vAUQ hH/HlCv7q1GYo0C9J4ySA5yixL2NBms5F9YG54zjfT7T02OTMLWhSIelMp9UT8/wULNi bmacrrn7fiOueot77soyJvqUys3+0B0zYO40M4FDv+YOPTaUbX17WwrUJ0zga+13GujU p3NsbbKnbrH+gHBITOt4Uah0drpnimPQt/++5S0Y6GQKYxcETfH+L7lI+ELeoW6zDSmL drew== X-Gm-Message-State: AOAM531o1Meth6jdQ8rnr/Mnh4bMKRDZSHkiMuvYtPO0+BSlpeBBaP6N 4P97N8n6XRfSaAmPPp4qDTplSRQNLeuIM7BQs5VqyO5wGvA= X-Google-Smtp-Source: ABdhPJzdru6LeOiQGf+55Ix0kpPCT7ddwOnMsXqp4qGxwIk6LvGbSETD05iD+WuETdMu9DKOOdegpqgpOM8qgrdLECk= X-Received: by 2002:a25:bc83:: with SMTP id e3mr42810637ybk.255.1636063284525; Thu, 04 Nov 2021 15:01:24 -0700 (PDT) MIME-Version: 1.0 From: Olaoluwa Osuntokun Date: Thu, 4 Nov 2021 15:01:13 -0700 Message-ID: To: lnd@lightning.engineering, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev Content-Type: multipart/alternative; boundary="000000000000e77fd905cffdaa33" Subject: [bitcoin-dev] Neutrino, Taproot, and The Evolution of BiPs 157/158 X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Nov 2021 22:01:27 -0000 --000000000000e77fd905cffdaa33 Content-Type: text/plain; charset="UTF-8" Hi y'all, If you're an active user of neutrino [8], then you probably heard about what went down over the past week or so on testnet as related to Taproot. First, i just wanted to reassure everyone that nothing is fundamentally broken with BIP 157/158 as it relates to taproot, and we already have a mitigation patch in place for the issue we encountered. The rest of this mail is structured in a FAQ style to make it easy to skim and extract the information that may be relevant to the reader. ## What happened on testnet? Neutrino nodes on testnet rejected a filter (thinking it was invalid) due to this transaction spending a taproot input [1]. This was due to a faulty heuristics in the neutrino _client code_ (not part of the protocol) that attempted to verify the contents of a filter more completely. In retrospect, the heuristic in question wasn't full proof, as it attempted to derive the _pk script_ of a transaction based on its input witness/sigScript. This worked pretty well in the context of segwit v0, but it isn't possible to exhaustively do as we don't know what future spends will look like. ## Is neutrino broken? No, the client side is fine, and the protocol is fine. The problematic heuristic has been removed in this PR [2], which will be included in lnd 0.14, and has been tagged with neutrino 0.13 [3]. To dig into _why_ we attempted to use such a heuristic, we'll need to revisit how BIP 158 works briefly. For each block, we insert the `pkScript`s of all the outputs, and also the prev out's pkScript (the script being spent) as well. This lets the filter compress script re-use in both inputs and outputs, and also makes it possible to implement some protocols in a more light-client friendly manner (for example Loop uses this to has the client watch HTLC _scripts_ being spent, as it doesn't necessarily know the txid/outpoint). The one issue with this, is that while clients can ensure that all the `pkScripts` of outputs have been inserted, they can't do the same for the inputs (which is why we added that heuristic in the client code). Luckily we know how to properly fix this at the protocol level, more on that below. ## How can I make sure my neutrino clients handle the Taproot upgrade on mainnet smoothly? Upgrade to 0.14 (assuming it's out in time), or apply this small patch [4]. The patch just demotes an error case to a warning message, so anyone running a fork should be able to easily apply the fix. Alongside, optionally extend these filter header guides [7]. We're looking into some intermediate ground where we can verify the scripts that we know are relevant to the node. ## How will BIP 158/157 evolve post taproot? In terms of adding more taproot specific functionality, I've had a number of items in my laundry list such as: * creating new segwit-only filters with re-parameterized fp rates (also examine other filter types such as pure outpoints, etc) * creating filters that include witness data to allow matching on internal/external keys, the control block, merkle root, annex, etc * add a new protocol extension to btcd (with a corresponding BIP) to allow notes to fetch block undo data (as described here [5]) to fully verify fetched filters or a node needs to reconcile conflicting filters * new filters that span across multiple blocks as previously worked on by Kalle Alm (couldn't find a link to his PR when typing this...) Make further progress towards a proposal that allows filters to be committed either as a soft-fork, or a "velvet fork" [6] where miners optionally commit to the past filter header chain. -- Laolu [1]: https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8 [2]: https://github.com/lightninglabs/neutrino/pull/234 [3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0 [4]: https://github.com/lightninglabs/neutrino/pull/234/files [5]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html [6]: https://eprint.iacr.org/2018/087 [7]: https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15 [8]: https://github.com/lightninglabs/neutrino --000000000000e77fd905cffdaa33 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi y'all,

If you're an active user of neutr= ino [8], then you probably heard about what
went down over the past week= or so on testnet as related to Taproot. First,
i just wanted to reassur= e everyone that nothing is fundamentally broken with
BIP 157/158 as it r= elates to taproot, and we already have a mitigation patch
in place for t= he issue we encountered.

The rest of this mail is structured in a FA= Q style to make it easy to skim
and extract the information that may be = relevant to the reader.

## What happened on testnet?

Neutrino= nodes on testnet rejected a filter (thinking it was invalid) due
to thi= s transaction spending a taproot input [1]. This was due to a faulty
heu= ristics in the neutrino _client code_ (not part of the protocol) that
at= tempted to verify the contents of a filter more completely.

In retr= ospect, the heuristic in question wasn't full proof, as it attemptedto derive the _pk script_ of a transaction based on its input
witness/s= igScript. This worked pretty well in the context of segwit v0, but
it is= n't possible to exhaustively do as we don't know what future spends=
will look like.

## Is neutrino broken?

No, the client sid= e is fine, and the protocol is fine.

The problematic heuristic has b= een removed in this PR [2], which will be
included in lnd 0.14, and has = been tagged with neutrino 0.13 [3].

To dig into _why_ we attempted t= o use such a heuristic, we'll need to
revisit how BIP 158 works brie= fly. For each block, we insert the `pkScript`s
of all the outputs, and a= lso the prev out's pkScript (the script being
spent) as well. This l= ets the filter compress script re-use in both inputs
and outputs, and al= so makes it possible to implement some protocols in a
more light-client = friendly manner (for example Loop uses this to has the
client watch HTLC= _scripts_ being spent, as it doesn't necessarily know the
txid/outp= oint).

The one issue with this, is that while clients can ensure tha= t all the
`pkScripts` of outputs have been inserted, they can't do t= he same for the
inputs (which is why we added that heuristic in the clie= nt code). Luckily we
know how to properly fix this at the protocol level= , more on that below.

## How can I make sure my neutrino clients han= dle the Taproot upgrade on mainnet smoothly?

Upgrade to 0.14 (assumi= ng it's out in time), or apply this small patch [4].
The patch just = demotes an error case to a warning message, so anyone running
a fork sho= uld be able to easily apply the fix.

Alongside, optionally extend th= ese filter header guides [7].

We're looking into some intermedia= te ground where we can verify the scripts
that we know are relevant to t= he node.

## How will BIP 158/157 evolve post taproot?

In term= s of adding more taproot specific functionality, I've had a number ofitems in my laundry list such as:

=C2=A0 * creating new segwit-onl= y filters with re-parameterized fp rates (also
=C2=A0 =C2=A0 examine oth= er filter types such as pure outpoints, etc)

=C2=A0 * creating filte= rs that include witness data to allow matching on
=C2=A0 =C2=A0 internal= /external keys, the control block, merkle root, annex, etc

=C2=A0 * = add a new protocol extension to btcd (with a corresponding BIP) to
=C2= =A0 =C2=A0 allow notes to fetch block undo data (as described here [5]) to = fully
=C2=A0 =C2=A0 verify fetched filters or a node needs to reconcile = conflicting filters

=C2=A0 * new filters that span across multiple b= locks as previously worked on by
=C2=A0 =C2=A0 Kalle Alm (couldn't f= ind a link to his PR when typing this...)

Make further progress towa= rds a proposal that allows filters to be committed
either as a soft-fork= , or a "velvet fork" [6] where miners optionally commit to
the= past filter header chain.


-- Laolu

[1]: https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b8= 10c53078773fb768acd2be1398e3f561cc3f19fb8
[2]: https://github.com/lightninglabs/= neutrino/pull/234
[3]: https://github.com/lightninglabs/neutrino/rel= eases/tag/v0.13.0
[4]: https://github.com/lightninglabs/neutrino/pull/234/= files
[5]: https://lists.linuxfoundation.org/pipe= rmail/bitcoin-dev/2019-February/016649.html
[6]: https://eprint.iacr.org/2018/087
[7]: https://github.com/ligh= tninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/= filtercontrol.go#L15
[8]: https://github.com/lightninglabs/neutrino
--000000000000e77fd905cffdaa33--