[bitcoin-dev] Neutrino, Taproot, and The Evolution of BiPs 157/158

[bitcoin-dev] Neutrino, Taproot, and The Evolution of BiPs 157/158

[Olaoluwa Osuntokun]

[-- Attachment #1: Type: text/plain, Size: 4219 bytes --]

Hi y'all,

If you're an active user of neutrino [8], then you probably heard about what
went down over the past week or so on testnet as related to Taproot. First,
i just wanted to reassure everyone that nothing is fundamentally broken with
BIP 157/158 as it relates to taproot, and we already have a mitigation patch
in place for the issue we encountered.

The rest of this mail is structured in a FAQ style to make it easy to skim
and extract the information that may be relevant to the reader.

## What happened on testnet?

Neutrino nodes on testnet rejected a filter (thinking it was invalid) due
to this transaction spending a taproot input [1]. This was due to a faulty
heuristics in the neutrino _client code_ (not part of the protocol) that
attempted to verify the contents of a filter more completely.

In retrospect, the heuristic in question wasn't full proof, as it attempted
to derive the _pk script_ of a transaction based on its input
witness/sigScript. This worked pretty well in the context of segwit v0, but
it isn't possible to exhaustively do as we don't know what future spends
will look like.

## Is neutrino broken?

No, the client side is fine, and the protocol is fine.

The problematic heuristic has been removed in this PR [2], which will be
included in lnd 0.14, and has been tagged with neutrino 0.13 [3].

To dig into _why_ we attempted to use such a heuristic, we'll need to
revisit how BIP 158 works briefly. For each block, we insert the `pkScript`s
of all the outputs, and also the prev out's pkScript (the script being
spent) as well. This lets the filter compress script re-use in both inputs
and outputs, and also makes it possible to implement some protocols in a
more light-client friendly manner (for example Loop uses this to has the
client watch HTLC _scripts_ being spent, as it doesn't necessarily know the
txid/outpoint).

The one issue with this, is that while clients can ensure that all the
`pkScripts` of outputs have been inserted, they can't do the same for the
inputs (which is why we added that heuristic in the client code). Luckily we
know how to properly fix this at the protocol level, more on that below.

## How can I make sure my neutrino clients handle the Taproot upgrade on
mainnet smoothly?

Upgrade to 0.14 (assuming it's out in time), or apply this small patch [4].
The patch just demotes an error case to a warning message, so anyone running
a fork should be able to easily apply the fix.

Alongside, optionally extend these filter header guides [7].

We're looking into some intermediate ground where we can verify the scripts
that we know are relevant to the node.

## How will BIP 158/157 evolve post taproot?

In terms of adding more taproot specific functionality, I've had a number of
items in my laundry list such as:

  * creating new segwit-only filters with re-parameterized fp rates (also
    examine other filter types such as pure outpoints, etc)

  * creating filters that include witness data to allow matching on
    internal/external keys, the control block, merkle root, annex, etc

  * add a new protocol extension to btcd (with a corresponding BIP) to
    allow notes to fetch block undo data (as described here [5]) to fully
    verify fetched filters or a node needs to reconcile conflicting filters

  * new filters that span across multiple blocks as previously worked on by
    Kalle Alm (couldn't find a link to his PR when typing this...)

Make further progress towards a proposal that allows filters to be committed
either as a soft-fork, or a "velvet fork" [6] where miners optionally
commit to
the past filter header chain.


-- Laolu

[1]:
https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8
[2]: https://github.com/lightninglabs/neutrino/pull/234
[3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0
[4]: https://github.com/lightninglabs/neutrino/pull/234/files
[5]:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html
[6]: https://eprint.iacr.org/2018/087
[7]:
https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15
[8]: https://github.com/lightninglabs/neutrino

[-- Attachment #2: Type: text/html, Size: 5243 bytes --]

Re: [bitcoin-dev] Neutrino, Taproot, and The Evolution of BiPs 157/158

[Olaoluwa Osuntokun]

[-- Attachment #1: Type: text/plain, Size: 4808 bytes --]

> In terms of adding more taproot specific functionality, I've had a number
of
> items in my laundry list such as:

Forgot to add this other item (also the list wasn't meant to be only tapoot
stuff):
  * reviving old projects to include a micropayment-for-data layer to
    incentivize nodes to serve the filters and other data

On Thu, Nov 4, 2021 at 3:01 PM Olaoluwa Osuntokun <laolu32@gmail.com> wrote:

> Hi y'all,
>
> If you're an active user of neutrino [8], then you probably heard about
> what
> went down over the past week or so on testnet as related to Taproot. First,
> i just wanted to reassure everyone that nothing is fundamentally broken
> with
> BIP 157/158 as it relates to taproot, and we already have a mitigation
> patch
> in place for the issue we encountered.
>
> The rest of this mail is structured in a FAQ style to make it easy to skim
> and extract the information that may be relevant to the reader.
>
> ## What happened on testnet?
>
> Neutrino nodes on testnet rejected a filter (thinking it was invalid) due
> to this transaction spending a taproot input [1]. This was due to a faulty
> heuristics in the neutrino _client code_ (not part of the protocol) that
> attempted to verify the contents of a filter more completely.
>
> In retrospect, the heuristic in question wasn't full proof, as it attempted
> to derive the _pk script_ of a transaction based on its input
> witness/sigScript. This worked pretty well in the context of segwit v0, but
> it isn't possible to exhaustively do as we don't know what future spends
> will look like.
>
> ## Is neutrino broken?
>
> No, the client side is fine, and the protocol is fine.
>
> The problematic heuristic has been removed in this PR [2], which will be
> included in lnd 0.14, and has been tagged with neutrino 0.13 [3].
>
> To dig into _why_ we attempted to use such a heuristic, we'll need to
> revisit how BIP 158 works briefly. For each block, we insert the
> `pkScript`s
> of all the outputs, and also the prev out's pkScript (the script being
> spent) as well. This lets the filter compress script re-use in both inputs
> and outputs, and also makes it possible to implement some protocols in a
> more light-client friendly manner (for example Loop uses this to has the
> client watch HTLC _scripts_ being spent, as it doesn't necessarily know the
> txid/outpoint).
>
> The one issue with this, is that while clients can ensure that all the
> `pkScripts` of outputs have been inserted, they can't do the same for the
> inputs (which is why we added that heuristic in the client code). Luckily
> we
> know how to properly fix this at the protocol level, more on that below.
>
> ## How can I make sure my neutrino clients handle the Taproot upgrade on
> mainnet smoothly?
>
> Upgrade to 0.14 (assuming it's out in time), or apply this small patch [4].
> The patch just demotes an error case to a warning message, so anyone
> running
> a fork should be able to easily apply the fix.
>
> Alongside, optionally extend these filter header guides [7].
>
> We're looking into some intermediate ground where we can verify the scripts
> that we know are relevant to the node.
>
> ## How will BIP 158/157 evolve post taproot?
>
> In terms of adding more taproot specific functionality, I've had a number
> of
> items in my laundry list such as:
>
>   * creating new segwit-only filters with re-parameterized fp rates (also
>     examine other filter types such as pure outpoints, etc)
>
>   * creating filters that include witness data to allow matching on
>     internal/external keys, the control block, merkle root, annex, etc
>
>   * add a new protocol extension to btcd (with a corresponding BIP) to
>     allow notes to fetch block undo data (as described here [5]) to fully
>     verify fetched filters or a node needs to reconcile conflicting filters
>
>   * new filters that span across multiple blocks as previously worked on by
>     Kalle Alm (couldn't find a link to his PR when typing this...)
>
> Make further progress towards a proposal that allows filters to be
> committed
> either as a soft-fork, or a "velvet fork" [6] where miners optionally
> commit to
> the past filter header chain.
>
>
> -- Laolu
>
> [1]:
> https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8
> [2]: https://github.com/lightninglabs/neutrino/pull/234
> [3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0
> [4]: https://github.com/lightninglabs/neutrino/pull/234/files
> [5]:
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html
> [6]: https://eprint.iacr.org/2018/087
> [7]:
> https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15
> [8]: https://github.com/lightninglabs/neutrino
>

[-- Attachment #2: Type: text/html, Size: 6109 bytes --]