[bitcoindev] SwiftSync - smarter synchronization with hints

[bitcoindev] SwiftSync - smarter synchronization with hints

[Ruben Somsen]

[-- Attachment #1: Type: text/plain, Size: 3481 bytes --]

Hi everyone,

SwiftSync is a new validation method that allows for near-stateless, fully
parallelizable validation of the Bitcoin blockchain via hints about which
outputs remain unspent (<100MB total). All other inputs/outputs are
efficiently crossed off inside a single hash aggregate that only reaches
zero if validation was successful and the hints were correct.

The main observation is that it can be much easier to validate that a given
UTXO set is correct than to compute it yourself. It allows us to no longer
require a stateful moment-to-moment UTXO set during IBD and allows
everything to be order independent. I'll briefly summarize the protocol,
before sharing the link to the full write-up.

Each output gets a boolean hint (e.g. committed into Bitcoin Core) about
whether or not it will still be in the UTXO set after validation completes.
If it does, we write it to disk (append-only - it won't be used until
SwiftSync finishes). If it does not, we hash the UTXO data and add it to an
aggregate. For each input, we once again hash the UTXO data and remove it
from the aggregate.

At the end, for every added output there should have been exactly one
removed input, bringing the end total of the aggregate to zero. If this is
indeed the case, we will have validated that the hints, and the resulting
UTXO set, were correct.

E.g. For spent outputs A, B and inputs C, D we calculate hash(UTXO_A||salt)
+ hash(UTXO_B||salt) - hash(UTXO_C||salt) - hash(UTXO_D||salt) == 0
(proving (A==C && B==D) || (A==D && B==C)).

There is one missing step. The UTXO data is only available when processing
the output, but not when processing the input. We resolve this by either
downloading the outputs that were spent for each block (equivalent to the
undo data, maybe 10-15% of a block), or we lean on assumevalid, making it
sufficient to only hash the outpoints (which are available in both the
output and input) rather than the full UTXO data.

Ignoring bandwidth, the expectation is that the speedup will be most
significant on either RAM limited devices and/or devices with many CPU
cores. Initial PoC benchmarks (thanks to theStack) show a 5.28x speed-up,
while currently still being largely sequential.

Many more details are in the full write-up:
https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c78133cd

It will answer the following questions (and more):

- How the hash aggregate can be made secure against the generalized
birthday problem
- How exactly assumevalid is utilized and what the implications are
- How we can still check for inflation when we skip the amounts with
assumevalid
- How we can validate transaction order while doing everything in parallel
- How we can perform the BIP30 duplicate output check without the UTXO set
- How this all relates to assumeutxo

To my knowledge, every validation check involving the UTXO set is covered,
but I'd be curious to hear if anything was overlooked or if you spot any
other issues.

Thanks for reading, and thanks to everyone who provided invaluable feedback
while the idea was coming together.

-- Ruben Somsen

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/CAPv7TjaM0tfbcBTRa0_713Bk6Y9jr%2BShOC1KZi2V3V2zooTXyg%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4042 bytes --]