[bitcoin-dev] assumeutxo and UTXO snapshots

[bitcoin-dev] assumeutxo and UTXO snapshots

[Nicolas Dorier]

[-- Attachment #1: Type: text/plain, Size: 19524 bytes --]

James,

You might be interested by my work which is currently used in production,
without any change to bitcoin core.

I properly explain how to verify the utxoset independently.

https://github.com/btcpayserver/btcpayserver-docker/blob/master/contrib/FastSync/README.md


People are using it, since I get around 10 download a day.
What can be done to help at Bitcoin Core level is actually very minimal.

First, instead of asking signers of by UTXOSet to sign the utxoset hash
from gettxoutsetinfo, I ask them to sign the hash of the tarball of my UTXO
Set.

The reason is that it is currently impossible to stop BitcoinD on a
specific block then asking the serialized hash of the UTXO Set.

So instead, a verifier download the tarball (300 blocks + utxoset at
specific height), sync to the latest block, then compare the
gettxoutsetinfo of the newly synched node with another trusted node. If it
match, the verifier sign the tarball.

I create a new utxoset snapshot every 6 months, so people have time to
verify it and add their signatures. (Approximately once every bitcoin core
release)

The easiest thing that could be done at Bitcoin Core level does not require
any code change, but a change in the release process.

The new process would be to ask to the gitian signers to not only build the
source themselves, but also verify a tarball following the procedure I
explain in the link above.

More complicated solution like signing the serialized utxoset itself, while
possible, would require bothersome code changes.

Nicolas,

On Wed, Apr 3, 2019 at 9:25 AM <
bitcoin-dev-request@lists.linuxfoundation.org> wrote:

> Send bitcoin-dev mailing list submissions to
>         bitcoin-dev@lists.linuxfoundation.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> or, via email, send a message with subject or body 'help' to
>         bitcoin-dev-request@lists.linuxfoundation.org
>
> You can reach the person managing the list at
>         bitcoin-dev-owner@lists.linuxfoundation.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of bitcoin-dev digest..."
>
>
> Today's Topics:
>
>    1. BIP: Bitcoin Integrated Address Feature? (nathanw@tutanota.com)
>    2. Re: BIP: Bitcoin Integrated Address Feature? (htimSxelA)
>    3. assumeutxo and UTXO snapshots (James O'Beirne)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 2 Apr 2019 18:53:11 +0200 (CEST)
> From: <nathanw@tutanota.com>
> To: <bitcoin-dev@lists.linuxfoundation.org>
> Subject: [bitcoin-dev] BIP: Bitcoin Integrated Address Feature?
> Message-ID: <LbTxyE4--3-1@tutanota.com>
> Content-Type: text/plain; charset="utf-8"
>
> To whom it may concern,
>
> I believe a missing feature in Bitcoin is the ability to have an
> "integrated address", where the address resolves into a Bitcoin address,
> and also a transaction message or some other kind of identifier.
>
> By having this feature we could enhance the security of exchange
> cold-wallet systems, by allowing them to easily receive all payments to a
> single address from an infinite number of customers. We would also greatly
> simplify the process of setting up and managing exchange cold-wallet
> systems, because we would eliminate the "sweeping" step required to move
> multiple customer deposits from a hot address into a single cold address.
>
> Although it would be nice to have all customers deposit directly into cold
> addresses, this quickly becomes impractical when large amounts of customers
> begin to use exchange wallets as their personal web-wallet, frequently
> depositing and withdrawing without trading action. You end up needing to
> have a staff member moving funds away from cold deposit addresses as a full
> time job - if you wish to handle customer funds in a completely secure
> manner.
>
> Thus we see that most exchanges now use the hot-deposit system, where
> customers deposit into a hot address that is then automatically swept into
> a singular cold address, by a service which holds customers private keys
> online. You can observe this service at work simply by making a deposit to
> most major exchanges (including the largest exchange Binance), as you will
> see the funds quickly being "swept" to their cold wallet address in a
> manner which heavily suggests automation by a program which possesses
> private keys to the address you are sending funds to. This means there is
> always the danger of a sophisticated hacker being able to capture private
> keys to customer deposit addresses (as they are clearly being held online).
> An integrated address would allow all exchanges using this automated
> hot-deposit service to easily switch to a far more secure alternative of
> having all customers depositing directly into their singular cold wallet
> address.
>
> There are several other more minor advantages such a feature would have,
> including:
> - Lower fees for exchanges (which could be passed onto customers), by
> reducing a transaction step out of the deposit-to-withdrawal flow.
> - Less need for large rescans after loading huge amounts of customer
> addresses into client software.
> - Exchanges can more easily provision deposit addresses to new customers
> in a secure manner, by simply generating a hex or other value, creating an
> integrated address from the cold wallet address, and then providing this to
> the customer.
> - By providing a singular cold address for exchanges publicly, customers
> can more easily verify that no man-in-the-middle has given them an
> incorrect address to deposit to.
> The integrated address could work by combining the Bitcoin address
> together with some kind of hex or other value, allowing users to choose the
> amount they wish to deposit themselves, but ensuring their deposits are
> uniquely trackable.
>
> I'm not sure if some kind of functionality already exists in BTC, as I
> haven't been able to find it. If not, can I submit a proposal to implement
> this? This feature would be a godsend to all exchange developers if it was
> widely accepted.
>
> Thanks for your time.
> Regards,
>
> Nathan Worsley
> CTO - LocalCoinSwap.Com
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190402/400c1e1b/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 2
> Date: Tue, 02 Apr 2019 20:01:34 +0000
> From: htimSxelA <htimsxela@protonmail.com>
> To: "nathanw@tutanota.com" <nathanw@tutanota.com>,      Bitcoin Protocol
>         Discussion <bitcoin-dev@lists.linuxfoundation.org>
> Subject: Re: [bitcoin-dev] BIP: Bitcoin Integrated Address Feature?
> Message-ID:
>
> <wtbAF1FAGePDAkY3xkqANuFJtAhEXvz0JeGWnc_OZcGEyFQb-1B590I3IbwtW2FBivur0yONbSQtxaWqiQTJeoDdadivtbGkWwJnLnnzQQE=@
> protonmail.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Hello,
>
> I see two immediate issues with this:
> 1. Increased resource requirements per transaction
> 2. Embedding identifying information into the blockchain is generally bad
> for privacy
>
> It may help your case to provide some technical details of how you'd like
> to see this implemented, but without overcoming the issues mentioned above
> I think this proposal will be a very tough sell.
>
> > ...this quickly becomes impractical when large amounts of customers
> begin to use exchange wallets as their personal web-wallet, frequently
> depositing and withdrawing without trading action. You end up needing to
> have a staff member moving funds away from cold deposit addresses as a full
> time job - if you wish to handle customer funds in a completely secure
> manner.
>
> I am not sure if I see how this issue is solved by your proposal.
> Assumedly, a human will still need to manually approve cold-wallet
> withdrawals in order to maintain security. So it seems to me that removing
> the 'hot-wallet' component of the backend would only amplify the need for
> human interaction.
>
> I assume you are familiar with hierarchical deterministic wallets? They
> can allow an exchange to assign/identify user deposits based on address
> derivation path. Keys for deposit addresses can be kept offline if wanted,
> and a proper implementation of an HD wallet system should also remove the
> need for rescans of user deposit addresses.
>
> There is also a functionality built into Bitcoin that allows a user to
> prove that they own the private keys to some address: signing an agreed
> upon message using the private key that controls that address.
> Unfortunately I don't think this is a workable solution for you, since the
> majority of modern wallet software does not include this feature-- but
> perhaps worth mentioning nonetheless.
>
> Best,
> Alex
>
> ??????? Original Message ???????
> On Tuesday, April 2, 2019 9:53 AM, Nathan Worsley via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> > To whom it may concern,
> >
> > I believe a missing feature in Bitcoin is the ability to have an
> "integrated address", where the address resolves into a Bitcoin address,
> and also a transaction message or some other kind of identifier.
> >
> > By having this feature we could enhance the security of exchange
> cold-wallet systems, by allowing them to easily receive all payments to a
> single address from an infinite number of customers. We would also greatly
> simplify the process of setting up and managing exchange cold-wallet
> systems, because we would eliminate the "sweeping" step required to move
> multiple customer deposits from a hot address into a single cold address.
> >
> > Although it would be nice to have all customers deposit directly into
> cold addresses, this quickly becomes impractical when large amounts of
> customers begin to use exchange wallets as their personal web-wallet,
> frequently depositing and withdrawing without trading action. You end up
> needing to have a staff member moving funds away from cold deposit
> addresses as a full time job - if you wish to handle customer funds in a
> completely secure manner.
> >
> > Thus we see that most exchanges now use the hot-deposit system, where
> customers deposit into a hot address that is then automatically swept into
> a singular cold address, by a service which holds customers private keys
> online. You can observe this service at work simply by making a deposit to
> most major exchanges (including the largest exchange Binance), as you will
> see the funds quickly being "swept" to their cold wallet address in a
> manner which heavily suggests automation by a program which possesses
> private keys to the address you are sending funds to. This means there is
> always the danger of a sophisticated hacker being able to capture private
> keys to customer deposit addresses (as they are clearly being held online).
> An integrated address would allow all exchanges using this automated
> hot-deposit service to easily switch to a far more secure alternative of
> having all customers depositing directly into their singular cold wallet
> address.
> >
> > There are several other more minor advantages such a feature would have,
> including:
> > - Lower fees for exchanges (which could be passed onto customers), by
> reducing a transaction step out of the deposit-to-withdrawal flow.
> > - Less need for large rescans after loading huge amounts of customer
> addresses into client software.
> > - Exchanges can more easily provision deposit addresses to new customers
> in a secure manner, by simply generating a hex or other value, creating an
> integrated address from the cold wallet address, and then providing this to
> the customer.
> > - By providing a singular cold address for exchanges publicly, customers
> can more easily verify that no man-in-the-middle has given them an
> incorrect address to deposit to.
> >
> > The integrated address could work by combining the Bitcoin address
> together with some kind of hex or other value, allowing users to choose the
> amount they wish to deposit themselves, but ensuring their deposits are
> uniquely trackable.
> >
> > I'm not sure if some kind of functionality already exists in BTC, as I
> haven't been able to find it. If not, can I submit a proposal to implement
> this? This feature would be a godsend to all exchange developers if it was
> widely accepted.
> >
> > Thanks for your time.
> >
> > Regards,
> >
> > Nathan Worsley
> > CTO - LocalCoinSwap.Com
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190402/ade34235/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 3
> Date: Tue, 2 Apr 2019 16:43:11 -0400
> From: "James O'Beirne" <james.obeirne@gmail.com>
> To: bitcoin-dev@lists.linuxfoundation.org
> Subject: [bitcoin-dev] assumeutxo and UTXO snapshots
> Message-ID:
>         <CAPfvXf+JS6ZhXUieWVxiaNa4uhhWwafCk3odMKy5F_yi=
> XwngA@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
>
> I'd like to discuss assumeutxo, which is an appealing and simple
> optimization in the spirit of assumevalid[0].
>
> # Motivation
>
> To start a fully validating bitcoin client from scratch, that client
> currently
> needs to perform an initial block download. To the surprise of no one, IBD
> takes a linear amount time based on the length of the chain's history. For
> clients running on modest hardware under limited bandwidth constraints,
> say a mobile device, completing IBD takes a considerable amount of time
> and thus poses serious usability challenges.
>
> As a result, having fully validating clients run on such hardware is rare
> and
> basically unrealistic. Clients with even moderate resource constraints
> are encouraged to rely on the SPV trust model. Though we have promising
> improvements to existing SPV modes pending deployment[1], it's worth
> thinking about a mechanism that would allow such clients to use trust
> models closer to full validation.
>
> The subject of this mail is a proposal for a complementary alternative to
> SPV
> modes, and which is in the spirit of an existing default, `assumevalid`. It
> may
> help modest clients transact under a security model that closely resembles
> full validation within minutes instead of hours or days.
>
> # assumeutxo
>
> The basic idea is to allow nodes to initialize using a serialized version
> of the
> UTXO set rendered by another node at some predetermined height. The
> initializing node syncs the headers chain from the network, then obtains
> and
> loads one of these UTXO snapshots (i.e. a serialized version of the UTXO
> set
> bundled with the block header indicating its "base" and some other
> metadata).
>
> Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> download of the 3.2GB snapshot[2].
>
> The node then syncs to the network tip and afterwards begins a simultaneous
> background validation (i.e., a conventional IBD) up to the base height of
> the
> snapshot in order to achieve full validation. Crucially, even while the
> background validation is happening the node can validate incoming blocks
> and
> transact with the benefit of the full (assumed-valid) UTXO set.
>
> Snapshots could be obtained from multiple separate peers in the same manner
> as
> block download, but I haven't put much thought into this. In concept it
> doesn't
> matter too much where the snapshots come from since their validity is
> determined via content hash.
>
> # Security
>
> Obviously there are some security implications due consideration. While
> this
> proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> Under assumevalid, a user can be tricked into transacting under a false
> history
> if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> parameter, sybils their node, and then feeds them a bogus chain
> encompassing
> all of the hard-coded checkpoints[3].
>
> The same attack is made easier in assumeutxo because, unlike in
> assumevalid,
> the attacker need not construct a valid PoW chain to get the victim's node
> into
> a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> false coin assignment.
>
> For this reason, I recommend that if we were to implement assumeutxo, we
> not
> allow its specification via commandline argument[4].
>
> Beyond this risk, I can't think of material differences in security
> relative to
> assumevalid, though I appeal to the list for help with this.
>
> # More fully validating clients
>
> A particularly exciting use-case for assumeutxo is the possibility of
> mobile
> devices functioning as fully validating nodes with access to the complete
> UTXO
> set (as an alternative to SPV models). The total resource burden needed to
> start a node
> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> + blocks_to_tip * 4MB) download and a few minutes of processing time, which
> sounds
> manageable for many mobile devices currently in use.
>
> A mobile user could initialize an assumed-valid bitcoin node within an
> hour,
> transact immediately, and complete a pruned full validation of their
> assumed-valid chain over the next few days, perhaps only doing the
> background
> IBD when their device has access to suitable high-bandwidth connections.
>
> If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> the road, it's easy to imagine an analogous process that would allow very
> fast
> startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
>
> ---
>
> I've created a related issue at our Github repository here:
>   https://github.com/bitcoin/bitcoin/issues/15605
>
> and have submitted a draft implementation of snapshot usage via RPC here:
>   https://github.com/bitcoin/bitcoin/pull/15606
>
> I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> plans for deployment steps should be discussed in the Github issue, and
> after all
> that my implementation may be reviewed as a sketch of the specific software
> changes necessary.
>
> Regards,
> James
>
>
> [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
> 2.10GHz
> [3]:
>
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> [4]: Marco Falke is due credit for this point
> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190402/46b25dd8/attachment.html
> >
>
> ------------------------------
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
> End of bitcoin-dev Digest, Vol 47, Issue 6
> ******************************************
>

[-- Attachment #2: Type: text/html, Size: 23951 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Kulpreet Singh]

Hi Nicolas,

I have a small question about FastSync.

Would it make sense to validate all blocks once FastSync is complete
and BTCPayServer has started accepting payments?

I am aware this will require changes to bitcoind. So this is just an
academic question to figure if there are problems with such an
approach, especially for merchants accepting payments who want to get
started immediately and still want to stay on a Raspberry PI.

Phase 1: FastSync from trusted UTXO set and start accepting payments.
Phase 2: Validate the entire blockchain - this will take X days on
Raspberry PI - but at least in in the end you can fully trust your own
node. In this phase, you'd do IBD, but instead of writing to db, just
verify that the validated block matches the on the the db and move on.

It is a pity leveldb doesn't allow multiple processes to open the db.
If so, phase 2 could have been a different process altogether as well.

Regards
Kulpreet



On Wed, 3 Apr 2019 at 21:23, Nicolas Dorier via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> James,
>
> You might be interested by my work which is currently used in production, without any change to bitcoin core.
>
> I properly explain how to verify the utxoset independently.
>
> https://github.com/btcpayserver/btcpayserver-docker/blob/master/contrib/FastSync/README.md
>
> People are using it, since I get around 10 download a day.
> What can be done to help at Bitcoin Core level is actually very minimal.
>
> First, instead of asking signers of by UTXOSet to sign the utxoset hash from gettxoutsetinfo, I ask them to sign the hash of the tarball of my UTXO Set.
>
> The reason is that it is currently impossible to stop BitcoinD on a specific block then asking the serialized hash of the UTXO Set.
>
> So instead, a verifier download the tarball (300 blocks + utxoset at specific height), sync to the latest block, then compare the gettxoutsetinfo of the newly synched node with another trusted node. If it match, the verifier sign the tarball.
>
> I create a new utxoset snapshot every 6 months, so people have time to verify it and add their signatures. (Approximately once every bitcoin core release)
>
> The easiest thing that could be done at Bitcoin Core level does not require any code change, but a change in the release process.
>
> The new process would be to ask to the gitian signers to not only build the source themselves, but also verify a tarball following the procedure I explain in the link above.
>
> More complicated solution like signing the serialized utxoset itself, while possible, would require bothersome code changes.
>
> Nicolas,
>
> On Wed, Apr 3, 2019 at 9:25 AM <bitcoin-dev-request@lists.linuxfoundation.org> wrote:
>>
>> Send bitcoin-dev mailing list submissions to
>>         bitcoin-dev@lists.linuxfoundation.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> or, via email, send a message with subject or body 'help' to
>>         bitcoin-dev-request@lists.linuxfoundation.org
>>
>> You can reach the person managing the list at
>>         bitcoin-dev-owner@lists.linuxfoundation.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of bitcoin-dev digest..."
>>
>>
>> Today's Topics:
>>
>>    1. BIP: Bitcoin Integrated Address Feature? (nathanw@tutanota.com)
>>    2. Re: BIP: Bitcoin Integrated Address Feature? (htimSxelA)
>>    3. assumeutxo and UTXO snapshots (James O'Beirne)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 2 Apr 2019 18:53:11 +0200 (CEST)
>> From: <nathanw@tutanota.com>
>> To: <bitcoin-dev@lists.linuxfoundation.org>
>> Subject: [bitcoin-dev] BIP: Bitcoin Integrated Address Feature?
>> Message-ID: <LbTxyE4--3-1@tutanota.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> To whom it may concern,
>>
>> I believe a missing feature in Bitcoin is the ability to have an "integrated address", where the address resolves into a Bitcoin address, and also a transaction message or some other kind of identifier.
>>
>> By having this feature we could enhance the security of exchange cold-wallet systems, by allowing them to easily receive all payments to a single address from an infinite number of customers. We would also greatly simplify the process of setting up and managing exchange cold-wallet systems, because we would eliminate the "sweeping" step required to move multiple customer deposits from a hot address into a single cold address.
>>
>> Although it would be nice to have all customers deposit directly into cold addresses, this quickly becomes impractical when large amounts of customers begin to use exchange wallets as their personal web-wallet, frequently depositing and withdrawing without trading action. You end up needing to have a staff member moving funds away from cold deposit addresses as a full time job - if you wish to handle customer funds in a completely secure manner.
>>
>> Thus we see that most exchanges now use the hot-deposit system, where customers deposit into a hot address that is then automatically swept into a singular cold address, by a service which holds customers private keys online. You can observe this service at work simply by making a deposit to most major exchanges (including the largest exchange Binance), as you will see the funds quickly being "swept" to their cold wallet address in a manner which heavily suggests automation by a program which possesses private keys to the address you are sending funds to. This means there is always the danger of a sophisticated hacker being able to capture private keys to customer deposit addresses (as they are clearly being held online). An integrated address would allow all exchanges using this automated hot-deposit service to easily switch to a far more secure alternative of having all customers depositing directly into their singular cold wallet address.
>>
>> There are several other more minor advantages such a feature would have, including:
>> - Lower fees for exchanges (which could be passed onto customers), by reducing a transaction step out of the deposit-to-withdrawal flow.
>> - Less need for large rescans after loading huge amounts of customer addresses into client software.
>> - Exchanges can more easily provision deposit addresses to new customers in a secure manner, by simply generating a hex or other value, creating an integrated address from the cold wallet address, and then providing this to the customer.
>> - By providing a singular cold address for exchanges publicly, customers can more easily verify that no man-in-the-middle has given them an incorrect address to deposit to.
>> The integrated address could work by combining the Bitcoin address together with some kind of hex or other value, allowing users to choose the amount they wish to deposit themselves, but ensuring their deposits are uniquely trackable.
>>
>> I'm not sure if some kind of functionality already exists in BTC, as I haven't been able to find it. If not, can I submit a proposal to implement this? This feature would be a godsend to all exchange developers if it was widely accepted.
>>
>> Thanks for your time.
>> Regards,
>>
>> Nathan Worsley
>> CTO - LocalCoinSwap.Com
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190402/400c1e1b/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 02 Apr 2019 20:01:34 +0000
>> From: htimSxelA <htimsxela@protonmail.com>
>> To: "nathanw@tutanota.com" <nathanw@tutanota.com>,      Bitcoin Protocol
>>         Discussion <bitcoin-dev@lists.linuxfoundation.org>
>> Subject: Re: [bitcoin-dev] BIP: Bitcoin Integrated Address Feature?
>> Message-ID:
>>         <wtbAF1FAGePDAkY3xkqANuFJtAhEXvz0JeGWnc_OZcGEyFQb-1B590I3IbwtW2FBivur0yONbSQtxaWqiQTJeoDdadivtbGkWwJnLnnzQQE=@protonmail.com>
>>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hello,
>>
>> I see two immediate issues with this:
>> 1. Increased resource requirements per transaction
>> 2. Embedding identifying information into the blockchain is generally bad for privacy
>>
>> It may help your case to provide some technical details of how you'd like to see this implemented, but without overcoming the issues mentioned above I think this proposal will be a very tough sell.
>>
>> > ...this quickly becomes impractical when large amounts of customers begin to use exchange wallets as their personal web-wallet, frequently depositing and withdrawing without trading action. You end up needing to have a staff member moving funds away from cold deposit addresses as a full time job - if you wish to handle customer funds in a completely secure manner.
>>
>> I am not sure if I see how this issue is solved by your proposal. Assumedly, a human will still need to manually approve cold-wallet withdrawals in order to maintain security. So it seems to me that removing the 'hot-wallet' component of the backend would only amplify the need for human interaction.
>>
>> I assume you are familiar with hierarchical deterministic wallets? They can allow an exchange to assign/identify user deposits based on address derivation path. Keys for deposit addresses can be kept offline if wanted, and a proper implementation of an HD wallet system should also remove the need for rescans of user deposit addresses.
>>
>> There is also a functionality built into Bitcoin that allows a user to prove that they own the private keys to some address: signing an agreed upon message using the private key that controls that address. Unfortunately I don't think this is a workable solution for you, since the majority of modern wallet software does not include this feature-- but perhaps worth mentioning nonetheless.
>>
>> Best,
>> Alex
>>
>> ??????? Original Message ???????
>> On Tuesday, April 2, 2019 9:53 AM, Nathan Worsley via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> > To whom it may concern,
>> >
>> > I believe a missing feature in Bitcoin is the ability to have an "integrated address", where the address resolves into a Bitcoin address, and also a transaction message or some other kind of identifier.
>> >
>> > By having this feature we could enhance the security of exchange cold-wallet systems, by allowing them to easily receive all payments to a single address from an infinite number of customers. We would also greatly simplify the process of setting up and managing exchange cold-wallet systems, because we would eliminate the "sweeping" step required to move multiple customer deposits from a hot address into a single cold address.
>> >
>> > Although it would be nice to have all customers deposit directly into cold addresses, this quickly becomes impractical when large amounts of customers begin to use exchange wallets as their personal web-wallet, frequently depositing and withdrawing without trading action. You end up needing to have a staff member moving funds away from cold deposit addresses as a full time job - if you wish to handle customer funds in a completely secure manner.
>> >
>> > Thus we see that most exchanges now use the hot-deposit system, where customers deposit into a hot address that is then automatically swept into a singular cold address, by a service which holds customers private keys online. You can observe this service at work simply by making a deposit to most major exchanges (including the largest exchange Binance), as you will see the funds quickly being "swept" to their cold wallet address in a manner which heavily suggests automation by a program which possesses private keys to the address you are sending funds to. This means there is always the danger of a sophisticated hacker being able to capture private keys to customer deposit addresses (as they are clearly being held online). An integrated address would allow all exchanges using this automated hot-deposit service to easily switch to a far more secure alternative of having all customers depositing directly into their singular cold wallet address.
>> >
>> > There are several other more minor advantages such a feature would have, including:
>> > - Lower fees for exchanges (which could be passed onto customers), by reducing a transaction step out of the deposit-to-withdrawal flow.
>> > - Less need for large rescans after loading huge amounts of customer addresses into client software.
>> > - Exchanges can more easily provision deposit addresses to new customers in a secure manner, by simply generating a hex or other value, creating an integrated address from the cold wallet address, and then providing this to the customer.
>> > - By providing a singular cold address for exchanges publicly, customers can more easily verify that no man-in-the-middle has given them an incorrect address to deposit to.
>> >
>> > The integrated address could work by combining the Bitcoin address together with some kind of hex or other value, allowing users to choose the amount they wish to deposit themselves, but ensuring their deposits are uniquely trackable.
>> >
>> > I'm not sure if some kind of functionality already exists in BTC, as I haven't been able to find it. If not, can I submit a proposal to implement this? This feature would be a godsend to all exchange developers if it was widely accepted.
>> >
>> > Thanks for your time.
>> >
>> > Regards,
>> >
>> > Nathan Worsley
>> > CTO - LocalCoinSwap.Com
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190402/ade34235/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Tue, 2 Apr 2019 16:43:11 -0400
>> From: "James O'Beirne" <james.obeirne@gmail.com>
>> To: bitcoin-dev@lists.linuxfoundation.org
>> Subject: [bitcoin-dev] assumeutxo and UTXO snapshots
>> Message-ID:
>>         <CAPfvXf+JS6ZhXUieWVxiaNa4uhhWwafCk3odMKy5F_yi=XwngA@mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi,
>>
>> I'd like to discuss assumeutxo, which is an appealing and simple
>> optimization in the spirit of assumevalid[0].
>>
>> # Motivation
>>
>> To start a fully validating bitcoin client from scratch, that client
>> currently
>> needs to perform an initial block download. To the surprise of no one, IBD
>> takes a linear amount time based on the length of the chain's history. For
>> clients running on modest hardware under limited bandwidth constraints,
>> say a mobile device, completing IBD takes a considerable amount of time
>> and thus poses serious usability challenges.
>>
>> As a result, having fully validating clients run on such hardware is rare
>> and
>> basically unrealistic. Clients with even moderate resource constraints
>> are encouraged to rely on the SPV trust model. Though we have promising
>> improvements to existing SPV modes pending deployment[1], it's worth
>> thinking about a mechanism that would allow such clients to use trust
>> models closer to full validation.
>>
>> The subject of this mail is a proposal for a complementary alternative to
>> SPV
>> modes, and which is in the spirit of an existing default, `assumevalid`. It
>> may
>> help modest clients transact under a security model that closely resembles
>> full validation within minutes instead of hours or days.
>>
>> # assumeutxo
>>
>> The basic idea is to allow nodes to initialize using a serialized version
>> of the
>> UTXO set rendered by another node at some predetermined height. The
>> initializing node syncs the headers chain from the network, then obtains and
>> loads one of these UTXO snapshots (i.e. a serialized version of the UTXO set
>> bundled with the block header indicating its "base" and some other
>> metadata).
>>
>> Based upon the snapshot, the node is able to quickly reconstruct its
>> chainstate,
>> and compares a hash of the resulting UTXO set to a preordained hash
>> hard-coded
>> in the software a la assumevalid. This all takes ~23 minutes, not
>> accounting for
>> download of the 3.2GB snapshot[2].
>>
>> The node then syncs to the network tip and afterwards begins a simultaneous
>> background validation (i.e., a conventional IBD) up to the base height of
>> the
>> snapshot in order to achieve full validation. Crucially, even while the
>> background validation is happening the node can validate incoming blocks and
>> transact with the benefit of the full (assumed-valid) UTXO set.
>>
>> Snapshots could be obtained from multiple separate peers in the same manner
>> as
>> block download, but I haven't put much thought into this. In concept it
>> doesn't
>> matter too much where the snapshots come from since their validity is
>> determined via content hash.
>>
>> # Security
>>
>> Obviously there are some security implications due consideration. While this
>> proposal is in the spirit of assumevalid, practical attacks may become
>> easier.
>> Under assumevalid, a user can be tricked into transacting under a false
>> history
>> if an attacker convinces them to start bitcoind with a malicious
>> `-assumevalid`
>> parameter, sybils their node, and then feeds them a bogus chain encompassing
>> all of the hard-coded checkpoints[3].
>>
>> The same attack is made easier in assumeutxo because, unlike in assumevalid,
>> the attacker need not construct a valid PoW chain to get the victim's node
>> into
>> a false state; they simply need to get the user to accept a bad
>> `-assumeutxo`
>> parameter and then supply them an easily made UTXO snapshot containing,
>> say, a
>> false coin assignment.
>>
>> For this reason, I recommend that if we were to implement assumeutxo, we not
>> allow its specification via commandline argument[4].
>>
>> Beyond this risk, I can't think of material differences in security
>> relative to
>> assumevalid, though I appeal to the list for help with this.
>>
>> # More fully validating clients
>>
>> A particularly exciting use-case for assumeutxo is the possibility of mobile
>> devices functioning as fully validating nodes with access to the complete
>> UTXO
>> set (as an alternative to SPV models). The total resource burden needed to
>> start a node
>> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
>> + blocks_to_tip * 4MB) download and a few minutes of processing time, which
>> sounds
>> manageable for many mobile devices currently in use.
>>
>> A mobile user could initialize an assumed-valid bitcoin node within an hour,
>> transact immediately, and complete a pruned full validation of their
>> assumed-valid chain over the next few days, perhaps only doing the
>> background
>> IBD when their device has access to suitable high-bandwidth connections.
>>
>> If we end up implementing an accumulator-based UTXO scaling design[5][6]
>> down
>> the road, it's easy to imagine an analogous process that would allow very
>> fast
>> startup using an accumulator of a few kilobytes in lieu of a multi-GB
>> snapshot.
>>
>> ---
>>
>> I've created a related issue at our Github repository here:
>>   https://github.com/bitcoin/bitcoin/issues/15605
>>
>> and have submitted a draft implementation of snapshot usage via RPC here:
>>   https://github.com/bitcoin/bitcoin/pull/15606
>>
>> I'd like to discuss here whether this is a good fit for Bitcoin
>> conceptually. Concrete
>> plans for deployment steps should be discussed in the Github issue, and
>> after all
>> that my implementation may be reviewed as a sketch of the specific software
>> changes necessary.
>>
>> Regards,
>> James
>>
>>
>> [0]:
>> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
>> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
>> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
>> 2.10GHz
>> [3]:
>> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
>> [4]: Marco Falke is due credit for this point
>> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
>> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190402/46b25dd8/attachment.html>
>>
>> ------------------------------
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>> End of bitcoin-dev Digest, Vol 47, Issue 6
>> ******************************************
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[James O'Beirne]

[-- Attachment #1: Type: text/plain, Size: 6535 bytes --]

Good morning all,

Over the past weeks I've had a number of conversations with a few frequent
contributors about this idea. I've condensed these discussions into a
proposal document which you can view here:
https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal

The document is structured as an FAQ, and so hopefully it addresses some of
the common questions that would come up in this thread. If you'd like to
comment, there's an associated pull request here:
https://github.com/jamesob/assumeutxo-docs/pull/1

Regards,
James


On Tue, Apr 2, 2019 at 4:43 PM James O'Beirne <james.obeirne@gmail.com>
wrote:

> Hi,
>
> I'd like to discuss assumeutxo, which is an appealing and simple
> optimization in the spirit of assumevalid[0].
>
> # Motivation
>
> To start a fully validating bitcoin client from scratch, that client
> currently
> needs to perform an initial block download. To the surprise of no one, IBD
> takes a linear amount time based on the length of the chain's history. For
> clients running on modest hardware under limited bandwidth constraints,
> say a mobile device, completing IBD takes a considerable amount of time
> and thus poses serious usability challenges.
>
> As a result, having fully validating clients run on such hardware is rare
> and
> basically unrealistic. Clients with even moderate resource constraints
> are encouraged to rely on the SPV trust model. Though we have promising
> improvements to existing SPV modes pending deployment[1], it's worth
> thinking about a mechanism that would allow such clients to use trust
> models closer to full validation.
>
> The subject of this mail is a proposal for a complementary alternative to
> SPV
> modes, and which is in the spirit of an existing default, `assumevalid`.
> It may
> help modest clients transact under a security model that closely resembles
> full validation within minutes instead of hours or days.
>
> # assumeutxo
>
> The basic idea is to allow nodes to initialize using a serialized version
> of the
> UTXO set rendered by another node at some predetermined height. The
> initializing node syncs the headers chain from the network, then obtains
> and
> loads one of these UTXO snapshots (i.e. a serialized version of the UTXO
> set
> bundled with the block header indicating its "base" and some other
> metadata).
>
> Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> download of the 3.2GB snapshot[2].
>
> The node then syncs to the network tip and afterwards begins a simultaneous
> background validation (i.e., a conventional IBD) up to the base height of
> the
> snapshot in order to achieve full validation. Crucially, even while the
> background validation is happening the node can validate incoming blocks
> and
> transact with the benefit of the full (assumed-valid) UTXO set.
>
> Snapshots could be obtained from multiple separate peers in the same
> manner as
> block download, but I haven't put much thought into this. In concept it
> doesn't
> matter too much where the snapshots come from since their validity is
> determined via content hash.
>
> # Security
>
> Obviously there are some security implications due consideration. While
> this
> proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> Under assumevalid, a user can be tricked into transacting under a false
> history
> if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> parameter, sybils their node, and then feeds them a bogus chain
> encompassing
> all of the hard-coded checkpoints[3].
>
> The same attack is made easier in assumeutxo because, unlike in
> assumevalid,
> the attacker need not construct a valid PoW chain to get the victim's node
> into
> a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> false coin assignment.
>
> For this reason, I recommend that if we were to implement assumeutxo, we
> not
> allow its specification via commandline argument[4].
>
> Beyond this risk, I can't think of material differences in security
> relative to
> assumevalid, though I appeal to the list for help with this.
>
> # More fully validating clients
>
> A particularly exciting use-case for assumeutxo is the possibility of
> mobile
> devices functioning as fully validating nodes with access to the complete
> UTXO
> set (as an alternative to SPV models). The total resource burden needed to
> start a node
> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> + blocks_to_tip * 4MB) download and a few minutes of processing time,
> which sounds
> manageable for many mobile devices currently in use.
>
> A mobile user could initialize an assumed-valid bitcoin node within an
> hour,
> transact immediately, and complete a pruned full validation of their
> assumed-valid chain over the next few days, perhaps only doing the
> background
> IBD when their device has access to suitable high-bandwidth connections.
>
> If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> the road, it's easy to imagine an analogous process that would allow very
> fast
> startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
>
> ---
>
> I've created a related issue at our Github repository here:
>   https://github.com/bitcoin/bitcoin/issues/15605
>
> and have submitted a draft implementation of snapshot usage via RPC here:
>   https://github.com/bitcoin/bitcoin/pull/15606
>
> I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> plans for deployment steps should be discussed in the Github issue, and
> after all
> that my implementation may be reviewed as a sketch of the specific software
> changes necessary.
>
> Regards,
> James
>
>
> [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
> 2.10GHz
> [3]:
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> [4]: Marco Falke is due credit for this point
> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188
>
>

[-- Attachment #2: Type: text/html, Size: 8830 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Dave Scotese]

[-- Attachment #1: Type: text/plain, Size: 2791 bytes --]

No piece of data that does have significance to the Bitcoin consensus can
be memorable because it occurs (about) every ten minutes. In order to get
something memorable to provide sanity (let's say, anti-sybil-attack)
checking, it has to be rare, but recurrent.  The opportunity is actually
already there, but it usually goes by without providing the benefits.

For example, I found this blog post
<http://www.righto.com/2014/02/ascii-bernanke-wikileaks-photographs.html>
by Ken Shirriff who describes artifacts that can be found in the
blockchain. These artifacts are not intimately tied to their location in
the blockchain, so anyone building an alternative blockchain can relatively
easily add the artifacts with the same timestamp and at the same height,
masking the counterfeit.  In order to prevent that, the memorable thing has
to be intimately tied to work-intensive results, like the ratio of the hash
to the target.  Nelson Mandela's image appearing in the blockchain does NOT
prove to me it's the blockchain I can see at blockchain.com right now, but
if the smallest block hash in that blockchain, on 12/13/13, after all the
zeroes, starts with 3da1 (144 * 65536 times as much work) and is one of the
three block hashes from that day that have two occurrences of a double-e
(about 256 times more work), then it will.  The problem is that I'll
probably forget most of those details - but not that Mandela's image went
in the blockchain near the end of 2013.

On Sat, Apr 13, 2019 at 12:09 PM Peter Todd <pete@petertodd.org> wrote:

> On Wed, Apr 03, 2019 at 02:39:32PM -0700, Dave Scotese via bitcoin-dev
> wrote:
> > Every block's hash is smaller than the difficulty at that time.  Block
> > 569927's hash was VERY small (started with 21 zeros).  The ratio of block
> > hash to difficulty requirement (0xffffffff - difficulty, I think) could
> be
> > used to identify blocks as "special," thus providing the opportunity to
> > popularize unimportant but memorable-and-therefore-useful details.  How
> can
> > they be useful if they are unimportant?  They are useful for sanity
> > checking.  For example, if the drunken bishop walk (or some other popular
> > randomart) produced by block 569927's hash looked like a face, that would
> > be memorable: "The block with the smallest hash in 2019 (maybe ever?)
> looks
> > like a face after the drunken bishop walk."
>
> As hashest smaller than the target have no significance to the Bitcoin
> consensus I'd suggest not basing any features on that property. It's just
> as
> arbitrary as picking whole decimal number block heights, yet has the
> additional
> downsides of being harder to compute, and being likely to confuse people
> as to
> how the Bitcoin consensus works.
>
> --
> https://petertodd.org 'peter'[:-1]@petertodd.org
>

[-- Attachment #2: Type: text/html, Size: 3453 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Omar Shibli]

[-- Attachment #1: Type: text/plain, Size: 6832 bytes --]

This sounds really promising to me, I think it could seriously improve the
current SPV trust model.

In abstract these are the possible setups today:

Full node: All history, 100% monetary sovereignty.

SPV: fancy term to Electrum trust model, random selection of nodes, with
full delegation of monetary responsibility.

I think in that spirit a hybrid approach of full node + spv.

As follows:
Hardware spv with only genesis hash block seeded, as a safe bootstrap, from
there only headers is needed for validation, and ongoing new fresh blocks
and associated historic blocks for conducting transactions.

On Wed, Apr 3, 2019 at 2:43 AM James O'Beirne via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi,
>
> I'd like to discuss assumeutxo, which is an appealing and simple
> optimization in the spirit of assumevalid[0].
>
> # Motivation
>
> To start a fully validating bitcoin client from scratch, that client
> currently
> needs to perform an initial block download. To the surprise of no one, IBD
> takes a linear amount time based on the length of the chain's history. For
> clients running on modest hardware under limited bandwidth constraints,
> say a mobile device, completing IBD takes a considerable amount of time
> and thus poses serious usability challenges.
>
> As a result, having fully validating clients run on such hardware is rare
> and
> basically unrealistic. Clients with even moderate resource constraints
> are encouraged to rely on the SPV trust model. Though we have promising
> improvements to existing SPV modes pending deployment[1], it's worth
> thinking about a mechanism that would allow such clients to use trust
> models closer to full validation.
>
> The subject of this mail is a proposal for a complementary alternative to
> SPV
> modes, and which is in the spirit of an existing default, `assumevalid`.
> It may
> help modest clients transact under a security model that closely resembles
> full validation within minutes instead of hours or days.
>
> # assumeutxo
>
> The basic idea is to allow nodes to initialize using a serialized version
> of the
> UTXO set rendered by another node at some predetermined height. The
> initializing node syncs the headers chain from the network, then obtains
> and
> loads one of these UTXO snapshots (i.e. a serialized version of the UTXO
> set
> bundled with the block header indicating its "base" and some other
> metadata).
>
> Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> download of the 3.2GB snapshot[2].
>
> The node then syncs to the network tip and afterwards begins a simultaneous
> background validation (i.e., a conventional IBD) up to the base height of
> the
> snapshot in order to achieve full validation. Crucially, even while the
> background validation is happening the node can validate incoming blocks
> and
> transact with the benefit of the full (assumed-valid) UTXO set.
>
> Snapshots could be obtained from multiple separate peers in the same
> manner as
> block download, but I haven't put much thought into this. In concept it
> doesn't
> matter too much where the snapshots come from since their validity is
> determined via content hash.
>
> # Security
>
> Obviously there are some security implications due consideration. While
> this
> proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> Under assumevalid, a user can be tricked into transacting under a false
> history
> if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> parameter, sybils their node, and then feeds them a bogus chain
> encompassing
> all of the hard-coded checkpoints[3].
>
> The same attack is made easier in assumeutxo because, unlike in
> assumevalid,
> the attacker need not construct a valid PoW chain to get the victim's node
> into
> a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> false coin assignment.
>
> For this reason, I recommend that if we were to implement assumeutxo, we
> not
> allow its specification via commandline argument[4].
>
> Beyond this risk, I can't think of material differences in security
> relative to
> assumevalid, though I appeal to the list for help with this.
>
> # More fully validating clients
>
> A particularly exciting use-case for assumeutxo is the possibility of
> mobile
> devices functioning as fully validating nodes with access to the complete
> UTXO
> set (as an alternative to SPV models). The total resource burden needed to
> start a node
> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> + blocks_to_tip * 4MB) download and a few minutes of processing time,
> which sounds
> manageable for many mobile devices currently in use.
>
> A mobile user could initialize an assumed-valid bitcoin node within an
> hour,
> transact immediately, and complete a pruned full validation of their
> assumed-valid chain over the next few days, perhaps only doing the
> background
> IBD when their device has access to suitable high-bandwidth connections.
>
> If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> the road, it's easy to imagine an analogous process that would allow very
> fast
> startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
>
> ---
>
> I've created a related issue at our Github repository here:
>   https://github.com/bitcoin/bitcoin/issues/15605
>
> and have submitted a draft implementation of snapshot usage via RPC here:
>   https://github.com/bitcoin/bitcoin/pull/15606
>
> I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> plans for deployment steps should be discussed in the Github issue, and
> after all
> that my implementation may be reviewed as a sketch of the specific software
> changes necessary.
>
> Regards,
> James
>
>
> [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
> 2.10GHz
> [3]:
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> [4]: Marco Falke is due credit for this point
> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-- 
Sent from Gmail Mobile

[-- Attachment #2: Type: text/html, Size: 9343 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

On Wed, Apr 03, 2019 at 02:39:32PM -0700, Dave Scotese via bitcoin-dev wrote:
> Every block's hash is smaller than the difficulty at that time.  Block
> 569927's hash was VERY small (started with 21 zeros).  The ratio of block
> hash to difficulty requirement (0xffffffff - difficulty, I think) could be
> used to identify blocks as "special," thus providing the opportunity to
> popularize unimportant but memorable-and-therefore-useful details.  How can
> they be useful if they are unimportant?  They are useful for sanity
> checking.  For example, if the drunken bishop walk (or some other popular
> randomart) produced by block 569927's hash looked like a face, that would
> be memorable: "The block with the smallest hash in 2019 (maybe ever?) looks
> like a face after the drunken bishop walk."

As hashest smaller than the target have no significance to the Bitcoin
consensus I'd suggest not basing any features on that property. It's just as
arbitrary as picking whole decimal number block heights, yet has the additional
downsides of being harder to compute, and being likely to confuse people as to
how the Bitcoin consensus works.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[James O'Beirne]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

I recommend that anyone following this thread read through the recent IRC
exchange between Greg Maxwell and Luke Dashjr:
 http://www.erisian.com.au/bitcoin-core-dev/log-2019-04-04.html
<http://www.erisian.com.au/bitcoin-core-dev/log-2019-04-04.html>

The conversation starts on line 205 at 2019-04-04T02:54:50.

On Thu, Apr 4, 2019 at 2:38 AM Jim Posen via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Learning C++ is something within everyone's capability. Even people who do
>> not
>> wish to learn it can hire someone to perform review for them.
>>
>
> Anyone with enough knowledge of C++ to audit the entire the Bitcoin Core
> codebase is more than capable of running it with assumeutxo disabled and
> checking the hard-coded vale themself.
>
>> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 1911 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Jim Posen]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

>
> Learning C++ is something within everyone's capability. Even people who do
> not
> wish to learn it can hire someone to perform review for them.
>

Anyone with enough knowledge of C++ to audit the entire the Bitcoin Core
codebase is more than capable of running it with assumeutxo disabled and
checking the hard-coded vale themself.

>

[-- Attachment #2: Type: text/html, Size: 737 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Ethan Scruples]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

> This is exactly the danger. UTXO snapshots are NOT an alternative to a
real IBD. There are HUGE security implications for this.

This is a perfect example of what I am talking about when I say that people
do not appear to notice that there is no important security implication to
be found here.

If there are huge security implications for this, then I am keen to hear
them. In the scenario I have described, what advantage does Bob have over
Alice? What actionable information has Bob gained, and what is the action
he can take with it in hand? What value does Bob receive in return for the
electricity he has spent validating the previous blocks? I cannot find any,
but I am open to hearing the answer, and I think others would benefit from
knowing it as well.

On Wed, Apr 3, 2019 at 10:49 PM Luke Dashjr <luke@dashjr.org> wrote:

> On Wednesday 03 April 2019 15:39:29 Ethan Scruples via bitcoin-dev wrote:
> > If we can get mandatory UTXO commitments soft forked into Bitcoin, we get
> > the advantage of a non-growing IBD,
>
> No, we don't. This is exactly the danger. UTXO snapshots are NOT an
> alternative to a real IBD. There are HUGE security implications for this.
> Frankly, the danger that someone would do such a thing is itself a good
> reason not to ever add UTXO commitments.
>
> Luke
>

[-- Attachment #2: Type: text/html, Size: 1698 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Luke Dashjr]

On Wednesday 03 April 2019 21:39:32 Dave Scotese via bitcoin-dev wrote:
> Luke's comment that it could "lead to users trusting third parties (like
> developers) way too much" is pertinent too, but I think an honest abatement
> of that concern is impossible without teaching everyone C++.

Learning C++ is something within everyone's capability. Even people who do not 
wish to learn it can hire someone to perform review for them.

> "Developers" 
> as an open group (anyone can fork the github repo, find a problem, and make
> an issue) deserve the trust we put in them, and that's because they're
> accountable (any such error found in the repo will have been put there by
> someone). 

No, we are not. We explicitly disclaim any warranty, and do not want your 
trust.

> The same thing goes for making it possible to download (*not 
> just the compiled software*, but) the entire UTXO Set if a commitment of it
> is hardcoded into the software, as James suggests. 

Verifying a UTXO set commitment is impossible short of a real IBD. It's not 
even comparable.

> We all trust 
> "developers" like that, and it's okay.

No, it isn't okay. There are plenty of fiat options if you want a trust-based 
currency. Bitcoin is supposed to be something more than that.

Luke

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Luke Dashjr]

On Wednesday 03 April 2019 15:39:29 Ethan Scruples via bitcoin-dev wrote:
> If we can get mandatory UTXO commitments soft forked into Bitcoin, we get
> the advantage of a non-growing IBD,

No, we don't. This is exactly the danger. UTXO snapshots are NOT an 
alternative to a real IBD. There are HUGE security implications for this. 
Frankly, the danger that someone would do such a thing is itself a good 
reason not to ever add UTXO commitments.

Luke

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Jim Posen]

[-- Attachment #1: Type: text/plain, Size: 7457 bytes --]

Big Concept ACK. I think this would be one of the biggest usability
improvements for Bitcoin and I see no security issues with the assumevalid
approach. I also agree that it's important to start work on this even
before the ultimate, perfect accumulator has been designed/tested and the
commitment scheme can always be upgraded later on. assumeutxo syncing
actually seems pretty orthogonal to the accumulator research.

I have a few questions

- So any nodes that do an initial sync will stop at the assumeutxo height,
serialize a snapshot of the chain state and store it? How many nodes are
expected to do this? Any idea how long this takes? Should it be enabled by
default?
- Would pruned nodes still download all historic blocks to double-check the
snapshot or only full nodes that intend to serve block data?
- How long are old snapshots retained? Presumably during a new release
nodes should keep at least a version back. Without P2P signalling of which
snapshots are available, they maybe have to keep all old snapshots or even
download old ones.

and comments

- The snapshot should probably be chunked up to minimize the amount of
bandwidth/IO/memory a malicious node could waste before you realize. Also,
it would make parallel downloading easier.

On Tue, Apr 2, 2019 at 4:43 PM James O'Beirne via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi,
>
> I'd like to discuss assumeutxo, which is an appealing and simple
> optimization in the spirit of assumevalid[0].
>
> # Motivation
>
> To start a fully validating bitcoin client from scratch, that client
> currently
> needs to perform an initial block download. To the surprise of no one, IBD
> takes a linear amount time based on the length of the chain's history. For
> clients running on modest hardware under limited bandwidth constraints,
> say a mobile device, completing IBD takes a considerable amount of time
> and thus poses serious usability challenges.
>
> As a result, having fully validating clients run on such hardware is rare
> and
> basically unrealistic. Clients with even moderate resource constraints
> are encouraged to rely on the SPV trust model. Though we have promising
> improvements to existing SPV modes pending deployment[1], it's worth
> thinking about a mechanism that would allow such clients to use trust
> models closer to full validation.
>
> The subject of this mail is a proposal for a complementary alternative to
> SPV
> modes, and which is in the spirit of an existing default, `assumevalid`.
> It may
> help modest clients transact under a security model that closely resembles
> full validation within minutes instead of hours or days.
>
> # assumeutxo
>
> The basic idea is to allow nodes to initialize using a serialized version
> of the
> UTXO set rendered by another node at some predetermined height. The
> initializing node syncs the headers chain from the network, then obtains
> and
> loads one of these UTXO snapshots (i.e. a serialized version of the UTXO
> set
> bundled with the block header indicating its "base" and some other
> metadata).
>
> Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> download of the 3.2GB snapshot[2].
>
> The node then syncs to the network tip and afterwards begins a simultaneous
> background validation (i.e., a conventional IBD) up to the base height of
> the
> snapshot in order to achieve full validation. Crucially, even while the
> background validation is happening the node can validate incoming blocks
> and
> transact with the benefit of the full (assumed-valid) UTXO set.
>
> Snapshots could be obtained from multiple separate peers in the same
> manner as
> block download, but I haven't put much thought into this. In concept it
> doesn't
> matter too much where the snapshots come from since their validity is
> determined via content hash.
>
> # Security
>
> Obviously there are some security implications due consideration. While
> this
> proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> Under assumevalid, a user can be tricked into transacting under a false
> history
> if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> parameter, sybils their node, and then feeds them a bogus chain
> encompassing
> all of the hard-coded checkpoints[3].
>
> The same attack is made easier in assumeutxo because, unlike in
> assumevalid,
> the attacker need not construct a valid PoW chain to get the victim's node
> into
> a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> false coin assignment.
>
> For this reason, I recommend that if we were to implement assumeutxo, we
> not
> allow its specification via commandline argument[4].
>
> Beyond this risk, I can't think of material differences in security
> relative to
> assumevalid, though I appeal to the list for help with this.
>
> # More fully validating clients
>
> A particularly exciting use-case for assumeutxo is the possibility of
> mobile
> devices functioning as fully validating nodes with access to the complete
> UTXO
> set (as an alternative to SPV models). The total resource burden needed to
> start a node
> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> + blocks_to_tip * 4MB) download and a few minutes of processing time,
> which sounds
> manageable for many mobile devices currently in use.
>
> A mobile user could initialize an assumed-valid bitcoin node within an
> hour,
> transact immediately, and complete a pruned full validation of their
> assumed-valid chain over the next few days, perhaps only doing the
> background
> IBD when their device has access to suitable high-bandwidth connections.
>
> If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> the road, it's easy to imagine an analogous process that would allow very
> fast
> startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
>
> ---
>
> I've created a related issue at our Github repository here:
>   https://github.com/bitcoin/bitcoin/issues/15605
>
> and have submitted a draft implementation of snapshot usage via RPC here:
>   https://github.com/bitcoin/bitcoin/pull/15606
>
> I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> plans for deployment steps should be discussed in the Github issue, and
> after all
> that my implementation may be reviewed as a sketch of the specific software
> changes necessary.
>
> Regards,
> James
>
>
> [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
> 2.10GHz
> [3]:
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> [4]: Marco Falke is due credit for this point
> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 9769 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Dave Scotese]

[-- Attachment #1: Type: text/plain, Size: 12666 bytes --]

Every block's hash is smaller than the difficulty at that time.  Block
569927's hash was VERY small (started with 21 zeros).  The ratio of block
hash to difficulty requirement (0xffffffff - difficulty, I think) could be
used to identify blocks as "special," thus providing the opportunity to
popularize unimportant but memorable-and-therefore-useful details.  How can
they be useful if they are unimportant?  They are useful for sanity
checking.  For example, if the drunken bishop walk (or some other popular
randomart) produced by block 569927's hash looked like a face, that would
be memorable: "The block with the smallest hash in 2019 (maybe ever?) looks
like a face after the drunken bishop walk."

If a few of these showed up each year, then Bob and/or Alice would have a
good chance of seeing that something was wrong if and when they checked.
It would not be surprising, given Ethan's assumption that the invalid block
Bob found contributed to Alice's UTXOs, that at some point, the history one
of them has would be missing the memorable things beginning at some block
height because, clearly, one of them has been forked.

Luke's comment that it could "lead to users trusting third parties (like
developers) way too much" is pertinent too, but I think an honest abatement
of that concern is impossible without teaching everyone C++.  "Developers"
as an open group (anyone can fork the github repo, find a problem, and make
an issue) deserve the trust we put in them, and that's because they're
accountable (any such error found in the repo will have been put there by
someone).  The same thing goes for making it possible to download (*not
just the compiled software*, but) the entire UTXO Set if a commitment of it
is hardcoded into the software, as James suggests.  We all trust
"developers" like that, and it's okay.  No one holds the "ring of power."

On Wed, Apr 3, 2019 at 8:39 AM Ethan Scruples via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Jonas,
>
> If we can get mandatory UTXO commitments soft forked into Bitcoin, we get
> the advantage of a non-growing IBD, which I think everyone would agree is a
> benefit that, uh, grows over time. The thing I do not see people noticing
> is that we actually pay little to no security price for this benefit.
>
> To see this, consider Alice, who starts from a UTXO snapshot made at
> current height - 50,000 and Bob who validates from genesis.
>
> After her partial validation, Alice is satisfied that she is in possession
> of the UTXO set-- she is in consensus with the rest of the network peers.
>
> However, Bob realizes that there is actually an invalid block at current
> height - 50,001.
>
> Three things to notice:
>
> 1. This scenario essentially cannot happen. There is no way that the
> miners are going to stack 50,000 blocks on top of an invalid block without
> the economic majority abandoning the invalid chain.
>
> 2. If this scenario DOES happen, Bob has learned about it too late for it
> to matter to Bob. The blockchain Bob wants to be on is the one that
> everyone has been using for the last year, whether or not it is besmirched
> by an invalid block.
>
> 3. If this scenario DOES happen, and Bob DOES want to reject the last
> 50,000 mined blocks as invalid, he may discover to his dismay that in the 1
> year since the invalid block, mischievous entities have enough time to mine
> equally weighted alternative histories from the Genesis block forward to
> the invalid block, meaning that Bob has no way to use POW to come to
> consensus with other Bobs out there.
>
> On Wed, Apr 3, 2019 at 3:33 AM Jonas Schnelli via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Thanks James for the post.
>>
>> I proposed a similar idea [1] back in 2016 with the difference of signing
>> the UTXO-set hash in a gitian-ish way.
>>
>> While the idea of UTXO-set-syncs are attractive, there are probably still
>> significant downsides in usability (compared to models with less security),
>> mainly:
>> * Assume the UTXO set is 6 weeks old (which seems a reasonable age for
>> providing enough security) a peer using that snapshot would still require
>> to download and verify ~6048 blocks (~7.9GB at 1.3MB blocks,… probably
>> CPU-days on a phone)
>> * Do we semi-trust the peer that servers the UTXO set (compared to a
>> block or tx which we can validate)? What channel to we use to serve the
>> snapshot?
>>
>> If the goal is to run a full node on a consumer device that is also been
>> used for other CPU intense operations (like a phone, etc.), I’m not sure if
>> this proposal will lead to a satisfactory user experience.
>>
>> The longer I think around this problem, the more I lean towards accepting
>> the fact that one need to use dedicated hardware in his own environment to
>> perform a painless full validation.
>>
>> /jonas
>>
>> [1]
>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-February/012478.html
>>
>> > Am 02.04.2019 um 22:43 schrieb James O'Beirne via bitcoin-dev <
>> bitcoin-dev@lists.linuxfoundation.org>:
>> >
>> > Hi,
>> >
>> > I'd like to discuss assumeutxo, which is an appealing and simple
>> > optimization in the spirit of assumevalid[0].
>> >
>> > # Motivation
>> >
>> > To start a fully validating bitcoin client from scratch, that client
>> currently
>> > needs to perform an initial block download. To the surprise of no one,
>> IBD
>> > takes a linear amount time based on the length of the chain's history.
>> For
>> > clients running on modest hardware under limited bandwidth constraints,
>> > say a mobile device, completing IBD takes a considerable amount of time
>> > and thus poses serious usability challenges.
>> >
>> > As a result, having fully validating clients run on such hardware is
>> rare and
>> > basically unrealistic. Clients with even moderate resource constraints
>> > are encouraged to rely on the SPV trust model. Though we have promising
>> > improvements to existing SPV modes pending deployment[1], it's worth
>> > thinking about a mechanism that would allow such clients to use trust
>> > models closer to full validation.
>> >
>> > The subject of this mail is a proposal for a complementary alternative
>> to SPV
>> > modes, and which is in the spirit of an existing default,
>> `assumevalid`. It may
>> > help modest clients transact under a security model that closely
>> resembles
>> > full validation within minutes instead of hours or days.
>> >
>> > # assumeutxo
>> >
>> > The basic idea is to allow nodes to initialize using a serialized
>> version of the
>> > UTXO set rendered by another node at some predetermined height. The
>> > initializing node syncs the headers chain from the network, then
>> obtains and
>> > loads one of these UTXO snapshots (i.e. a serialized version of the
>> UTXO set
>> > bundled with the block header indicating its "base" and some other
>> metadata).
>> >
>> > Based upon the snapshot, the node is able to quickly reconstruct its
>> chainstate,
>> > and compares a hash of the resulting UTXO set to a preordained hash
>> hard-coded
>> > in the software a la assumevalid. This all takes ~23 minutes, not
>> accounting for
>> > download of the 3.2GB snapshot[2].
>> >
>> > The node then syncs to the network tip and afterwards begins a
>> simultaneous
>> > background validation (i.e., a conventional IBD) up to the base height
>> of the
>> > snapshot in order to achieve full validation. Crucially, even while the
>> > background validation is happening the node can validate incoming
>> blocks and
>> > transact with the benefit of the full (assumed-valid) UTXO set.
>> >
>> > Snapshots could be obtained from multiple separate peers in the same
>> manner as
>> > block download, but I haven't put much thought into this. In concept it
>> doesn't
>> > matter too much where the snapshots come from since their validity is
>> > determined via content hash.
>> >
>> > # Security
>> >
>> > Obviously there are some security implications due consideration. While
>> this
>> > proposal is in the spirit of assumevalid, practical attacks may become
>> easier.
>> > Under assumevalid, a user can be tricked into transacting under a false
>> history
>> > if an attacker convinces them to start bitcoind with a malicious
>> `-assumevalid`
>> > parameter, sybils their node, and then feeds them a bogus chain
>> encompassing
>> > all of the hard-coded checkpoints[3].
>> >
>> > The same attack is made easier in assumeutxo because, unlike in
>> assumevalid,
>> > the attacker need not construct a valid PoW chain to get the victim's
>> node into
>> > a false state; they simply need to get the user to accept a bad
>> `-assumeutxo`
>> > parameter and then supply them an easily made UTXO snapshot containing,
>> say, a
>> > false coin assignment.
>> >
>> > For this reason, I recommend that if we were to implement assumeutxo,
>> we not
>> > allow its specification via commandline argument[4].
>> >
>> > Beyond this risk, I can't think of material differences in security
>> relative to
>> > assumevalid, though I appeal to the list for help with this.
>> >
>> > # More fully validating clients
>> >
>> > A particularly exciting use-case for assumeutxo is the possibility of
>> mobile
>> > devices functioning as fully validating nodes with access to the
>> complete UTXO
>> > set (as an alternative to SPV models). The total resource burden needed
>> to start a node
>> > from scratch based on a snapshot is, at time of writing, a ~(3.2GB
>> > + blocks_to_tip * 4MB) download and a few minutes of processing time,
>> which sounds
>> > manageable for many mobile devices currently in use.
>> >
>> > A mobile user could initialize an assumed-valid bitcoin node within an
>> hour,
>> > transact immediately, and complete a pruned full validation of their
>> > assumed-valid chain over the next few days, perhaps only doing the
>> background
>> > IBD when their device has access to suitable high-bandwidth connections.
>> >
>> > If we end up implementing an accumulator-based UTXO scaling
>> design[5][6] down
>> > the road, it's easy to imagine an analogous process that would allow
>> very fast
>> > startup using an accumulator of a few kilobytes in lieu of a multi-GB
>> snapshot.
>> >
>> > ---
>> >
>> > I've created a related issue at our Github repository here:
>> >   https://github.com/bitcoin/bitcoin/issues/15605
>> >
>> > and have submitted a draft implementation of snapshot usage via RPC
>> here:
>> >   https://github.com/bitcoin/bitcoin/pull/15606
>> >
>> > I'd like to discuss here whether this is a good fit for Bitcoin
>> conceptually. Concrete
>> > plans for deployment steps should be discussed in the Github issue, and
>> after all
>> > that my implementation may be reviewed as a sketch of the specific
>> software
>> > changes necessary.
>> >
>> > Regards,
>> > James
>> >
>> >
>> > [0]:
>> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
>> > [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
>> > [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116
>> CPU @ 2.10GHz
>> > [3]:
>> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
>> > [4]: Marco Falke is due credit for this point
>> > [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
>> > [6]: Boneh, Bunz, Fisch on accumulators:
>> https://eprint.iacr.org/2018/1188
>> >
>> > _______________________________________________
>> > bitcoin-dev mailing list
>> > bitcoin-dev@lists.linuxfoundation.org
>> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>


-- 
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy <http://www.litmocracy.com> and Meme Racing
<http://www.memeracing.net> (in alpha).
I'm the webmaster for The Voluntaryist <http://www.voluntaryist.com> which
now accepts Bitcoin.
I also code for The Dollar Vigilante <http://dollarvigilante.com/>.
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto

[-- Attachment #2: Type: text/html, Size: 15585 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[James O'Beirne]

[-- Attachment #1: Type: text/plain, Size: 9754 bytes --]

Thanks for the reply, Jonas. I should've figured someone had hit the
mailing list with this one before!

In hindsight, I may have overemphasized the use of this for low-powered
mobile devices. Indeed I think this may also be a worthwhile optimization
for common hardware too.

On the margin, if a user wants to interact with Bitcoin they will download
software that allows them to do it immediately - this results in many
people defaulting to a light client. If Bitcoin were able to initialize
from scratch in a comparable amount of time and then populate the full
chain in the background, we may have many more people *incidentally*
running full nodes.

Regardless of whether or not we use UTXO snapshots per se, I'd argue that
the pattern of doing some kind of quick initialization (whether it's with
assumed-valid data, or headers-contingent data like BIP157) and then
performing full validation in the background is a good way to ensure that
we have a healthier population of full nodes than we would otherwise.

For this reason, and (as Ethan points out) because IBD's linear setup time
is infeasible in the long-term, I think this pattern is an obvious
direction for the bitcoin client to go.

> * Do we semi-trust the peer that servers the UTXO set (compared to a
block or tx which we can validate)? What channel to we use to serve the
snapshot?

As you note in your post from 2016, where and how we retrieve the snapshot
is more or less immaterial because we compare a hash of its contents to a
previously specified value that the code ships with (the `assumeutxo`
hash). We don't need to trust the source serving it to us, although
bandwidth DoS prevention via some kind chunked delivery from peers would be
worth thinking about.

Regards,
James

On Wed, Apr 3, 2019 at 2:37 AM Jonas Schnelli <dev@jonasschnelli.ch> wrote:

> Thanks James for the post.
>
> I proposed a similar idea [1] back in 2016 with the difference of signing
> the UTXO-set hash in a gitian-ish way.
>
> While the idea of UTXO-set-syncs are attractive, there are probably still
> significant downsides in usability (compared to models with less security),
> mainly:
> * Assume the UTXO set is 6 weeks old (which seems a reasonable age for
> providing enough security) a peer using that snapshot would still require
> to download and verify ~6048 blocks (~7.9GB at 1.3MB blocks,… probably
> CPU-days on a phone)
> * Do we semi-trust the peer that servers the UTXO set (compared to a block
> or tx which we can validate)? What channel to we use to serve the snapshot?
>
> If the goal is to run a full node on a consumer device that is also been
> used for other CPU intense operations (like a phone, etc.), I’m not sure if
> this proposal will lead to a satisfactory user experience.
>
> The longer I think around this problem, the more I lean towards accepting
> the fact that one need to use dedicated hardware in his own environment to
> perform a painless full validation.
>
> /jonas
>
> [1]
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-February/012478.html
>
> > Am 02.04.2019 um 22:43 schrieb James O'Beirne via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org>:
> >
> > Hi,
> >
> > I'd like to discuss assumeutxo, which is an appealing and simple
> > optimization in the spirit of assumevalid[0].
> >
> > # Motivation
> >
> > To start a fully validating bitcoin client from scratch, that client
> currently
> > needs to perform an initial block download. To the surprise of no one,
> IBD
> > takes a linear amount time based on the length of the chain's history.
> For
> > clients running on modest hardware under limited bandwidth constraints,
> > say a mobile device, completing IBD takes a considerable amount of time
> > and thus poses serious usability challenges.
> >
> > As a result, having fully validating clients run on such hardware is
> rare and
> > basically unrealistic. Clients with even moderate resource constraints
> > are encouraged to rely on the SPV trust model. Though we have promising
> > improvements to existing SPV modes pending deployment[1], it's worth
> > thinking about a mechanism that would allow such clients to use trust
> > models closer to full validation.
> >
> > The subject of this mail is a proposal for a complementary alternative
> to SPV
> > modes, and which is in the spirit of an existing default, `assumevalid`.
> It may
> > help modest clients transact under a security model that closely
> resembles
> > full validation within minutes instead of hours or days.
> >
> > # assumeutxo
> >
> > The basic idea is to allow nodes to initialize using a serialized
> version of the
> > UTXO set rendered by another node at some predetermined height. The
> > initializing node syncs the headers chain from the network, then obtains
> and
> > loads one of these UTXO snapshots (i.e. a serialized version of the UTXO
> set
> > bundled with the block header indicating its "base" and some other
> metadata).
> >
> > Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> > and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> > in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> > download of the 3.2GB snapshot[2].
> >
> > The node then syncs to the network tip and afterwards begins a
> simultaneous
> > background validation (i.e., a conventional IBD) up to the base height
> of the
> > snapshot in order to achieve full validation. Crucially, even while the
> > background validation is happening the node can validate incoming blocks
> and
> > transact with the benefit of the full (assumed-valid) UTXO set.
> >
> > Snapshots could be obtained from multiple separate peers in the same
> manner as
> > block download, but I haven't put much thought into this. In concept it
> doesn't
> > matter too much where the snapshots come from since their validity is
> > determined via content hash.
> >
> > # Security
> >
> > Obviously there are some security implications due consideration. While
> this
> > proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> > Under assumevalid, a user can be tricked into transacting under a false
> history
> > if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> > parameter, sybils their node, and then feeds them a bogus chain
> encompassing
> > all of the hard-coded checkpoints[3].
> >
> > The same attack is made easier in assumeutxo because, unlike in
> assumevalid,
> > the attacker need not construct a valid PoW chain to get the victim's
> node into
> > a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> > parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> > false coin assignment.
> >
> > For this reason, I recommend that if we were to implement assumeutxo, we
> not
> > allow its specification via commandline argument[4].
> >
> > Beyond this risk, I can't think of material differences in security
> relative to
> > assumevalid, though I appeal to the list for help with this.
> >
> > # More fully validating clients
> >
> > A particularly exciting use-case for assumeutxo is the possibility of
> mobile
> > devices functioning as fully validating nodes with access to the
> complete UTXO
> > set (as an alternative to SPV models). The total resource burden needed
> to start a node
> > from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> > + blocks_to_tip * 4MB) download and a few minutes of processing time,
> which sounds
> > manageable for many mobile devices currently in use.
> >
> > A mobile user could initialize an assumed-valid bitcoin node within an
> hour,
> > transact immediately, and complete a pruned full validation of their
> > assumed-valid chain over the next few days, perhaps only doing the
> background
> > IBD when their device has access to suitable high-bandwidth connections.
> >
> > If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> > the road, it's easy to imagine an analogous process that would allow
> very fast
> > startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
> >
> > ---
> >
> > I've created a related issue at our Github repository here:
> >   https://github.com/bitcoin/bitcoin/issues/15605
> >
> > and have submitted a draft implementation of snapshot usage via RPC here:
> >   https://github.com/bitcoin/bitcoin/pull/15606
> >
> > I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> > plans for deployment steps should be discussed in the Github issue, and
> after all
> > that my implementation may be reviewed as a sketch of the specific
> software
> > changes necessary.
> >
> > Regards,
> > James
> >
> >
> > [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> > [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> > [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU
> @ 2.10GHz
> > [3]:
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> > [4]: Marco Falke is due credit for this point
> > [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> > [6]: Boneh, Bunz, Fisch on accumulators:
> https://eprint.iacr.org/2018/1188
> >
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 11871 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Ethan Scruples]

[-- Attachment #1: Type: text/plain, Size: 9752 bytes --]

Jonas,

If we can get mandatory UTXO commitments soft forked into Bitcoin, we get
the advantage of a non-growing IBD, which I think everyone would agree is a
benefit that, uh, grows over time. The thing I do not see people noticing
is that we actually pay little to no security price for this benefit.

To see this, consider Alice, who starts from a UTXO snapshot made at
current height - 50,000 and Bob who validates from genesis.

After her partial validation, Alice is satisfied that she is in possession
of the UTXO set-- she is in consensus with the rest of the network peers.

However, Bob realizes that there is actually an invalid block at current
height - 50,001.

Three things to notice:

1. This scenario essentially cannot happen. There is no way that the miners
are going to stack 50,000 blocks on top of an invalid block without the
economic majority abandoning the invalid chain.

2. If this scenario DOES happen, Bob has learned about it too late for it
to matter to Bob. The blockchain Bob wants to be on is the one that
everyone has been using for the last year, whether or not it is besmirched
by an invalid block.

3. If this scenario DOES happen, and Bob DOES want to reject the last
50,000 mined blocks as invalid, he may discover to his dismay that in the 1
year since the invalid block, mischievous entities have enough time to mine
equally weighted alternative histories from the Genesis block forward to
the invalid block, meaning that Bob has no way to use POW to come to
consensus with other Bobs out there.

On Wed, Apr 3, 2019 at 3:33 AM Jonas Schnelli via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Thanks James for the post.
>
> I proposed a similar idea [1] back in 2016 with the difference of signing
> the UTXO-set hash in a gitian-ish way.
>
> While the idea of UTXO-set-syncs are attractive, there are probably still
> significant downsides in usability (compared to models with less security),
> mainly:
> * Assume the UTXO set is 6 weeks old (which seems a reasonable age for
> providing enough security) a peer using that snapshot would still require
> to download and verify ~6048 blocks (~7.9GB at 1.3MB blocks,… probably
> CPU-days on a phone)
> * Do we semi-trust the peer that servers the UTXO set (compared to a block
> or tx which we can validate)? What channel to we use to serve the snapshot?
>
> If the goal is to run a full node on a consumer device that is also been
> used for other CPU intense operations (like a phone, etc.), I’m not sure if
> this proposal will lead to a satisfactory user experience.
>
> The longer I think around this problem, the more I lean towards accepting
> the fact that one need to use dedicated hardware in his own environment to
> perform a painless full validation.
>
> /jonas
>
> [1]
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-February/012478.html
>
> > Am 02.04.2019 um 22:43 schrieb James O'Beirne via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org>:
> >
> > Hi,
> >
> > I'd like to discuss assumeutxo, which is an appealing and simple
> > optimization in the spirit of assumevalid[0].
> >
> > # Motivation
> >
> > To start a fully validating bitcoin client from scratch, that client
> currently
> > needs to perform an initial block download. To the surprise of no one,
> IBD
> > takes a linear amount time based on the length of the chain's history.
> For
> > clients running on modest hardware under limited bandwidth constraints,
> > say a mobile device, completing IBD takes a considerable amount of time
> > and thus poses serious usability challenges.
> >
> > As a result, having fully validating clients run on such hardware is
> rare and
> > basically unrealistic. Clients with even moderate resource constraints
> > are encouraged to rely on the SPV trust model. Though we have promising
> > improvements to existing SPV modes pending deployment[1], it's worth
> > thinking about a mechanism that would allow such clients to use trust
> > models closer to full validation.
> >
> > The subject of this mail is a proposal for a complementary alternative
> to SPV
> > modes, and which is in the spirit of an existing default, `assumevalid`.
> It may
> > help modest clients transact under a security model that closely
> resembles
> > full validation within minutes instead of hours or days.
> >
> > # assumeutxo
> >
> > The basic idea is to allow nodes to initialize using a serialized
> version of the
> > UTXO set rendered by another node at some predetermined height. The
> > initializing node syncs the headers chain from the network, then obtains
> and
> > loads one of these UTXO snapshots (i.e. a serialized version of the UTXO
> set
> > bundled with the block header indicating its "base" and some other
> metadata).
> >
> > Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> > and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> > in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> > download of the 3.2GB snapshot[2].
> >
> > The node then syncs to the network tip and afterwards begins a
> simultaneous
> > background validation (i.e., a conventional IBD) up to the base height
> of the
> > snapshot in order to achieve full validation. Crucially, even while the
> > background validation is happening the node can validate incoming blocks
> and
> > transact with the benefit of the full (assumed-valid) UTXO set.
> >
> > Snapshots could be obtained from multiple separate peers in the same
> manner as
> > block download, but I haven't put much thought into this. In concept it
> doesn't
> > matter too much where the snapshots come from since their validity is
> > determined via content hash.
> >
> > # Security
> >
> > Obviously there are some security implications due consideration. While
> this
> > proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> > Under assumevalid, a user can be tricked into transacting under a false
> history
> > if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> > parameter, sybils their node, and then feeds them a bogus chain
> encompassing
> > all of the hard-coded checkpoints[3].
> >
> > The same attack is made easier in assumeutxo because, unlike in
> assumevalid,
> > the attacker need not construct a valid PoW chain to get the victim's
> node into
> > a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> > parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> > false coin assignment.
> >
> > For this reason, I recommend that if we were to implement assumeutxo, we
> not
> > allow its specification via commandline argument[4].
> >
> > Beyond this risk, I can't think of material differences in security
> relative to
> > assumevalid, though I appeal to the list for help with this.
> >
> > # More fully validating clients
> >
> > A particularly exciting use-case for assumeutxo is the possibility of
> mobile
> > devices functioning as fully validating nodes with access to the
> complete UTXO
> > set (as an alternative to SPV models). The total resource burden needed
> to start a node
> > from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> > + blocks_to_tip * 4MB) download and a few minutes of processing time,
> which sounds
> > manageable for many mobile devices currently in use.
> >
> > A mobile user could initialize an assumed-valid bitcoin node within an
> hour,
> > transact immediately, and complete a pruned full validation of their
> > assumed-valid chain over the next few days, perhaps only doing the
> background
> > IBD when their device has access to suitable high-bandwidth connections.
> >
> > If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> > the road, it's easy to imagine an analogous process that would allow
> very fast
> > startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
> >
> > ---
> >
> > I've created a related issue at our Github repository here:
> >   https://github.com/bitcoin/bitcoin/issues/15605
> >
> > and have submitted a draft implementation of snapshot usage via RPC here:
> >   https://github.com/bitcoin/bitcoin/pull/15606
> >
> > I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> > plans for deployment steps should be discussed in the Github issue, and
> after all
> > that my implementation may be reviewed as a sketch of the specific
> software
> > changes necessary.
> >
> > Regards,
> > James
> >
> >
> > [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> > [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> > [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU
> @ 2.10GHz
> > [3]:
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> > [4]: Marco Falke is due credit for this point
> > [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> > [6]: Boneh, Bunz, Fisch on accumulators:
> https://eprint.iacr.org/2018/1188
> >
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 12059 bytes --]

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Luke Dashjr]

This would lead to users trusting third parties (like developers) way too 
much.

Furthermore, removing the ability for users to easily set it removes the one 
reasonable use case: where the user has already verified the state at some 
point previously, and saved the hash (ie, as backup of the UTXO set).

Luke


On Tuesday 02 April 2019 20:43:11 James O'Beirne via bitcoin-dev wrote:
> Hi,
>
> I'd like to discuss assumeutxo, which is an appealing and simple
> optimization in the spirit of assumevalid[0].
>
> # Motivation
>
> To start a fully validating bitcoin client from scratch, that client
> currently
> needs to perform an initial block download. To the surprise of no one, IBD
> takes a linear amount time based on the length of the chain's history. For
> clients running on modest hardware under limited bandwidth constraints,
> say a mobile device, completing IBD takes a considerable amount of time
> and thus poses serious usability challenges.
>
> As a result, having fully validating clients run on such hardware is rare
> and
> basically unrealistic. Clients with even moderate resource constraints
> are encouraged to rely on the SPV trust model. Though we have promising
> improvements to existing SPV modes pending deployment[1], it's worth
> thinking about a mechanism that would allow such clients to use trust
> models closer to full validation.
>
> The subject of this mail is a proposal for a complementary alternative to
> SPV
> modes, and which is in the spirit of an existing default, `assumevalid`. It
> may
> help modest clients transact under a security model that closely resembles
> full validation within minutes instead of hours or days.
>
> # assumeutxo
>
> The basic idea is to allow nodes to initialize using a serialized version
> of the
> UTXO set rendered by another node at some predetermined height. The
> initializing node syncs the headers chain from the network, then obtains
> and loads one of these UTXO snapshots (i.e. a serialized version of the
> UTXO set bundled with the block header indicating its "base" and some other
> metadata).
>
> Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> download of the 3.2GB snapshot[2].
>
> The node then syncs to the network tip and afterwards begins a simultaneous
> background validation (i.e., a conventional IBD) up to the base height of
> the
> snapshot in order to achieve full validation. Crucially, even while the
> background validation is happening the node can validate incoming blocks
> and transact with the benefit of the full (assumed-valid) UTXO set.
>
> Snapshots could be obtained from multiple separate peers in the same manner
> as
> block download, but I haven't put much thought into this. In concept it
> doesn't
> matter too much where the snapshots come from since their validity is
> determined via content hash.
>
> # Security
>
> Obviously there are some security implications due consideration. While
> this proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> Under assumevalid, a user can be tricked into transacting under a false
> history
> if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> parameter, sybils their node, and then feeds them a bogus chain
> encompassing all of the hard-coded checkpoints[3].
>
> The same attack is made easier in assumeutxo because, unlike in
> assumevalid, the attacker need not construct a valid PoW chain to get the
> victim's node into
> a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> false coin assignment.
>
> For this reason, I recommend that if we were to implement assumeutxo, we
> not allow its specification via commandline argument[4].
>
> Beyond this risk, I can't think of material differences in security
> relative to
> assumevalid, though I appeal to the list for help with this.
>
> # More fully validating clients
>
> A particularly exciting use-case for assumeutxo is the possibility of
> mobile devices functioning as fully validating nodes with access to the
> complete UTXO
> set (as an alternative to SPV models). The total resource burden needed to
> start a node
> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> + blocks_to_tip * 4MB) download and a few minutes of processing time, which
> sounds
> manageable for many mobile devices currently in use.
>
> A mobile user could initialize an assumed-valid bitcoin node within an
> hour, transact immediately, and complete a pruned full validation of their
> assumed-valid chain over the next few days, perhaps only doing the
> background
> IBD when their device has access to suitable high-bandwidth connections.
>
> If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> the road, it's easy to imagine an analogous process that would allow very
> fast
> startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
>
> ---
>
> I've created a related issue at our Github repository here:
>   https://github.com/bitcoin/bitcoin/issues/15605
>
> and have submitted a draft implementation of snapshot usage via RPC here:
>   https://github.com/bitcoin/bitcoin/pull/15606
>
> I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> plans for deployment steps should be discussed in the Github issue, and
> after all
> that my implementation may be reviewed as a sketch of the specific software
> changes necessary.
>
> Regards,
> James
>
>
> [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
> 2.10GHz
> [3]:
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L1
>61 [4]: Marco Falke is due credit for this point
> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188

Re: [bitcoin-dev] assumeutxo and UTXO snapshots

[Jonas Schnelli]

[-- Attachment #1: Type: text/plain, Size: 7462 bytes --]

Thanks James for the post.

I proposed a similar idea [1] back in 2016 with the difference of signing the UTXO-set hash in a gitian-ish way.

While the idea of UTXO-set-syncs are attractive, there are probably still significant downsides in usability (compared to models with less security), mainly:
* Assume the UTXO set is 6 weeks old (which seems a reasonable age for providing enough security) a peer using that snapshot would still require to download and verify ~6048 blocks (~7.9GB at 1.3MB blocks,… probably CPU-days on a phone)
* Do we semi-trust the peer that servers the UTXO set (compared to a block or tx which we can validate)? What channel to we use to serve the snapshot?

If the goal is to run a full node on a consumer device that is also been used for other CPU intense operations (like a phone, etc.), I’m not sure if this proposal will lead to a satisfactory user experience.

The longer I think around this problem, the more I lean towards accepting the fact that one need to use dedicated hardware in his own environment to perform a painless full validation.

/jonas

[1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-February/012478.html

> Am 02.04.2019 um 22:43 schrieb James O'Beirne via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>:
> 
> Hi,
> 
> I'd like to discuss assumeutxo, which is an appealing and simple
> optimization in the spirit of assumevalid[0].
> 
> # Motivation
> 
> To start a fully validating bitcoin client from scratch, that client currently
> needs to perform an initial block download. To the surprise of no one, IBD
> takes a linear amount time based on the length of the chain's history. For
> clients running on modest hardware under limited bandwidth constraints,
> say a mobile device, completing IBD takes a considerable amount of time
> and thus poses serious usability challenges.
> 
> As a result, having fully validating clients run on such hardware is rare and
> basically unrealistic. Clients with even moderate resource constraints
> are encouraged to rely on the SPV trust model. Though we have promising
> improvements to existing SPV modes pending deployment[1], it's worth
> thinking about a mechanism that would allow such clients to use trust
> models closer to full validation.
> 
> The subject of this mail is a proposal for a complementary alternative to SPV
> modes, and which is in the spirit of an existing default, `assumevalid`. It may
> help modest clients transact under a security model that closely resembles
> full validation within minutes instead of hours or days.
> 
> # assumeutxo
> 
> The basic idea is to allow nodes to initialize using a serialized version of the
> UTXO set rendered by another node at some predetermined height. The
> initializing node syncs the headers chain from the network, then obtains and
> loads one of these UTXO snapshots (i.e. a serialized version of the UTXO set
> bundled with the block header indicating its "base" and some other metadata).
> 
> Based upon the snapshot, the node is able to quickly reconstruct its chainstate,
> and compares a hash of the resulting UTXO set to a preordained hash hard-coded
> in the software a la assumevalid. This all takes ~23 minutes, not accounting for
> download of the 3.2GB snapshot[2].
> 
> The node then syncs to the network tip and afterwards begins a simultaneous
> background validation (i.e., a conventional IBD) up to the base height of the
> snapshot in order to achieve full validation. Crucially, even while the
> background validation is happening the node can validate incoming blocks and
> transact with the benefit of the full (assumed-valid) UTXO set.
> 
> Snapshots could be obtained from multiple separate peers in the same manner as
> block download, but I haven't put much thought into this. In concept it doesn't
> matter too much where the snapshots come from since their validity is
> determined via content hash.
> 
> # Security
> 
> Obviously there are some security implications due consideration. While this
> proposal is in the spirit of assumevalid, practical attacks may become easier.
> Under assumevalid, a user can be tricked into transacting under a false history
> if an attacker convinces them to start bitcoind with a malicious `-assumevalid`
> parameter, sybils their node, and then feeds them a bogus chain encompassing
> all of the hard-coded checkpoints[3].
> 
> The same attack is made easier in assumeutxo because, unlike in assumevalid,
> the attacker need not construct a valid PoW chain to get the victim's node into
> a false state; they simply need to get the user to accept a bad `-assumeutxo`
> parameter and then supply them an easily made UTXO snapshot containing, say, a
> false coin assignment.
> 
> For this reason, I recommend that if we were to implement assumeutxo, we not
> allow its specification via commandline argument[4].
> 
> Beyond this risk, I can't think of material differences in security relative to
> assumevalid, though I appeal to the list for help with this.
> 
> # More fully validating clients
> 
> A particularly exciting use-case for assumeutxo is the possibility of mobile
> devices functioning as fully validating nodes with access to the complete UTXO
> set (as an alternative to SPV models). The total resource burden needed to start a node
> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> + blocks_to_tip * 4MB) download and a few minutes of processing time, which sounds
> manageable for many mobile devices currently in use.
> 
> A mobile user could initialize an assumed-valid bitcoin node within an hour,
> transact immediately, and complete a pruned full validation of their
> assumed-valid chain over the next few days, perhaps only doing the background
> IBD when their device has access to suitable high-bandwidth connections.
> 
> If we end up implementing an accumulator-based UTXO scaling design[5][6] down
> the road, it's easy to imagine an analogous process that would allow very fast
> startup using an accumulator of a few kilobytes in lieu of a multi-GB snapshot.
> 
> ---
> 
> I've created a related issue at our Github repository here:
>   https://github.com/bitcoin/bitcoin/issues/15605
> 
> and have submitted a draft implementation of snapshot usage via RPC here:
>   https://github.com/bitcoin/bitcoin/pull/15606
> 
> I'd like to discuss here whether this is a good fit for Bitcoin conceptually. Concrete
> plans for deployment steps should be discussed in the Github issue, and after all
> that my implementation may be reviewed as a sketch of the specific software
> changes necessary.
> 
> Regards,
> James
> 
> 
> [0]: https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @ 2.10GHz
> [3]: https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> [4]: Marco Falke is due credit for this point
> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bitcoin-dev] assumeutxo and UTXO snapshots

[James O'Beirne]

[-- Attachment #1: Type: text/plain, Size: 5622 bytes --]

Hi,

I'd like to discuss assumeutxo, which is an appealing and simple
optimization in the spirit of assumevalid[0].

# Motivation

To start a fully validating bitcoin client from scratch, that client
currently
needs to perform an initial block download. To the surprise of no one, IBD
takes a linear amount time based on the length of the chain's history. For
clients running on modest hardware under limited bandwidth constraints,
say a mobile device, completing IBD takes a considerable amount of time
and thus poses serious usability challenges.

As a result, having fully validating clients run on such hardware is rare
and
basically unrealistic. Clients with even moderate resource constraints
are encouraged to rely on the SPV trust model. Though we have promising
improvements to existing SPV modes pending deployment[1], it's worth
thinking about a mechanism that would allow such clients to use trust
models closer to full validation.

The subject of this mail is a proposal for a complementary alternative to
SPV
modes, and which is in the spirit of an existing default, `assumevalid`. It
may
help modest clients transact under a security model that closely resembles
full validation within minutes instead of hours or days.

# assumeutxo

The basic idea is to allow nodes to initialize using a serialized version
of the
UTXO set rendered by another node at some predetermined height. The
initializing node syncs the headers chain from the network, then obtains and
loads one of these UTXO snapshots (i.e. a serialized version of the UTXO set
bundled with the block header indicating its "base" and some other
metadata).

Based upon the snapshot, the node is able to quickly reconstruct its
chainstate,
and compares a hash of the resulting UTXO set to a preordained hash
hard-coded
in the software a la assumevalid. This all takes ~23 minutes, not
accounting for
download of the 3.2GB snapshot[2].

The node then syncs to the network tip and afterwards begins a simultaneous
background validation (i.e., a conventional IBD) up to the base height of
the
snapshot in order to achieve full validation. Crucially, even while the
background validation is happening the node can validate incoming blocks and
transact with the benefit of the full (assumed-valid) UTXO set.

Snapshots could be obtained from multiple separate peers in the same manner
as
block download, but I haven't put much thought into this. In concept it
doesn't
matter too much where the snapshots come from since their validity is
determined via content hash.

# Security

Obviously there are some security implications due consideration. While this
proposal is in the spirit of assumevalid, practical attacks may become
easier.
Under assumevalid, a user can be tricked into transacting under a false
history
if an attacker convinces them to start bitcoind with a malicious
`-assumevalid`
parameter, sybils their node, and then feeds them a bogus chain encompassing
all of the hard-coded checkpoints[3].

The same attack is made easier in assumeutxo because, unlike in assumevalid,
the attacker need not construct a valid PoW chain to get the victim's node
into
a false state; they simply need to get the user to accept a bad
`-assumeutxo`
parameter and then supply them an easily made UTXO snapshot containing,
say, a
false coin assignment.

For this reason, I recommend that if we were to implement assumeutxo, we not
allow its specification via commandline argument[4].

Beyond this risk, I can't think of material differences in security
relative to
assumevalid, though I appeal to the list for help with this.

# More fully validating clients

A particularly exciting use-case for assumeutxo is the possibility of mobile
devices functioning as fully validating nodes with access to the complete
UTXO
set (as an alternative to SPV models). The total resource burden needed to
start a node
from scratch based on a snapshot is, at time of writing, a ~(3.2GB
+ blocks_to_tip * 4MB) download and a few minutes of processing time, which
sounds
manageable for many mobile devices currently in use.

A mobile user could initialize an assumed-valid bitcoin node within an hour,
transact immediately, and complete a pruned full validation of their
assumed-valid chain over the next few days, perhaps only doing the
background
IBD when their device has access to suitable high-bandwidth connections.

If we end up implementing an accumulator-based UTXO scaling design[5][6]
down
the road, it's easy to imagine an analogous process that would allow very
fast
startup using an accumulator of a few kilobytes in lieu of a multi-GB
snapshot.

---

I've created a related issue at our Github repository here:
  https://github.com/bitcoin/bitcoin/issues/15605

and have submitted a draft implementation of snapshot usage via RPC here:
  https://github.com/bitcoin/bitcoin/pull/15606

I'd like to discuss here whether this is a good fit for Bitcoin
conceptually. Concrete
plans for deployment steps should be discussed in the Github issue, and
after all
that my implementation may be reviewed as a sketch of the specific software
changes necessary.

Regards,
James


[0]:
https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
[1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
[2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
2.10GHz
[3]:
https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
[4]: Marco Falke is due credit for this point
[5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
[6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188

[-- Attachment #2: Type: text/html, Size: 7473 bytes --]