[bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[Wladimir J. van der Laan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Binaries for bitcoin Core version 0.12.0rc1 are available from:

    https://bitcoin.org/bin/bitcoin-core-0.12.0/test/

Source code can be found on github under the signed tag

    https://github.com/bitcoin/bitcoin/tree/v0.12.0rc1

This is a release candidate for a new major version release, bringing new
features, bug fixes, as well as other improvements.

Preliminary release notes for the release can be found here:

    https://github.com/bitcoin/bitcoin/blob/0.12/doc/release-notes.md

Release candidates are test versions for releases. When no critical problems
are found, this release candidate will be tagged as 0.12.0.

Please report bugs using the issue tracker at github:

    https://github.com/bitcoin/bitcoin/issues

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJWm2fGAAoJEHSBCwEjRsmm274H/2BH3QD4AlJ87mQ8g6bzzv7h
S8m/EEDmpOuMgM6uF5PzWQ84yNfSyMItq7Y3cU8p9Fv+JD6ic1ZQPPQ0MQc3KtDx
EeF3wQ2iJe/ggBFcwrz0eIxfEOEo1mi5ooWMVSsnCKQU0IpMtq7ToMvhi/39ACnj
GsVRBJYlFoRCBh1LKkcyID7Fh7JstMgMrLEcrCy46T9h2EQEevlLydkwY26ENYUO
BasWXMaysdeKieO5S6tM6MD/50Bd19jHvjzvkeRY5+nZIdrNR1b5n7diCLEUa7b4
79oIqjdKF+4ns5Qgc+iVhIktthRyrHLrWxX7N8Ky+hSVj1OAKFZfdp4skgAzQUE=
=oVOV
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[xor]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On Sunday, January 17, 2016 11:08:08 AM Wladimir J. van der Laan via bitcoin-
dev wrote:
> Preliminary release notes for the release can be found here:
> 
>     https://github.com/bitcoin/bitcoin/blob/0.12/doc/release-notes.md


The part which lists raw Git pull requests says:
> #6057 ac5476e re-enable wallet in autoprune

But the main, handwritten part does not mention this.
Is pruning really finished, i.e. could I safely use it as a wallet "end-user"?

IMHO it would be one of the most interesting feature for users, as it could 
fix the issue of taking >60 GB of disk space.

So if it is finished, please mention that
- it's finished
- how to enable it.


Thanks for your hard work! :)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[Wladimir J. van der Laan]

On Sun, Jan 17, 2016 at 11:57:28PM +0100, xor--- via bitcoin-dev wrote:
> On Sunday, January 17, 2016 11:08:08 AM Wladimir J. van der Laan via bitcoin-
> dev wrote:
> > Preliminary release notes for the release can be found here:
> > 
> >     https://github.com/bitcoin/bitcoin/blob/0.12/doc/release-notes.md
> 
> 
> The part which lists raw Git pull requests says:
> > #6057 ac5476e re-enable wallet in autoprune
> 
> But the main, handwritten part does not mention this.
> Is pruning really finished, i.e. could I safely use it as a wallet "end-user"?

It has been tested in git for almost half a year. This RC is the first binary
release that contains the functionality.

It is extremely unlikely that the wallet will eat your coins (always backup
nevertheless), but I can't guarantee there won't be some issue where the wallet
and chain get out of sync and you're forced to redownload the blockchain.

> IMHO it would be one of the most interesting feature for users, as it could 
> fix the issue of taking >60 GB of disk space.
> 
> So if it is finished, please mention that
> - it's finished
> - how to enable it.

How to enable it is still the same as mentioned in the 0.11 release notes:
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#block-file-pruning

Wladimir

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[xor]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Monday, January 18, 2016 12:14:16 PM Wladimir J. van der Laan wrote:
> It has been tested in git for almost half a year. This RC is the first
> binary release that contains the functionality.
> 
> It is extremely unlikely that the wallet will eat your coins (always backup
> nevertheless), but I can't guarantee there won't be some issue where the
> wallet and chain get out of sync and you're forced to redownload the
> blockchain.

I think I asked the wrong way, sorry: My question was not really meant at 
whether it is bug-free (testing that is the purpose of a release candidate, so 
we of course don't know yet), but rather whether it is at least feature 
complete now.

Remember, the previous v0.11.0 release notes said:
> Block pruning is currently incompatible with running a wallet due to the
> fact that block data is used for rescanning the wallet and importing keys
> or addresses (which require a rescan.) However, running the wallet with
> block pruning will be supported in the near future, subject to those
> limitations.

So I'm interested whether this limitation has been lifted, and the whole 
feature is considered as finished.

If yes, I would highly recommend advertising it in the new release notes - as 
said, the disk space reduction is a big deal.


Thank you!

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[Wladimir J. van der Laan]

> 
> So I'm interested whether this limitation has been lifted, and the whole 
> feature is considered as finished.

Yes, it's exactly that limitation that has been lifted!

> If yes, I would highly recommend advertising it in the new release notes - as 
> said, the disk space reduction is a big deal.

Good idea, has been added by Marco Falke in commit fa31133,

Wladimir

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[xor]

[-- Attachment #1: Type: text/plain, Size: 2182 bytes --]

On Monday, January 25, 2016 01:03:17 PM Wladimir J. van der Laan wrote:
> > If yes, I would highly recommend advertising it in the new release notes -
> > as said, the disk space reduction is a big deal.
> 
> Good idea, has been added by Marco Falke in commit fa31133,

Thanks. The RC2 changelog now says:

> To enable block pruning set prune=<N> on the command line or in
> bitcoin.conf, where N is the number of MiB to allot for raw block & undo
> data.

From having read the Bitcoin whitepaper quite a few months ago ago, I have the 
very very basic understanding that pruning is meant to:
- delete old transaction data which merely "moves coins around"
- instead only store the "origin" (= block where coins were mined) and 
"current location" of the coins, i.e. the unspent transactions. Notably, I 
understood it as "this is as secure as storing everything, since we know where 
the coins were created, and where they are".

So from that point of view, I would assume that there is a "natural" amount of 
megabytes which a fully pruned blockchain consists of: It would be defined by 
the final amount of unspent coins.
I thereby am confused why it is possible to configure a number of megabytes 
"to allot for raw block & undo data". I would rather expect pruning just to be 
a boolean on/off flag, and the number of megabytes to be an automatically 
computed result from the natural size of the dataset.
And especially, I fear that I could set N too low, and as a result, it would 
delete "too much". I mean could this result in even security relevant 
transaction data being deleted?

Thus, it would be nice if you could yet once more edit the release notes to:
- explain why a N must be given
- what a "safe" value of N is. I.e. how large must N be at least to not delete 
security-relevant stuff?
- maybe mention if there is a "auto" setting for N to ensure that it choses a 
safe value on its own?

Sorry if my descriptions are from a layman's point of view. I intentionally 
did *not* re-read the Bitcoin whitepaper to have a better understanding:
I think having a layman's understanding is a good usability test for such 
stuff.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[Marco Falke]

All of this is already implemented in the bitcoind and bitcoin gui.

The theoretic minimum for the prune target would be 0 (just the header
of the current best block) as Bitcoin Core already stores the
chainstate (about 2 GiB) regardless of what you set for -prune=<N>.

In practice, the minimum is 510, so reorgs and small rescans (may not
be implemented in 0.12) are still possible.

The clients won't let you set it below that target:
"Prune configured below the minimum of 550 MiB. Please use a higher number."

Also, keep in mind Bitcoin Core comes with a help message explaining
-prune and other command line options

--Marco

2016-01-25 13:27 GMT+01:00 xor--- via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org>:
> On Monday, January 25, 2016 01:03:17 PM Wladimir J. van der Laan wrote:
>> > If yes, I would highly recommend advertising it in the new release notes -
>> > as said, the disk space reduction is a big deal.
>>
>> Good idea, has been added by Marco Falke in commit fa31133,
>
> Thanks. The RC2 changelog now says:
>
>> To enable block pruning set prune=<N> on the command line or in
>> bitcoin.conf, where N is the number of MiB to allot for raw block & undo
>> data.
>
> From having read the Bitcoin whitepaper quite a few months ago ago, I have the
> very very basic understanding that pruning is meant to:
> - delete old transaction data which merely "moves coins around"
> - instead only store the "origin" (= block where coins were mined) and
> "current location" of the coins, i.e. the unspent transactions. Notably, I
> understood it as "this is as secure as storing everything, since we know where
> the coins were created, and where they are".
>
> So from that point of view, I would assume that there is a "natural" amount of
> megabytes which a fully pruned blockchain consists of: It would be defined by
> the final amount of unspent coins.
> I thereby am confused why it is possible to configure a number of megabytes
> "to allot for raw block & undo data". I would rather expect pruning just to be
> a boolean on/off flag, and the number of megabytes to be an automatically
> computed result from the natural size of the dataset.
> And especially, I fear that I could set N too low, and as a result, it would
> delete "too much". I mean could this result in even security relevant
> transaction data being deleted?
>
> Thus, it would be nice if you could yet once more edit the release notes to:
> - explain why a N must be given
> - what a "safe" value of N is. I.e. how large must N be at least to not delete
> security-relevant stuff?
> - maybe mention if there is a "auto" setting for N to ensure that it choses a
> safe value on its own?
>
> Sorry if my descriptions are from a layman's point of view. I intentionally
> did *not* re-read the Bitcoin whitepaper to have a better understanding:
> I think having a layman's understanding is a good usability test for such
> stuff.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[Wladimir J. van der Laan]

> > To enable block pruning set prune=<N> on the command line or in
> > bitcoin.conf, where N is the number of MiB to allot for raw block & undo
> > data.
> 

> From having read the Bitcoin whitepaper quite a few months ago ago, I have the 
> very very basic understanding that pruning is meant to:
> - delete old transaction data which merely "moves coins around"
> - instead only store the "origin" (= block where coins were mined) and 
> "current location" of the coins, i.e. the unspent transactions. Notably, I 
> understood it as "this is as secure as storing everything, since we know where 
> the coins were created, and where they are".
> 
> So from that point of view, I would assume that there is a "natural" amount of 
> megabytes which a fully pruned blockchain consists of: It would be defined by 
> the final amount of unspent coins.
> I thereby am confused why it is possible to configure a number of megabytes 
> "to allot for raw block & undo data". I would rather expect pruning just to be 
> a boolean on/off flag, and the number of megabytes to be an automatically 
> computed result from the natural size of the dataset.
> And especially, I fear that I could set N too low, and as a result, it would 
> delete "too much". I mean could this result in even security relevant 
> transaction data being deleted?

The term 'pruning', unfortunately is very much overused and overloaded in the
bitcoin ecosystem. Satoshi's paper refers to UTXO pruning. This is Pieter Wuille's "ultraprune",
which has been part of Bitcoin Core for more than three years.

Block pruning is not mentioned in that paper because it is just administrative,
the only reason that nodes store historical blocks at all is to serve it to other nodes,
as well as to catch up the wallet status and for -reindexes.

> Thus, it would be nice if you could yet once more edit the release notes to:

I don't have time to work on the release notes right now, but if someone else
wants to contribute that'd be awesome.

> - explain why a N must be given

To give a quotum. The point is that the user can choose how much harddisk space they want to
dedicate to block storage.

Block data that is stored can be used by other software, or potentially be
served to other nodes. The latter is not implemented at the moment - it would require
a change to the P2P protocol, thus right now pruning nodes don't serve block
data at all.

> - what a "safe" value of N is. I.e. how large must N be at least to not delete 
> security-relevant stuff?

There is no security compromise with pruning. Any value of N is intended to be safe.

Very low values would delete undo data that may be necessary in a reorganization,
but this is prohibited by argument checks.

Release notes are not meant as a replacement or supplement for documentation.
The documentation for -prune is this:

  -prune=<n>
       Reduce storage requirements by pruning (deleting) old blocks. This mode
       is incompatible with -txindex and -rescan. Warning: Reverting this
       setting requires re-downloading the entire blockchain. (default: 0 =
       disable pruning blocks, >550 = target size in MiB to use for block
       files)

> - maybe mention if there is a "auto" setting for N to ensure that it choses a 
> safe value on its own?

As said, there is no safe or unsafe value. The lowest acceptable value is just as safe
as storing everything.

Wladimir

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[Simon Selitsky]

>> Block data that is stored can be used by other software, or potentially be
>> served to other nodes. The latter is not implemented at the moment - it would require a change to the P2P protocol, thus right now pruning nodes don't serve block data at all.

Why is the minimum storage quota of 550 MiB necessary for pruning nodes
if the block data is not served to other nodes ? Could the client just do transaction verification and transaction relaying and only keep the block(s) 
being verified on disk ?


On Jan 25, 2016, at 10:05 AM, Wladimir J. van der Laan via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

>>> To enable block pruning set prune=<N> on the command line or in
>>> bitcoin.conf, where N is the number of MiB to allot for raw block & undo
>>> data.
> 
>> From having read the Bitcoin whitepaper quite a few months ago ago, I have the 
>> very very basic understanding that pruning is meant to:
>> - delete old transaction data which merely "moves coins around"
>> - instead only store the "origin" (= block where coins were mined) and 
>> "current location" of the coins, i.e. the unspent transactions. Notably, I 
>> understood it as "this is as secure as storing everything, since we know where 
>> the coins were created, and where they are".
>> 
>> So from that point of view, I would assume that there is a "natural" amount of 
>> megabytes which a fully pruned blockchain consists of: It would be defined by 
>> the final amount of unspent coins.
>> I thereby am confused why it is possible to configure a number of megabytes 
>> "to allot for raw block & undo data". I would rather expect pruning just to be 
>> a boolean on/off flag, and the number of megabytes to be an automatically 
>> computed result from the natural size of the dataset.
>> And especially, I fear that I could set N too low, and as a result, it would 
>> delete "too much". I mean could this result in even security relevant 
>> transaction data being deleted?
> 
> The term 'pruning', unfortunately is very much overused and overloaded in the
> bitcoin ecosystem. Satoshi's paper refers to UTXO pruning. This is Pieter Wuille's "ultraprune",
> which has been part of Bitcoin Core for more than three years.
> 
> Block pruning is not mentioned in that paper because it is just administrative,
> the only reason that nodes store historical blocks at all is to serve it to other nodes,
> as well as to catch up the wallet status and for -reindexes.
> 
>> Thus, it would be nice if you could yet once more edit the release notes to:
> 
> I don't have time to work on the release notes right now, but if someone else
> wants to contribute that'd be awesome.
> 
>> - explain why a N must be given
> 
> To give a quotum. The point is that the user can choose how much harddisk space they want to
> dedicate to block storage.
> 
> Block data that is stored can be used by other software, or potentially be
> served to other nodes. The latter is not implemented at the moment - it would require
> a change to the P2P protocol, thus right now pruning nodes don't serve block
> data at all.
> 
>> - what a "safe" value of N is. I.e. how large must N be at least to not delete 
>> security-relevant stuff?
> 
> There is no security compromise with pruning. Any value of N is intended to be safe.
> 
> Very low values would delete undo data that may be necessary in a reorganization,
> but this is prohibited by argument checks.
> 
> Release notes are not meant as a replacement or supplement for documentation.
> The documentation for -prune is this:
> 
>  -prune=<n>
>       Reduce storage requirements by pruning (deleting) old blocks. This mode
>       is incompatible with -txindex and -rescan. Warning: Reverting this
>       setting requires re-downloading the entire blockchain. (default: 0 =
>       disable pruning blocks, >550 = target size in MiB to use for block
>       files)
> 
>> - maybe mention if there is a "auto" setting for N to ensure that it choses a 
>> safe value on its own?
> 
> As said, there is no safe or unsafe value. The lowest acceptable value is just as safe
> as storing everything.
> 
> Wladimir
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[Jonas Schnelli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


> Why is the minimum storage quota of 550 MiB necessary for pruning
> nodes if the block data is not served to other nodes ? Could the
> client just do transaction verification and transaction relaying
> and only keep the block(s) being verified on disk ?
> 

We try to allow reorgs ~288 blocks deep also in pruning:

- From code comments:
Require that user allocate at least 550MB for block & undo files
(blk???.dat and rev???.dat)
At 1MB per block, 288 blocks = 288MB.
Add 15% for Undo data = 331MB
Add 20% for Orphan block rate = 397MB
We want the low water mark after pruning to be at least 397 MB and
since we prune in
full block file chunks, we need the high water mark which triggers the
prune to be
one 128MB block file + added 15% undo data = 147MB greater for a total
of 545MB
Setting the target to > than 550MB will make it likely we can respect
the target.


</jonas>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWpklQAAoJECnUvLZBb1Ps464P+gNqN+Rs5MnAFg5Hukxcj9BU
f+Zm5B99VMT3qWjJUjk05NTPLoa6vS6I6pkanWNlzmquZSardW0rW/7JS8wu8BiA
ILP3gMWMiF2w//0o+4uEhQt5FhUKQGUVgtXDprc/6WeOfpEBunk+/YTmFPpSMQym
w9roH+vrMBHNogSXjdIsn3qPGdVWWuc1PeeluMthN/f7Y5Y5kcyUJvvJmhNbNspG
UaGqh7vCDBvaHmxKuPRvqPlSqvwXjA3kxDP1s+VBtLGKnJzVoBqBEsody0UscQQO
RRvxbEdaRL1iTVgA0orsDCOMsBaUcKiZ4tlJUd+Z+ifHCVJ5Szl5fsqIIElF8vOk
hy8++T4XqPEZqlDnAIpOxE0eGnByvdkUrFew60nA+A+ivY7GkCFhMz8AP4VHrhFS
UOU2wDuBOsA6ssqkxMmc5Vizyb6CmL2Ho0csPqabvfRYk5VZACc63FbJ2xcdjDZz
CufvfJZ3O5dgSy29fn9XQHsU8qSn0DteSU/9OiHAJmvkqrvB0yT21kIQXUWiqYGk
xDvc/SVpttYqaW2hAgjFG9NGJ/D2dpliYNUjgwmijUVZuI9bkJ68l5CfpOzXYnNJ
e1AFzBeHVCKSn0advmTVdyybslU32g7ytzJQcQP2b8a4GQYEI6DNhTA5HTxWaj//
O+Cm0CkAbW1vNJqSolnU
=lDcA
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] Bitcoin Core 0.12.0 release candidate 1 available

[xor]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

On Monday, January 25, 2016 04:05:59 PM Wladimir J. van der Laan wrote:
> I don't have time to work on the release notes right now, but if someone
> else wants to contribute that'd be awesome.

I cooked my first pull request to resolve this:
https://github.com/bitcoin/bitcoin/pull/7416

Thanks for all the explanations you folks provided! :)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]