[bitcoin-dev] [BIP Proposal] Buried Deployments

[bitcoin-dev] [BIP Proposal] Buried Deployments

[Suhas Daftuar]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

Hi,

Recently Bitcoin Core merged a simplification to the consensus rules
surrounding deployment of BIPs 34, 66, and 65 (
https://github.com/bitcoin/bitcoin/pull/8391), and though the change is a
minor one, I thought it was worth documenting the rationale in a BIP for
posterity.

Here's the abstract:

Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner
signaling in block version numbers. Now that the chain has long since
passed the blocks at which those consensus rules have triggered, we can (as
a simplification and optimization) replace the trigger mechanism by caching
the block heights at which those consensus rules became enforced.

The full draft can be found here:

https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki

[-- Attachment #2: Type: text/html, Size: 1190 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

NACK

Horrible precedent (hardcoding rule changes based on the assumption that large forks indicate a catastrophic failure), extremely poor process (already shipped, now the discussion), and not even a material performance optimization (the checks are avoidable once activated until a sufficiently deep reorg deactivates them).

e

> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> Hi,
> 
> Recently Bitcoin Core merged a simplification to the consensus rules surrounding deployment of BIPs 34, 66, and 65 (https://github.com/bitcoin/bitcoin/pull/8391), and though the change is a minor one, I thought it was worth documenting the rationale in a BIP for posterity.
> 
> Here's the abstract:
> 
> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner signaling in block version numbers. Now that the chain has long since passed the blocks at which those consensus rules have triggered, we can (as a simplification and optimization) replace the trigger mechanism by caching the block heights at which those consensus rules became enforced.
> 
> The full draft can be found here: 
> 
> https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 2371 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Suhas Daftuar]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

Just want to clarify two points:

This change has not yet appeared in any released software (but I assume it
will be in the next release, 0.14.0).

I agree that the performance optimization is not the point of this change;
I can modify the BIP draft to de-emphasize that further (perhaps remove
mention of it entirely).

On Mon, Nov 14, 2016 at 1:47 PM, Eric Voskuil <eric@voskuil.org> wrote:

> NACK
>
> Horrible precedent (hardcoding rule changes based on the assumption that
> large forks indicate a catastrophic failure), extremely poor process
> (already shipped, now the discussion), and not even a material performance
> optimization (the checks are avoidable once activated until a sufficiently
> deep reorg deactivates them).
>
> e
>
> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> Hi,
>
> Recently Bitcoin Core merged a simplification to the consensus rules
> surrounding deployment of BIPs 34, 66, and 65 (https://github.com/bitcoin/
> bitcoin/pull/8391), and though the change is a minor one, I thought it
> was worth documenting the rationale in a BIP for posterity.
>
> Here's the abstract:
>
> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner
> signaling in block version numbers. Now that the chain has long since
> passed the blocks at which those consensus rules have triggered, we can (as
> a simplification and optimization) replace the trigger mechanism by caching
> the block heights at which those consensus rules became enforced.
>
> The full draft can be found here:
>
> https://github.com/sdaftuar/bips/blob/buried-deployments/
> bip-buried-deployments.mediawiki
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 3182 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Btc Drak]

I think this is already covered in the BIP text:-

"As of November 2016, the most recent of these changes (BIP 65,
enforced since December 2015) has nearly 50,000 blocks built on top of
it. The occurrence of such a reorg that would cause the activating
block to be disconnected would raise fundamental concerns about the
security assumptions of Bitcoin, a far bigger issue than any
non-backwards compatible change.

So while this proposal could theoretically result in a consensus
split, it is extremely unlikely, and in particular any such
circumstances would be sufficiently damaging to the Bitcoin network to
dwarf any concerns about the effects of this proposed change."


On Mon, Nov 14, 2016 at 6:47 PM, Eric Voskuil via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> NACK
>
> Horrible precedent (hardcoding rule changes based on the assumption that
> large forks indicate a catastrophic failure), extremely poor process
> (already shipped, now the discussion), and not even a material performance
> optimization (the checks are avoidable once activated until a sufficiently
> deep reorg deactivates them).
>
> e
>
> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> Hi,
>
> Recently Bitcoin Core merged a simplification to the consensus rules
> surrounding deployment of BIPs 34, 66, and 65
> (https://github.com/bitcoin/bitcoin/pull/8391), and though the change is a
> minor one, I thought it was worth documenting the rationale in a BIP for
> posterity.
>
> Here's the abstract:
>
> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner
> signaling in block version numbers. Now that the chain has long since passed
> the blocks at which those consensus rules have triggered, we can (as a
> simplification and optimization) replace the trigger mechanism by caching
> the block heights at which those consensus rules became enforced.
>
> The full draft can be found here:
>
> https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

Actually this does nothing to provide justification for this consensus rule change. It is just an attempt to deflect criticism from the fact that it is such a change.

e

> On Nov 15, 2016, at 9:45 AM, Btc Drak <btcdrak@gmail.com> wrote:
> 
> I think this is already covered in the BIP text:-
> 
> "As of November 2016, the most recent of these changes (BIP 65,
> enforced since December 2015) has nearly 50,000 blocks built on top of
> it. The occurrence of such a reorg that would cause the activating
> block to be disconnected would raise fundamental concerns about the
> security assumptions of Bitcoin, a far bigger issue than any
> non-backwards compatible change.
> 
> So while this proposal could theoretically result in a consensus
> split, it is extremely unlikely, and in particular any such
> circumstances would be sufficiently damaging to the Bitcoin network to
> dwarf any concerns about the effects of this proposed change."
> 
> 
> On Mon, Nov 14, 2016 at 6:47 PM, Eric Voskuil via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> NACK
>> 
>> Horrible precedent (hardcoding rule changes based on the assumption that
>> large forks indicate a catastrophic failure), extremely poor process
>> (already shipped, now the discussion), and not even a material performance
>> optimization (the checks are avoidable once activated until a sufficiently
>> deep reorg deactivates them).
>> 
>> e
>> 
>> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> 
>> Hi,
>> 
>> Recently Bitcoin Core merged a simplification to the consensus rules
>> surrounding deployment of BIPs 34, 66, and 65
>> (https://github.com/bitcoin/bitcoin/pull/8391), and though the change is a
>> minor one, I thought it was worth documenting the rationale in a BIP for
>> posterity.
>> 
>> Here's the abstract:
>> 
>> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner
>> signaling in block version numbers. Now that the chain has long since passed
>> the blocks at which those consensus rules have triggered, we can (as a
>> simplification and optimization) replace the trigger mechanism by caching
>> the block heights at which those consensus rules became enforced.
>> 
>> The full draft can be found here:
>> 
>> https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki
>> 
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> 
>> 
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> 

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Jameson Lopp]

[-- Attachment #1: Type: text/plain, Size: 3686 bytes --]

Since "buried deployments" are specifically in reference to historical
consensus changes, I think the question is more one of human consensus than
machine consensus. Is there any disagreement amongst Bitcoin users that
BIP34 activated at block 227931, BIP65 activated at block 388381, and BIP66
activated at block 363725? Somehow I doubt it.

It seems to me that this change is merely cementing into place a few
attributes of the blockchain's history that are not in dispute.

- Jameson

On Tue, Nov 15, 2016 at 5:42 PM, Eric Voskuil via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Actually this does nothing to provide justification for this consensus
> rule change. It is just an attempt to deflect criticism from the fact that
> it is such a change.
>
> e
>
> > On Nov 15, 2016, at 9:45 AM, Btc Drak <btcdrak@gmail.com> wrote:
> >
> > I think this is already covered in the BIP text:-
> >
> > "As of November 2016, the most recent of these changes (BIP 65,
> > enforced since December 2015) has nearly 50,000 blocks built on top of
> > it. The occurrence of such a reorg that would cause the activating
> > block to be disconnected would raise fundamental concerns about the
> > security assumptions of Bitcoin, a far bigger issue than any
> > non-backwards compatible change.
> >
> > So while this proposal could theoretically result in a consensus
> > split, it is extremely unlikely, and in particular any such
> > circumstances would be sufficiently damaging to the Bitcoin network to
> > dwarf any concerns about the effects of this proposed change."
> >
> >
> > On Mon, Nov 14, 2016 at 6:47 PM, Eric Voskuil via bitcoin-dev
> > <bitcoin-dev@lists.linuxfoundation.org> wrote:
> >> NACK
> >>
> >> Horrible precedent (hardcoding rule changes based on the assumption that
> >> large forks indicate a catastrophic failure), extremely poor process
> >> (already shipped, now the discussion), and not even a material
> performance
> >> optimization (the checks are avoidable once activated until a
> sufficiently
> >> deep reorg deactivates them).
> >>
> >> e
> >>
> >> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev
> >> <bitcoin-dev@lists.linuxfoundation.org> wrote:
> >>
> >> Hi,
> >>
> >> Recently Bitcoin Core merged a simplification to the consensus rules
> >> surrounding deployment of BIPs 34, 66, and 65
> >> (https://github.com/bitcoin/bitcoin/pull/8391), and though the change
> is a
> >> minor one, I thought it was worth documenting the rationale in a BIP for
> >> posterity.
> >>
> >> Here's the abstract:
> >>
> >> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner
> >> signaling in block version numbers. Now that the chain has long since
> passed
> >> the blocks at which those consensus rules have triggered, we can (as a
> >> simplification and optimization) replace the trigger mechanism by
> caching
> >> the block heights at which those consensus rules became enforced.
> >>
> >> The full draft can be found here:
> >>
> >> https://github.com/sdaftuar/bips/blob/buried-deployments/
> bip-buried-deployments.mediawiki
> >>
> >> _______________________________________________
> >> bitcoin-dev mailing list
> >> bitcoin-dev@lists.linuxfoundation.org
> >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> >>
> >>
> >> _______________________________________________
> >> bitcoin-dev mailing list
> >> bitcoin-dev@lists.linuxfoundation.org
> >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> >>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 6020 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 4661 bytes --]

This sort of statement represents one consequence of the aforementioned bad precedent.

Are checkpoints good now? Are hard forks okay now?

What is the maximum depth of a reorg allowed by this non-machine consensus?

Shouldn't we just define a max depth so that all cruft deeper than that can just be discarded on a regular basis?

Why are there activation heights defined by this hard fork if it's not possible to reorg back to them?

The "BIP" is neither a Proposal (it's been decided, just documenting for posterity), nor an Improvement (there is no actual benefit, just some tidying up in the notoriously obtuse satoshi code base), nor Bitcoin (a hard fork defines an alt coin, so from Aug 4 forward it has been CoreCoin).

e

> On Nov 16, 2016, at 5:29 AM, Jameson Lopp <jameson.lopp@gmail.com> wrote:
> 
> Since "buried deployments" are specifically in reference to historical consensus changes, I think the question is more one of human consensus than machine consensus. Is there any disagreement amongst Bitcoin users that BIP34 activated at block 227931, BIP65 activated at block 388381, and BIP66 activated at block 363725? Somehow I doubt it.
> 
> It seems to me that this change is merely cementing into place a few attributes of the blockchain's history that are not in dispute.
> 
> - Jameson
> 
>> On Tue, Nov 15, 2016 at 5:42 PM, Eric Voskuil via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> Actually this does nothing to provide justification for this consensus rule change. It is just an attempt to deflect criticism from the fact that it is such a change.
>> 
>> e
>> 
>> > On Nov 15, 2016, at 9:45 AM, Btc Drak <btcdrak@gmail.com> wrote:
>> >
>> > I think this is already covered in the BIP text:-
>> >
>> > "As of November 2016, the most recent of these changes (BIP 65,
>> > enforced since December 2015) has nearly 50,000 blocks built on top of
>> > it. The occurrence of such a reorg that would cause the activating
>> > block to be disconnected would raise fundamental concerns about the
>> > security assumptions of Bitcoin, a far bigger issue than any
>> > non-backwards compatible change.
>> >
>> > So while this proposal could theoretically result in a consensus
>> > split, it is extremely unlikely, and in particular any such
>> > circumstances would be sufficiently damaging to the Bitcoin network to
>> > dwarf any concerns about the effects of this proposed change."
>> >
>> >
>> > On Mon, Nov 14, 2016 at 6:47 PM, Eric Voskuil via bitcoin-dev
>> > <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> >> NACK
>> >>
>> >> Horrible precedent (hardcoding rule changes based on the assumption that
>> >> large forks indicate a catastrophic failure), extremely poor process
>> >> (already shipped, now the discussion), and not even a material performance
>> >> optimization (the checks are avoidable once activated until a sufficiently
>> >> deep reorg deactivates them).
>> >>
>> >> e
>> >>
>> >> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev
>> >> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Recently Bitcoin Core merged a simplification to the consensus rules
>> >> surrounding deployment of BIPs 34, 66, and 65
>> >> (https://github.com/bitcoin/bitcoin/pull/8391), and though the change is a
>> >> minor one, I thought it was worth documenting the rationale in a BIP for
>> >> posterity.
>> >>
>> >> Here's the abstract:
>> >>
>> >> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated via miner
>> >> signaling in block version numbers. Now that the chain has long since passed
>> >> the blocks at which those consensus rules have triggered, we can (as a
>> >> simplification and optimization) replace the trigger mechanism by caching
>> >> the block heights at which those consensus rules became enforced.
>> >>
>> >> The full draft can be found here:
>> >>
>> >> https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki
>> >>
>> >> _______________________________________________
>> >> bitcoin-dev mailing list
>> >> bitcoin-dev@lists.linuxfoundation.org
>> >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> >>
>> >>
>> >> _______________________________________________
>> >> bitcoin-dev mailing list
>> >> bitcoin-dev@lists.linuxfoundation.org
>> >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> >>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

[-- Attachment #2: Type: text/html, Size: 7198 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Tier Nolan]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Wed, Nov 16, 2016 at 1:58 PM, Eric Voskuil via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Are checkpoints good now? Are hard forks okay now?
>

I think that at least one checkpoint should be included.  The assumption is
that no 50k re-orgs will happen, and that assumption should be directly
checked.

Checkpointing only needs to happen during the headers-first part of the
download.

If the block at the BIP-65 height is checkpointed, then the comparisons for
the other ones are automatically correct.  They are unnecessary, since the
checkpoint protects all earlier block, but many people would like to be
able to verify the legacy chain.

This makes the change a soft-fork rather than a hard fork.  Chains that
don't go through the checkpoint are rejected but no new chains are allowed.

[-- Attachment #2: Type: text/html, Size: 1243 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Alex Morcos]

[-- Attachment #1: Type: text/plain, Size: 2563 bytes --]

I think we are misunderstanding the effect of this change.
It's still "OK" for a 50k re-org to happen.
We're just saying that if it does, we will now have potentially introduced
a hard fork between new client and old clients if the reorg contains
earlier signaling for the most recent ISM soft fork and then blocks which
do not conform to that soft fork before the block height encoded activation.

I think the argument is this doesn't substantially add to the confusion or
usability of the system as its likely that old software won't even handle
50k block reorgs cleanly anyway and there will clearly have to be human
coordination at the time of the event.  In the unlikely event that the new
chain does cause such a hard fork, that coordination can result in everyone
upgrading to software that supports the new rules anyway.

So no, I don't think we should add a checkpoint.  I think we should all
just agree to a hard fork that only has a very very slim chance of any
practical effect.

In response to Thomas' email.  I think ideally we would treat these soft
forks the way we did BIP30 which is to say that we're just introducing a
further soft fork that applies to all blocks except for the historical
exceptions.  So then its a almost no-op soft fork with no risk of hard
fork.   This however isn't practical with at least BIP 34 without storing
the hashes of all 200K blocks that don't meet the requirement.



On Wed, Nov 16, 2016 at 9:18 AM, Tier Nolan via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Wed, Nov 16, 2016 at 1:58 PM, Eric Voskuil via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Are checkpoints good now? Are hard forks okay now?
>>
>
> I think that at least one checkpoint should be included.  The assumption
> is that no 50k re-orgs will happen, and that assumption should be directly
> checked.
>
> Checkpointing only needs to happen during the headers-first part of the
> download.
>
> If the block at the BIP-65 height is checkpointed, then the comparisons
> for the other ones are automatically correct.  They are unnecessary, since
> the checkpoint protects all earlier block, but many people would like to be
> able to verify the legacy chain.
>
> This makes the change a soft-fork rather than a hard fork.  Chains that
> don't go through the checkpoint are rejected but no new chains are allowed.
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 3672 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

On Wed, Nov 16, 2016 at 09:32:24AM -0500, Alex Morcos via bitcoin-dev wrote:
> I think we are misunderstanding the effect of this change.
> It's still "OK" for a 50k re-org to happen.
> We're just saying that if it does, we will now have potentially introduced
> a hard fork between new client and old clients if the reorg contains
> earlier signaling for the most recent ISM soft fork and then blocks which
> do not conform to that soft fork before the block height encoded activation.
> 
> I think the argument is this doesn't substantially add to the confusion or
> usability of the system as its likely that old software won't even handle
> 50k block reorgs cleanly anyway and there will clearly have to be human
> coordination at the time of the event.  In the unlikely event that the new
> chain does cause such a hard fork, that coordination can result in everyone
> upgrading to software that supports the new rules anyway.
> 
> So no, I don't think we should add a checkpoint.  I think we should all
> just agree to a hard fork that only has a very very slim chance of any
> practical effect.

So, conceptually, another way to deal with this is to hardcode a blockhash
where we allow blocks in a chain ending with that blockhash to _not_ follow
BIP65, up until that blockhash, and any blockchain without that blockhash must
respect BIP65 for all blocks in the chain.

This is a softfork: we've only added rules that made otherwise valid chains
invalid, and at the same time we are still accepting large reorgs (albeit under
stricter rules than before).

I'd suggest we call this a exemption hash - we've exempted a particular
blockchains from a soft-forked rule that we would otherwise enforce.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

I would suggest that, before discussing how best to fork the chain to meet this objective, we consider the objective.

The implementers have acknowledged that this does not represent a performance improvement. Especially given that this was apparently not initially understood, that alone is good reason for them to reconsider.

The remaining stated objective is reduction of code complexity. Let us be very clear, a proposal to change the protocol must be considered independently of any particular implementation of the protocol. While the implementation of BIP34 style activation may be hugely complex in the satoshi code, it is definitely not complex as a matter of necessity.

Activation constitutes maybe a dozen lines of additional code in libbitcoin. The need to hit the chain (or cache) to obtain historical header info will remain for proof of work, so this change doesn't even accomplish some sort of beneficial isolation from blockchain history.

So, at best, we are talking about various ways to introduce a consensus fork so that a well designed implementation  can remove a tiny amount of already-written code and associated tests. In my opinion this is embarrassingly poor reasoning. It would be much more productive to reduce satoshi code complexity in ways that do not impact the protocol. There are a *huge* number of such opportunities, and in fact activation is one of them. Once that is done, we can talk about forking to reduce source code complexity.

These fork suggestions actually increase *necessary* complexity for any implantation that takes a rational approach to forks. By rational I mean *additive*. Deleting rules from Bitcoin code is simply bad design. Rules are never removed, they are added. A new rule to modify an old rule is simply a new rule. This is new and additional code. So please don't assume in this "proposal" that this makes development simpler for other implementations, that is not a necessary conclusion.

e

> On Nov 16, 2016, at 1:01 PM, Peter Todd via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
>> On Wed, Nov 16, 2016 at 09:32:24AM -0500, Alex Morcos via bitcoin-dev wrote:
>> I think we are misunderstanding the effect of this change.
>> It's still "OK" for a 50k re-org to happen.
>> We're just saying that if it does, we will now have potentially introduced
>> a hard fork between new client and old clients if the reorg contains
>> earlier signaling for the most recent ISM soft fork and then blocks which
>> do not conform to that soft fork before the block height encoded activation.
>> 
>> I think the argument is this doesn't substantially add to the confusion or
>> usability of the system as its likely that old software won't even handle
>> 50k block reorgs cleanly anyway and there will clearly have to be human
>> coordination at the time of the event.  In the unlikely event that the new
>> chain does cause such a hard fork, that coordination can result in everyone
>> upgrading to software that supports the new rules anyway.
>> 
>> So no, I don't think we should add a checkpoint.  I think we should all
>> just agree to a hard fork that only has a very very slim chance of any
>> practical effect.
> 
> So, conceptually, another way to deal with this is to hardcode a blockhash
> where we allow blocks in a chain ending with that blockhash to _not_ follow
> BIP65, up until that blockhash, and any blockchain without that blockhash must
> respect BIP65 for all blocks in the chain.
> 
> This is a softfork: we've only added rules that made otherwise valid chains
> invalid, and at the same time we are still accepting large reorgs (albeit under
> stricter rules than before).
> 
> I'd suggest we call this a exemption hash - we've exempted a particular
> blockchains from a soft-forked rule that we would otherwise enforce.
> 
> -- 
> https://petertodd.org 'peter'[:-1]@petertodd.org
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Luke Dashjr]

On Wednesday, November 16, 2016 9:01:00 PM Peter Todd via bitcoin-dev wrote:
> So, conceptually, another way to deal with this is to hardcode a blockhash
> where we allow blocks in a chain ending with that blockhash to _not_ follow
> BIP65, up until that blockhash, and any blockchain without that blockhash
> must respect BIP65 for all blocks in the chain.
> 
> This is a softfork: we've only added rules that made otherwise valid chains
> invalid, and at the same time we are still accepting large reorgs (albeit
> under stricter rules than before).
> 
> I'd suggest we call this a exemption hash - we've exempted a particular
> blockchains from a soft-forked rule that we would otherwise enforce.

While this is technically a softfork, I think it may behave somewhat like a 
hardfork if we're not careful. Particularly, is the chain up to the block 
immediately before the checkpoint itself valid on its own, or does it simply 
become retroactively valid when it hits that checkpoint?

P.S. Your PGP signature is invalid?

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Thomas Kerin]

[-- Attachment #1: Type: text/plain, Size: 6616 bytes --]

BIP30 actually was given similar treatment after a reasonable amount of
time had passed.
https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2392

You are also missing BIP50: 'March 2013 Chain For Post-Mortem', which
neither benefited nor improved bitcoin, but did document an event for
posterity.

This is not a hard fork. Removing ISM just means we've committed to
those soft-forks only locking into the chain we use now.

On 11/16/2016 01:58 PM, Eric Voskuil via bitcoin-dev wrote:
> This sort of statement represents one consequence of the
> aforementioned bad precedent.
>
> Are checkpoints good now? Are hard forks okay now?
>
> What is the maximum depth of a reorg allowed by this non-machine
> consensus?
>
> Shouldn't we just define a max depth so that all cruft deeper than
> that can just be discarded on a regular basis?
>
> Why are there activation heights defined by this hard fork if it's not
> possible to reorg back to them?
>
> The "BIP" is neither a Proposal (it's been decided, just documenting
> for posterity), nor an Improvement (there is no actual benefit, just
> some tidying up in the notoriously obtuse satoshi code base), nor
> Bitcoin (a hard fork defines an alt coin, so from Aug 4 forward it has
> been CoreCoin).
>
> e
>
> On Nov 16, 2016, at 5:29 AM, Jameson Lopp <jameson.lopp@gmail.com
> <mailto:jameson.lopp@gmail.com>> wrote:
>
>> Since "buried deployments" are specifically in reference to
>> historical consensus changes, I think the question is more one of
>> human consensus than machine consensus. Is there any disagreement
>> amongst Bitcoin users that BIP34 activated at block 227931, BIP65
>> activated at block 388381, and BIP66 activated at block 363725?
>> Somehow I doubt it.
>>
>> It seems to me that this change is merely cementing into place a few
>> attributes of the blockchain's history that are not in dispute.
>>
>> - Jameson
>>
>> On Tue, Nov 15, 2016 at 5:42 PM, Eric Voskuil via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>
>>     Actually this does nothing to provide justification for this
>>     consensus rule change. It is just an attempt to deflect criticism
>>     from the fact that it is such a change.
>>
>>     e
>>
>>     > On Nov 15, 2016, at 9:45 AM, Btc Drak <btcdrak@gmail.com
>>     <mailto:btcdrak@gmail.com>> wrote:
>>     >
>>     > I think this is already covered in the BIP text:-
>>     >
>>     > "As of November 2016, the most recent of these changes (BIP 65,
>>     > enforced since December 2015) has nearly 50,000 blocks built on
>>     top of
>>     > it. The occurrence of such a reorg that would cause the activating
>>     > block to be disconnected would raise fundamental concerns about the
>>     > security assumptions of Bitcoin, a far bigger issue than any
>>     > non-backwards compatible change.
>>     >
>>     > So while this proposal could theoretically result in a consensus
>>     > split, it is extremely unlikely, and in particular any such
>>     > circumstances would be sufficiently damaging to the Bitcoin
>>     network to
>>     > dwarf any concerns about the effects of this proposed change."
>>     >
>>     >
>>     > On Mon, Nov 14, 2016 at 6:47 PM, Eric Voskuil via bitcoin-dev
>>     > <bitcoin-dev@lists.linuxfoundation.org
>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>     >> NACK
>>     >>
>>     >> Horrible precedent (hardcoding rule changes based on the
>>     assumption that
>>     >> large forks indicate a catastrophic failure), extremely poor
>>     process
>>     >> (already shipped, now the discussion), and not even a material
>>     performance
>>     >> optimization (the checks are avoidable once activated until a
>>     sufficiently
>>     >> deep reorg deactivates them).
>>     >>
>>     >> e
>>     >>
>>     >> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev
>>     >> <bitcoin-dev@lists.linuxfoundation.org
>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>     >>
>>     >> Hi,
>>     >>
>>     >> Recently Bitcoin Core merged a simplification to the consensus
>>     rules
>>     >> surrounding deployment of BIPs 34, 66, and 65
>>     >> (https://github.com/bitcoin/bitcoin/pull/8391
>>     <https://github.com/bitcoin/bitcoin/pull/8391>), and though the
>>     change is a
>>     >> minor one, I thought it was worth documenting the rationale in
>>     a BIP for
>>     >> posterity.
>>     >>
>>     >> Here's the abstract:
>>     >>
>>     >> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated
>>     via miner
>>     >> signaling in block version numbers. Now that the chain has
>>     long since passed
>>     >> the blocks at which those consensus rules have triggered, we
>>     can (as a
>>     >> simplification and optimization) replace the trigger mechanism
>>     by caching
>>     >> the block heights at which those consensus rules became enforced.
>>     >>
>>     >> The full draft can be found here:
>>     >>
>>     >>
>>     https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki
>>     <https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki>
>>     >>
>>     >> _______________________________________________
>>     >> bitcoin-dev mailing list
>>     >> bitcoin-dev@lists.linuxfoundation.org
>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>     >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>     <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
>>     >>
>>     >>
>>     >> _______________________________________________
>>     >> bitcoin-dev mailing list
>>     >> bitcoin-dev@lists.linuxfoundation.org
>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>     >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>     <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
>>     >>
>>     _______________________________________________
>>     bitcoin-dev mailing list
>>     bitcoin-dev@lists.linuxfoundation.org
>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>     https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>     <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
>>
>>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[-- Attachment #2: Type: text/html, Size: 12178 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Jorge Timón]

On Wed, Nov 16, 2016 at 3:18 PM, Thomas Kerin via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> BIP30 actually was given similar treatment after a reasonable amount of time
> had passed.
> https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2392

This is not really the same. BIP30 is not validated after BIP34 is
active because blocks complying with BIP34 will always necessarily
comply with BIP30 (ie coinbases cannot be duplicated after they
include the block height).

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

On 11/16/2016 03:58 PM, Jorge Timón via bitcoin-dev wrote:
> On Wed, Nov 16, 2016 at 3:18 PM, Thomas Kerin via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> BIP30 actually was given similar treatment after a reasonable amount of time
>> had passed.
>> https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2392
> 
> This is not really the same. BIP30 is not validated after BIP34 is
> active because blocks complying with BIP34 will always necessarily
> comply with BIP30 (ie coinbases cannot be duplicated after they
> include the block height).

This is a misinterpretation of BIP30. Duplicate transaction hashes can
and will happen and are perfectly valid in Bitcoin. BIP34 does not
prevent this.

e


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Alex Morcos]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

huh?
can you give an example of how a duplicate transaction hash (in the same
chain) can happen given BIP34?


On Wed, Nov 16, 2016 at 7:00 PM, Eric Voskuil via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On 11/16/2016 03:58 PM, Jorge Timón via bitcoin-dev wrote:
> > On Wed, Nov 16, 2016 at 3:18 PM, Thomas Kerin via bitcoin-dev
> > <bitcoin-dev@lists.linuxfoundation.org> wrote:
> >> BIP30 actually was given similar treatment after a reasonable amount of
> time
> >> had passed.
> >> https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2392
> >
> > This is not really the same. BIP30 is not validated after BIP34 is
> > active because blocks complying with BIP34 will always necessarily
> > comply with BIP30 (ie coinbases cannot be duplicated after they
> > include the block height).
>
> This is a misinterpretation of BIP30. Duplicate transaction hashes can
> and will happen and are perfectly valid in Bitcoin. BIP34 does not
> prevent this.
>
> e
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 2026 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 1539 bytes --]

On 11/16/2016 05:24 PM, Alex Morcos wrote:
> huh?
> can you give an example of how a duplicate transaction hash (in the same
> chain) can happen given BIP34?

"The pigeonhole principle arises in computer science. For example,
collisions are inevitable in a hash table because the number of possible
keys exceeds the number of indices in the array. A hashing algorithm, no
matter how clever, cannot avoid these collisions."

https://en.wikipedia.org/wiki/Pigeonhole_principle

e

> On Wed, Nov 16, 2016 at 7:00 PM, Eric Voskuil via bitcoin-dev wrote:
> 
>     On 11/16/2016 03:58 PM, Jorge Timón via bitcoin-dev wrote:
>     > On Wed, Nov 16, 2016 at 3:18 PM, Thomas Kerin via bitcoin-dev
>     > <bitcoin-dev@lists.linuxfoundation.org
>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>     >> BIP30 actually was given similar treatment after a reasonable amount of time
>     >> had passed.
>     >> https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2392
>     <https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2392>
>     >
>     > This is not really the same. BIP30 is not validated after BIP34 is
>     > active because blocks complying with BIP34 will always necessarily
>     > comply with BIP30 (ie coinbases cannot be duplicated after they
>     > include the block height).
> 
>     This is a misinterpretation of BIP30. Duplicate transaction hashes can
>     and will happen and are perfectly valid in Bitcoin. BIP34 does not
>     prevent this.
> 
>     e


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 8041 bytes --]

On 11/16/2016 06:18 AM, Thomas Kerin wrote:
> BIP30 actually was given similar treatment after a reasonable amount of
> time had passed.
> https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2392

BIP30 was the resolution to a catostrophic protocol flaw that would
impact any block whether above or below the point where the rule was
applied. Applying it to all future blocks, regardless of whether there
is a reorg back to genesis, was the only option as far as I can tell. So
the comparison to an unnecessary fork is hardly apt.

> You are also missing BIP50: 'March 2013 Chain For Post-Mortem', which
> neither benefited nor improved bitcoin, but did document an event for
> posterity.

BIP50 documents the release of an "unexpected" hard fork to a large
number of users. Given that Core code is considered by some to be the
*definition* of the true protocol, this led to two "legitimate" Bitcoin
chains. Leveraging the centralized state of Bitcoin mining, the
development team was able to kill the newer chain. This was simply an
altcoin that didn't survive because people stopped using it.

Anyone can create an altcoin - the question here is specifically, why
would we want to do so in this case.

> This is not a hard fork. Removing ISM just means we've committed to
> those soft-forks only locking into the chain we use now.

There didn't seem to be any confusion among the implementers that it is
a hard fork.

I will correct one implication I made below. The heights in the proposal
are required in the absence of BIP34-style activation so that the soft
fork validation rules can be properly enforced at those points (whether
or not a deep reorg happens).

e

> On 11/16/2016 01:58 PM, Eric Voskuil via bitcoin-dev wrote:
>> This sort of statement represents one consequence of the
>> aforementioned bad precedent.
>>
>> Are checkpoints good now? Are hard forks okay now?
>>
>> What is the maximum depth of a reorg allowed by this non-machine
>> consensus?
>>
>> Shouldn't we just define a max depth so that all cruft deeper than
>> that can just be discarded on a regular basis?
>>
>> Why are there activation heights defined by this hard fork if it's not
>> possible to reorg back to them?
>>
>> The "BIP" is neither a Proposal (it's been decided, just documenting
>> for posterity), nor an Improvement (there is no actual benefit, just
>> some tidying up in the notoriously obtuse satoshi code base), nor
>> Bitcoin (a hard fork defines an alt coin, so from Aug 4 forward it has
>> been CoreCoin).
>>
>> e
>>
>> On Nov 16, 2016, at 5:29 AM, Jameson Lopp <jameson.lopp@gmail.com
>> <mailto:jameson.lopp@gmail.com>> wrote:
>>
>>> Since "buried deployments" are specifically in reference to
>>> historical consensus changes, I think the question is more one of
>>> human consensus than machine consensus. Is there any disagreement
>>> amongst Bitcoin users that BIP34 activated at block 227931, BIP65
>>> activated at block 388381, and BIP66 activated at block 363725?
>>> Somehow I doubt it.
>>>
>>> It seems to me that this change is merely cementing into place a few
>>> attributes of the blockchain's history that are not in dispute.
>>>
>>> - Jameson
>>>
>>> On Tue, Nov 15, 2016 at 5:42 PM, Eric Voskuil via bitcoin-dev
>>> <bitcoin-dev@lists.linuxfoundation.org
>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>>
>>>     Actually this does nothing to provide justification for this
>>>     consensus rule change. It is just an attempt to deflect criticism
>>>     from the fact that it is such a change.
>>>
>>>     e
>>>
>>>     > On Nov 15, 2016, at 9:45 AM, Btc Drak <btcdrak@gmail.com
>>>     <mailto:btcdrak@gmail.com>> wrote:
>>>     >
>>>     > I think this is already covered in the BIP text:-
>>>     >
>>>     > "As of November 2016, the most recent of these changes (BIP 65,
>>>     > enforced since December 2015) has nearly 50,000 blocks built on
>>>     top of
>>>     > it. The occurrence of such a reorg that would cause the activating
>>>     > block to be disconnected would raise fundamental concerns about the
>>>     > security assumptions of Bitcoin, a far bigger issue than any
>>>     > non-backwards compatible change.
>>>     >
>>>     > So while this proposal could theoretically result in a consensus
>>>     > split, it is extremely unlikely, and in particular any such
>>>     > circumstances would be sufficiently damaging to the Bitcoin
>>>     network to
>>>     > dwarf any concerns about the effects of this proposed change."
>>>     >
>>>     >
>>>     > On Mon, Nov 14, 2016 at 6:47 PM, Eric Voskuil via bitcoin-dev
>>>     > <bitcoin-dev@lists.linuxfoundation.org
>>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>>     >> NACK
>>>     >>
>>>     >> Horrible precedent (hardcoding rule changes based on the
>>>     assumption that
>>>     >> large forks indicate a catastrophic failure), extremely poor
>>>     process
>>>     >> (already shipped, now the discussion), and not even a material
>>>     performance
>>>     >> optimization (the checks are avoidable once activated until a
>>>     sufficiently
>>>     >> deep reorg deactivates them).
>>>     >>
>>>     >> e
>>>     >>
>>>     >> On Nov 14, 2016, at 10:17 AM, Suhas Daftuar via bitcoin-dev
>>>     >> <bitcoin-dev@lists.linuxfoundation.org
>>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>>     >>
>>>     >> Hi,
>>>     >>
>>>     >> Recently Bitcoin Core merged a simplification to the consensus
>>>     rules
>>>     >> surrounding deployment of BIPs 34, 66, and 65
>>>     >> (https://github.com/bitcoin/bitcoin/pull/8391
>>>     <https://github.com/bitcoin/bitcoin/pull/8391>), and though the
>>>     change is a
>>>     >> minor one, I thought it was worth documenting the rationale in
>>>     a BIP for
>>>     >> posterity.
>>>     >>
>>>     >> Here's the abstract:
>>>     >>
>>>     >> Prior soft forks (BIP 34, BIP 65, and BIP 66) were activated
>>>     via miner
>>>     >> signaling in block version numbers. Now that the chain has
>>>     long since passed
>>>     >> the blocks at which those consensus rules have triggered, we
>>>     can (as a
>>>     >> simplification and optimization) replace the trigger mechanism
>>>     by caching
>>>     >> the block heights at which those consensus rules became enforced.
>>>     >>
>>>     >> The full draft can be found here:
>>>     >>
>>>     >>
>>>     https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki
>>>     <https://github.com/sdaftuar/bips/blob/buried-deployments/bip-buried-deployments.mediawiki>
>>>     >>
>>>     >> _______________________________________________
>>>     >> bitcoin-dev mailing list
>>>     >> bitcoin-dev@lists.linuxfoundation.org
>>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>>     >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>     <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
>>>     >>
>>>     >>
>>>     >> _______________________________________________
>>>     >> bitcoin-dev mailing list
>>>     >> bitcoin-dev@lists.linuxfoundation.org
>>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>>     >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>     <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
>>>     >>
>>>     _______________________________________________
>>>     bitcoin-dev mailing list
>>>     bitcoin-dev@lists.linuxfoundation.org
>>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>>     https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>     <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
>>>
>>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Jorge Timón]

On Wed, Nov 16, 2016 at 2:58 PM, Eric Voskuil via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> This sort of statement represents one consequence of the aforementioned bad
> precedent.
>
> Are checkpoints good now?

Checkpoints are not necessary for consensus and work is being done to
remove them completely from Bitcoin Core in particular.

> Are hard forks okay now?

I personally think uncontroversial hardforks are ok.

> What is the maximum depth of a reorg allowed by this non-machine consensus?

Good question, specially if we plan to do this with future buried
deployments. What about 1 year reorg?

> Shouldn't we just define a max depth so that all cruft deeper than that can
> just be discarded on a regular basis?

Not sure I understand this question.

> Why are there activation heights defined by this hard fork if it's not
> possible to reorg back to them?

If this is a hardfork, it is one that will only be visible if/when
there's a very deep reorg , one of the kind where we can practically
consider Bitcoin done (and only if some nodes keep the ISM code).
But I could accept that definition. Another way to see it (even though
other said the optimization part was not important) as such an
optimization and simplification.
The way I see it, ISM and BIP9 are just coordination mechanisms for
uncontroversial rule changes.
Once the coordination happened and is long in the past, I really don't
see the problem with replacing the mechanism with a simpler height.

> The "BIP" is neither a Proposal (it's been decided, just documenting for
> posterity), nor an Improvement (there is no actual benefit, just some
> tidying up in the notoriously obtuse satoshi code base), nor Bitcoin (a hard
> fork defines an alt coin, so from Aug 4 forward it has been CoreCoin).

Mhmm, I disagree on the notion that any hardfork necessarily
represents an altcoin.
It is certainly an improvement in the sense that it simplifies
implementations and optimizes validation. You may argue that you don't
consider the improvement important though.
These changes to Bitcoin Core could be rolled back (and obviously
other implementations don't need to adopt them unless they want to
benefit from the simplification/optimization or fear such a long
reaorg), but I really hope we don't.

Trying to understand you better...
Accepting your definition of this as a hardfork, do you oppose to it
simply because it is a hardfork, or because you consider this
"hardfork" a bad idea for some reason I am missing?

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Pieter Wuille]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On Wed, Nov 16, 2016 at 5:58 AM, Eric Voskuil via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> This sort of statement represents one consequence of the aforementioned
> bad precedent.
>
> Are checkpoints good now?
>

So far in this discussion, and in a thread that has forked off, I think 3
cases of implementation aspects have been mentioned that under certain
circumstances result in the validity of chains changing:
* Buried softforks (by simplifying the activation rules for certain rules)
* Not verifying BIP30 after BIP34 is active (since only under a SHA256^2
collision a duplicate txid can occur)
* The existence (and/or removal) of checkpoints (in one form or another).

None of these will influence the accepted main chain, however. If they ever
do, Bitcoin has far worse things to worry about (years-deep reorgs, or
SHA256 collisions).

If you were trying to point out that buried softforks are similar to
checkpoints in this regard, I agree. So are checkpoints good now? I believe
we should get rid of checkpoints because they seem to be misunderstood as a
security feature rather than as an optimization. I don't think buried
softforks have that problem.

-- 
Pieter

[-- Attachment #2: Type: text/html, Size: 1697 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

On 11/16/2016 05:50 PM, Pieter Wuille wrote:

> If you were trying to point out that buried softforks are similar to
> checkpoints in this regard, I agree.

Yes, that was my point.

> So are checkpoints good now?
> I believe we should get rid of checkpoints because they seem to be
misunderstood as a security feature rather than as an optimization.

Or maybe because they place control of the "true chain" in the hands of
those selecting the checkpoints? It's not a great leap for the parties
distributing the checkpoints to become the central authority.

I recommend users of our node validate the full chain without
checkpoints and from that chain select their own checkpoints and place
them into config. From that point forward they can apply the
optimization. Checkpoints should never be hardcoded into the source.

> I don't think buried softforks have that problem.

I find "buried softfork" a curious name as you are using it. You seem to
be implying that this type of change is itself a softfork as opposed to
a hardfork that changes the activation of a softfork. It was my
understanding that the term referred to the 3 softforks that were being
"buried", or the proposal, but not the burial itself.

Nevertheless, this proposal shouldn't have "that problem" because it is
clearly neither a security feature nor an optimization. That is the
first issue that needs to be addressed.

e


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Pieter Wuille]

[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]

On Wed, Nov 16, 2016 at 6:16 PM, Eric Voskuil <eric@voskuil.org> wrote:

> On 11/16/2016 05:50 PM, Pieter Wuille wrote:
>


> > So are checkpoints good now?
> > I believe we should get rid of checkpoints because they seem to be
> misunderstood as a security feature rather than as an optimization.
>
> Or maybe because they place control of the "true chain" in the hands of
> those selecting the checkpoints? It's not a great leap for the parties
> distributing the checkpoints to become the central authority.
>

Yes, they can be used to control the "true chain", and this has happened
with various forks. But developers inevitably have this possibility, if you
ignore review. If review is good enough to catch unintended consensus
changes, it is certainly enough to catch the introduction of an invalid
checkpoint. The risk you point out is real, but the way to deal with it is
good review and release practices.

I wish we had never used checkpoints the way we did, but here we are.
Because of this, I want to get rid of them. However, It's not because I
think they offer an excessive power to developers - but because they're
often perceived this way (partially as a result of how they've been abused
in other systems).


> I recommend users of our node validate the full chain without
> checkpoints and from that chain select their own checkpoints and place
> them into config. From that point forward they can apply the
> optimization. Checkpoints should never be hardcoded into the source.
>

Having users with the discipline you suggest would be wonderful to have. I
don't think it's very realistic, though, and I fear that the result would
be that everyone copies their config from one or a few websites "because
that's what everyone uses".

> I don't think buried softforks have that problem.
>
> I find "buried softfork" a curious name as you are using it. You seem to
> be implying that this type of change is itself a softfork as opposed to
> a hardfork that changes the activation of a softfork. It was my
> understanding that the term referred to the 3 softforks that were being
> "buried", or the proposal, but not the burial itself.
>

I do not consider the practice of "buried softforks" to be a fork at all.
It is a change that modifies the validity of a theoretically construable
chain from invalid to valid. However, a reorganization to that theoretical
chain itself is likely already impossible due to the vast number of blocks
to rewind, and economic damage that is far greater than chain divergence
itself.


> Nevertheless, this proposal shouldn't have "that problem" because it is
> clearly neither a security feature nor an optimization. That is the
> first issue that needs to be addressed.


It is clearly not a security feature, agreed. But how would you propose to
avoid the ISM checks for BIP34 and BIP66 all the time? I feel this approach
is a perfectly reasonable choice for code that likely won't ever affect the
valid chain again.

Cheers,

-- 
Pieter

[-- Attachment #2: Type: text/html, Size: 4278 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 6680 bytes --]

On 11/16/2016 06:47 PM, Pieter Wuille wrote:
> On Wed, Nov 16, 2016 at 6:16 PM, Eric Voskuil <eric@voskuil.org
> <mailto:eric@voskuil.org>> wrote:
> 
>     On 11/16/2016 05:50 PM, Pieter Wuille wrote:
> 
>     > So are checkpoints good now?
>     > I believe we should get rid of checkpoints because they seem to be
>     misunderstood as a security feature rather than as an optimization.
> 
>     Or maybe because they place control of the "true chain" in the hands of
>     those selecting the checkpoints? It's not a great leap for the parties
>     distributing the checkpoints to become the central authority.
> 
> Yes, they can be used to control the "true chain", and this has happened
> with various forks. But developers inevitably have this possibility, if
> you ignore review. If review is good enough to catch unintended
> consensus changes, it is certainly enough to catch the introduction of
> an invalid checkpoint. The risk you point out is real, but the way to
> deal with it is good review and release practices.
> 
> I wish we had never used checkpoints the way we did, but here we are.
> Because of this, I want to get rid of them. However, It's not because I
> think they offer an excessive power to developers - but because they're
> often perceived this way (partially as a result of how they've been
> abused in other systems).
>  
>     I recommend users of our node validate the full chain without
>     checkpoints and from that chain select their own checkpoints and place
>     them into config. From that point forward they can apply the
>     optimization. Checkpoints should never be hardcoded into the source.
> 
> Having users with the discipline you suggest would be wonderful to have.
> I don't think it's very realistic, though, and I fear that the result
> would be that everyone copies their config from one or a few websites
> "because that's what everyone uses".

Certainly, but embedding them in the code makes that a practical
certainty. People cannot be prevented from doing dumb things, but let's
not make it hard for them to be smart.

>     > I don't think buried softforks have that problem.
> 
>     I find "buried softfork" a curious name as you are using it. You seem to
>     be implying that this type of change is itself a softfork as opposed to
>     a hardfork that changes the activation of a softfork. It was my
>     understanding that the term referred to the 3 softforks that were being
>     "buried", or the proposal, but not the burial itself.


> I do not consider the practice of "buried softforks" to be a fork at
> all. It is a change that modifies the validity of a theoretically
> construable chain from invalid to valid.

I was out at a Bitcoin meetup when I read this and I think beer actually
came out of my nose.

> However, a reorganization to
> that theoretical chain itself is likely already impossible due to the
> vast number of blocks to rewind, and economic damage that is far greater
> than chain divergence itself.

It's either possible or it is not. If it is not there is no reason for a
proposal - just make the change and don't bother to tell anyone. The
reason we are having this discussion is because it is not impossible.

>     Nevertheless, this proposal shouldn't have "that problem" because it is
>     clearly neither a security feature nor an optimization. That is the
>     first issue that needs to be addressed.
> 
> It is clearly not a security feature, agreed. But how would you propose
> to avoid the ISM checks for BIP34 and BIP66 all the time?

I'll call straw man on the question. It is not important to avoid the
activation checks. The question is whether there is a material
performance optimization in eliminating them. This would have to be
significant enough to rise to the level of a change to the protocol.
Having said that there are a few options:

1. The naive approach to activation is, for each new block, to query the
store for the previous 1000 block headers (to the extent there are that
many), and just do so forever, summing up after the query. This is the
most straightforward but also the most costly approach.

2. A slightly less costly approach is, for each new block, to reverse
iterate over the store until all decisions can be made. This would be an
improvement below activation in that it would take it takes as little as
251 vs. 1000 queries to make the determinations.

3. A further improvement is available by caching the height of full
activation of all three soft forks. Unless there is a subsequent reorg
with a fork point prior that height, there is never a need to make
another query. Once fully activated the activation height is cached to
the store (otherwise just query the last 1000 versions at startup to
determine the state), eliminating any ongoing material cost.

4. We may also be interested in optimizing initial block download. A
cache of the last 1000 block versions can be maintained by adding each
to a circular buffer as they are committed. This eliminates *all*
querying for block versions unless:

(1) there is a restart prior to full activation - in which case there is
a query of up to 1000 versions to prime the cache.

(2) there is a potential reorg after full activation, and the fork point
precedes the saved full activation height - in which case the cache must
be reprimed.

(3) there is a potential reorg. before reaching full activation - in
which case the cache must be backfilled with a query for a number of
versions equal to the depth of the fork point.

During initial block download potential reorgs are exceedingly rare
(reorgs don't have potential unless they have sufficient work to
overcome the long chain) and the cost of handling them as described
above is trivial. The cost of priming the cache is immaterial in the
context of a restart.

So even with a full chain validation one is not likely to *ever* need to
query the store. The memory cost of the cache is strictly 3 bits per
block (375 bytes total). A simpler less memory-sensitive approach is to
use one byte (1,000 bytes total). The computational cost is trivial.

This should already be implemented. A protocol fork (or "change that
modifies the validity of a theoretically construable chain from invalid
to valid") to avoid doing so is not a performance optimization.

> I feel this
> approach is a perfectly reasonable choice for code that likely won't
> ever affect the valid chain again.

I find it to be completely unsupportable as there is no security,
performance, or feature benefit in it.

e





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [bitcoin-dev] [BIP Proposal] Buried Deployments

[Tom Zander]

Here is my thinking.

The BIP process is about changes to a living project which is the bitcoin 
prptocol.
This specific BIP got accepted and we know in the blockchain that
this event (the acceptance) is recorded.
Before a certain block the rules were one way, after they were changed.

I have no problem with changing the *code* to be less complex because it 
already knows the past. A checkpoint is the same, it is the registeration of 
a past event.
This makes software less complex and still capable of checking the entire 
blockchain from genesis.

I don’t see any harm in this change. I see prudent software engineering 
practices.


On Monday, 14 November 2016 10:47:35 CET Eric Voskuil via bitcoin-dev wrote:
> NACK
> 
> Horrible precedent (hardcoding rule changes based on the assumption that
> large forks indicate a catastrophic failure), extremely poor process
> (already shipped, now the discussion), and not even a material
> performance optimization (the checks are avoidable once activated until a
> sufficiently deep reorg deactivates them).


-- 
Tom Zander
Blog: https://zander.github.io
Vlog: https://vimeo.com/channels/tomscryptochannel