[bitcoin-dev] BIP draft: Hardfork bit

[bitcoin-dev] BIP draft: Hardfork bit

[jl2012]

Please feel free to comment, for technical issues and language

BIP: ??
Title: Hardfork bit
Author: jl2012 <jl2012@xbt.hk>
Status: Draft
Type: Standard Track
Created: 2015-07-23

Abstract

This document specifies a proposed change to the semantics of the most  
significant bit of the “version” field in Bitcoin block headers, as a  
mechanism to indicate a hardfork is deployed. It alleviates certain  
risks related to a hardfork by introducing an explicit “point of no  
return” in the blockchain. This is a general mechanism which should be  
employed by any planned hardfork in the future.

Motivation

Hardforks in Bitcoin are usually considered as difficult and risky, because:

1) Hardforks require not only support of miners, but also, most  
importantly, supermajority support of the Bitcoin economy. As a  
result, softfork deployment mechanisms described in BIP 34 or BIP XX  
“Version bits” (https://gist.github.com/sipa/bf69659f43e763540550) are  
not enough for introducing hardforks safely.

2) Full nodes and SPV nodes following original consensus rules may not  
be aware of the deployment of a hardfork. They may stick to an  
economic-minority fork and unknowingly accept devalued legacy tokens.

3) In the case which the original consensus rules are also valid under  
the new consensus rules, users following the new chain may  
unexpectedly reorg back to the original chain if it grows faster than  
the new one. People may find their confirmed transactions becoming  
unconfirmed and lose money.

The first issue involves soliciting support for a hardfork proposal,  
which is more a political topic than a technical one. This proposal  
aims at alleviating the risks related to the second and third issues.  
It should be employed by any planned hardfork in the future.

Definitions

See BIP YY “Motivation and deployment of consensus rules changes”
https://github.com/jtimon/bips/blob/bip-forks/bip-forks.org

Specification

Hardfork bit: The most significant bit in nVersion is defined as the  
hardfork bit. Currently, blocks with this header bit setting to 1 are  
invalid, since BIP34 interprets nVersion as a signed number and  
requires it to be >=2 (with BIP66, >=3). Among the 640 bits in the  
block header, this is the only one which is fixed and serves no  
purpose, and therefore the best way to indicate the deployment of a  
hardfork.

Flag block: Any planned hardfork must have one and only one flag block  
which is the “point of no return”. To ensure monotonicity, flag block  
should be determined by block height, or as the first block with  
GetMedianTimePast() greater than a threshold. Other mechanisms could  
be difficult for SPV nodes to follow. The height/time threshold could  
be a predetermined value or relative to other events (e.g. 1000 blocks  
/ 10 days after 75% of miner support). The exact mechanism is out of  
the scope of this BIP. No matter what mechanism is used, the threshold  
is consensus critical. It must be publicly verifiable with only  
blockchain data and the programme source code, and preferably  
SPV-friendly.

Flag block is constructed in a way that nodes with the original  
consensus rules must reject. On the other hand, nodes with the new  
consensus rules must reject a block if it is not a flag block while it  
is supposed to be. To achieve these goals, the flag block must 1) have  
the hardfork bit setting to 1, 2) include a short predetermined unique  
description of the hardfork anywhere in its coinbase, and 3) follow  
any other rules required by the hardfork. If these conditions are not  
fully satisfied, upgraded nodes shall reject the block.

The hardfork bit must be turned off in the decedents of the flag  
block, until the deployment of the next hardfork. The requirement of  
coinbase message is also limited to the flag block. In the rare case  
that multiple hardforks share the same flag block, the coinbase shall  
include all relevant messages and the order/position of the messages  
shall not be consensus critical.

Although a hardfork is officially deployed after the flag block, the  
exact behavioural change is out of the scope of this BIP. For example,  
a hardfork may not be fully active until certain time after the flag  
block.

Automatic warning system: When a flag block is found on the network,  
full nodes and SPV nodes should look into its coinbase. They should  
alert their users and/or stop accepting incoming transactions if it is  
an unknown hardfork. It should be noted that the warning system could  
become a DoS vector if the attacker is willing to give up the block  
reward. Therefore, the warning may be issued only if a few blocks are  
built on top of the flag block in a reasonable time frame. This will  
in turn increase the risk in case of a real planned hardfork so it is  
up to the wallet programmers to decide the optimal strategy. Human  
warning system (e.g. the emergency alert system in Bitcoin Core) could  
fill the gap.

Compatibility

As a mechanism to indicate hardfork deployment, this BIP breaks  
backward compatibility intentionally. However, without further changes  
in the block header format, full nodes and SPV nodes could still  
verify the PoW of a flag block and its descendants.

This proposal is also compatible with the BIP XX “Version bits”. The  
version bits mechanism could be employed to measure miner support  
towards a hardfork proposal, and to determine the height or time  
threshold of the flag block. Also, miners of the flag block may still  
cast votes for other concurrent softfork or hardfork proposals as  
normal.

After the flag block is generated, a miner may support either fork but  
not both. It is not possible for miners in one fork to attack or  
overtake the other fork because the forks are mutually exclusive.

Re: [bitcoin-dev] BIP draft: Hardfork bit

[Tier Nolan]

[-- Attachment #1: Type: text/plain, Size: 2970 bytes --]

On Thu, Jul 23, 2015 at 5:23 PM, jl2012 via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> 2) Full nodes and SPV nodes following original consensus rules may not be
> aware of the deployment of a hardfork. They may stick to an
> economic-minority fork and unknowingly accept devalued legacy tokens.
>

This change means that they are kicked off the main chain immediately when
the fork activates.

The change is itself a hard fork.  Clients have be updated to get the
benefits.

3) In the case which the original consensus rules are also valid under the
> new consensus rules, users following the new chain may unexpectedly reorg
> back to the original chain if it grows faster than the new one. People may
> find their confirmed transactions becoming unconfirmed and lose money.
>

I don't understand the situation here.  Is the assumption of a group of
miners suddenly switching (for example, they realise that they didn't
intend to support the new rules)?

>
> Flag block is constructed in a way that nodes with the original consensus
> rules must reject. On the other hand, nodes with the new consensus rules
> must reject a block if it is not a flag block while it is supposed to be.
> To achieve these goals, the flag block must 1) have the hardfork bit
> setting to 1, 2) include a short predetermined unique description of the
> hardfork anywhere in its coinbase, and 3) follow any other rules required
> by the hardfork. If these conditions are not fully satisfied, upgraded
> nodes shall reject the block.
>

Ok, so set the bit and then include BIP-GIT-HASH of the canonical BIP on
github in the coinbase?

Since it is a hard fork, the version field could be completely
re-purposed.  Set the bit and add the BIP number as the lower bits in the
version field.  This lets SPV clients check if they know about the hard
fork.

There network protocol could be updated to add getdata support for asking
for a coinbase only merkleblock.  This would allow SPV clients to obtain
the coinbase.

Automatic warning system: When a flag block is found on the network, full
> nodes and SPV nodes should look into its coinbase. They should alert their
> users and/or stop accepting incoming transactions if it is an unknown
> hardfork. It should be noted that the warning system could become a DoS
> vector if the attacker is willing to give up the block reward. Therefore,
> the warning may be issued only if a few blocks are built on top of the flag
> block in a reasonable time frame. This will in turn increase the risk in
> case of a real planned hardfork so it is up to the wallet programmers to
> decide the optimal strategy. Human warning system (e.g. the emergency alert
> system in Bitcoin Core) could fill the gap.
>

If the rule was that hard forks only take effect 100 blocks after the flag
block, then this problem is eliminated.

Emergency hard forks may still have to take effect immediately though, so
it would have to be a custom not a rule.

[-- Attachment #2: Type: text/html, Size: 3836 bytes --]

Re: [bitcoin-dev] BIP draft: Hardfork bit

[jl2012]

Quoting Tier Nolan via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>:

> On Thu, Jul 23, 2015 at 5:23 PM, jl2012 via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> 2) Full nodes and SPV nodes following original consensus rules may not be
>> aware of the deployment of a hardfork. They may stick to an
>> economic-minority fork and unknowingly accept devalued legacy tokens.
>>
>
> This change means that they are kicked off the main chain immediately when
> the fork activates.
>
> The change is itself a hard fork.  Clients have be updated to get the
> benefits.

I refrain from calling it the "main chain". I use "original chain" and  
"new chain" instead as I make no assumption about the distribution of  
mining power. This BIP still works when we have a 50/50 hardfork. The  
main point is to protect all users on both chains, and allow them to  
make an informed choice.


> 3) In the case which the original consensus rules are also valid under the
>> new consensus rules, users following the new chain may unexpectedly reorg
>> back to the original chain if it grows faster than the new one. People may
>> find their confirmed transactions becoming unconfirmed and lose money.
>>
>
> I don't understand the situation here.  Is the assumption of a group of
> miners suddenly switching (for example, they realise that they didn't
> intend to support the new rules)?
>

Again, as I make no assumption about the mining power distribution,  
the new chain may actually have less miner support. Without any  
protection (AFAIK, for example, BIP100, 101, 102), the weaker new  
chain will get 51%-attacked by the original chain constantly.


>>
>> Flag block is constructed in a way that nodes with the original consensus
>> rules must reject. On the other hand, nodes with the new consensus rules
>> must reject a block if it is not a flag block while it is supposed to be.
>> To achieve these goals, the flag block must 1) have the hardfork bit
>> setting to 1, 2) include a short predetermined unique description of the
>> hardfork anywhere in its coinbase, and 3) follow any other rules required
>> by the hardfork. If these conditions are not fully satisfied, upgraded
>> nodes shall reject the block.
>>
>
> Ok, so set the bit and then include BIP-GIT-HASH of the canonical BIP on
> github in the coinbase?

I guess the git hash is not known until the code is written? (correct  
me if I'm wrong) As the coinbase message is consensus-critical, it  
must be part of the source code and therefore you can't use any kind  
of hash of the code itself (a chicken-and-egg problem)

> Since it is a hard fork, the version field could be completely
> re-purposed.  Set the bit and add the BIP number as the lower bits in the
> version field.  This lets SPV clients check if they know about the hard
> fork.

This may not be compatible with the other version bits voting mechanisms.

> There network protocol could be updated to add getdata support for asking
> for a coinbase only merkleblock.  This would allow SPV clients to obtain
> the coinbase.

Yes


> Automatic warning system: When a flag block is found on the network, full
>> nodes and SPV nodes should look into its coinbase. They should alert their
>> users and/or stop accepting incoming transactions if it is an unknown
>> hardfork. It should be noted that the warning system could become a DoS
>> vector if the attacker is willing to give up the block reward. Therefore,
>> the warning may be issued only if a few blocks are built on top of the flag
>> block in a reasonable time frame. This will in turn increase the risk in
>> case of a real planned hardfork so it is up to the wallet programmers to
>> decide the optimal strategy. Human warning system (e.g. the emergency alert
>> system in Bitcoin Core) could fill the gap.
>>
>
> If the rule was that hard forks only take effect 100 blocks after the flag
> block, then this problem is eliminated.
>
> Emergency hard forks may still have to take effect immediately though, so
> it would have to be a custom not a rule.

The flag block itself is a hardfork already and old miners will not  
mine on top of the flag block. So your suggestion won't be helpful in  
this situation.

To make it really meaningful, we need to consume one more bit of the  
'version' field ("notice bit"). Supporting miners will turn on the  
notice bit, and include a message in coinbase ("notice block"). When a  
full node/SPV node find many notice blocks with the same coinbase  
message, they could bet that the subsequent flag block is a legit one.  
However, an attacker may still troll you by injecting an invalid flag  
block after many legit notice blocks. So I'm not sure if it is worth  
the added complexity.

Re: [bitcoin-dev] BIP draft: Hardfork bit

[Gareth Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

>I don't understand the situation here.  Is the assumption of a group of
>miners suddenly switching (for example, they realise that they didn't
>intend to support the new rules)?

Or they're economically rational miners, and a large difficulty decrease on the original chain, without an equally large decrease in the value of original chain tokens, has made it profitable to switch?

It's dangerous to assume all miners will continue to support the side they have initially signaled. They're only invested in the chain they mine for a short time (until coinbase maturity.)

If both sides if a fork retain value you'd expect mining to redistribute itself WRT short term profitability at every difficulty adjustment, irrespective of initially signaled support for the fork.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFABAEBCgAqBQJVsZIuIxxHYXJldGggV2lsbGlhbXMgPGdhY3J1eEBnbWFpbC5j
b20+AAoJEEY5w2E3jkVEN7YIAIlgaAahHCssIEXYzqB1gRKYRP4fPsq8NtOMrkki
dc1gfKmprlPDShFvu2Hn5L8amP51ouRpmDSJwNyU//1DyU5p1tcWTAtkHr6SY7TY
uJtcPMM03BUD2i3rXSY4FbpWn8aOoUnQrkYFhx5Y/Aru+l47C0I5KF4fgMag7FhI
RxkTFylvq7uWvu0QCUkVh1MgohNxMqIGAvE5t8yoj5LxrNzOq95TcOGwFngWCdJM
a5BADFjq7v4j/+cP748ZTPcLUusTQTwEuIsCIhpwwiKADsy1FKjmAdHhKTff/6wn
cpNYvwimKNSSESCwzAnxekaJCTXpEOWQV7/6FO9vJbTMKw8=
=/hJk
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] BIP draft: Hardfork bit

[Michael Ruddy]

>> Ok, so set the bit and then include BIP-GIT-HASH of the canonical BIP on
>> github in the coinbase?
>
>
> I guess the git hash is not known until the code is written? (correct me if
> I'm wrong) As the coinbase message is consensus-critical, it must be part of
> the source code and therefore you can't use any kind of hash of the code
> itself (a chicken-and-egg problem)
>

I took Tier's comment to mean that the commit hash (taken from git) of
the BIP for the particular hard fork being rolled out via your hard
fork bit mechanism could be used in the coinbase.
The commit for the BIP is separate from the commit for the code
changes implementing it.

By using the commit hash of the BIP, it means that the BIP cannot
specify itself what to put in the coinbase.
I'd suggest that your hard fork bit proposal allows BIP authors to
specify, within a 20 byte limit (to let them ripe160 a message or
something), the unique value to use.
This would be just as well and would allow the specific hard fork BIPs
to be updated without having to make code changes (e.g.- for simple
grammatical updates, post deployment clarification, etc...).

Perhaps more preferable, in order to keep the coinbases small, or, if
you're worried about BIP authors using duplicate values, then just
specify in your proposal that the coinbase message include
"BIP<NUMBER>" as BIP numbers are not going to be reused. All you are
trying to achieve is something that can be uniquely pattern matched.

Are the reasons for your use of the coinbase message 1) to guard
against one or more hard forks piggy-backing on another's flag block,
without prior collaboration, and 2) to have a nicer message in the
alerting system? If so, you might want to spell that out in your
proposal. At first, I was thinking that the coinbase thing was not
entirely necessary since the possibility of multiple hard forks taking
place concurrently is low. Multiple forking changes could be
coordinated to all use the same possible voting mechanisms, median
timestamp, and thus flag block. But if the concern is about competing
hard forks where two or more forks abandon the original chain and
attempt to use the same checkpoint block, then I can see why you'd
need it.

I was reading this proposal alongside another of your messages
(https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009520.html)
proposing a version 0 or even a >1MB block in the specific case of a
block size limit hard fork. I can see the >1MB flag block creating DoS
banning problems. That's more of a practical issue than a consensus
issue due to the existing mono-culture of full nodes. So, currently a
>1MB flag block is not as appealing as a version 0 or this hard fork
bit proposal. Besides the DoS ban, the >1MB proposal would mean that
older nodes do not have the chance to notice a fork is happening (for
alerting) as they would with a version 0 or hard fork bit.

A version 0 flag block would not be able to contain any transactions
unless the flag block version was assumed to be that of the most
recent version at the time. This is because we'd want to enforce the
rules of the previous soft forks that had been locked in. Other than
that, the version 0 idea seems pretty similar to the hard fork bit
proposal because you need more context than just the block itself to
determine if it's a valid flag block (and this extra context is
probably not great, but I don't have a better idea right now).

Were those reasons part of why you progressed towards this hard fork
bit proposal?

Re: [bitcoin-dev] BIP draft: Hardfork bit

[jl2012]

Although the chance is very slim, it is possible to have multiple 
hardforks sharing the same flag block. For example, different proposals 
may decide the flag time based on voting result and 2 proposals may have 
the same flag time just by chance. The coinbase message is to preclude 
any potential ambiguity. It also provides additional info to warning 
system of non-upgrading nodes.

If we are pretty sure that there won't be other hardfork proposal at the 
same time, the coinbase message may not be necessary. With some prior 
collaboration, this may also be avoided (e.g. no sharing flag block 
allowed as consensus rules of the hardforks)

The "version 0" idea is not compatible with the version bits voting 
system, so I have this hardfork bit BIP after thinking more carefully. 
Otherwise they are technically similar.

Michael Ruddy 於 2015-08-01 09:05 寫到:
>>> Ok, so set the bit and then include BIP-GIT-HASH of the canonical BIP 
>>> on
>>> github in the coinbase?
>> 
>> 
>> I guess the git hash is not known until the code is written? (correct 
>> me if
>> I'm wrong) As the coinbase message is consensus-critical, it must be 
>> part of
>> the source code and therefore you can't use any kind of hash of the 
>> code
>> itself (a chicken-and-egg problem)
>> 
> 
> I took Tier's comment to mean that the commit hash (taken from git) of
> the BIP for the particular hard fork being rolled out via your hard
> fork bit mechanism could be used in the coinbase.
> The commit for the BIP is separate from the commit for the code
> changes implementing it.
> 
> By using the commit hash of the BIP, it means that the BIP cannot
> specify itself what to put in the coinbase.
> I'd suggest that your hard fork bit proposal allows BIP authors to
> specify, within a 20 byte limit (to let them ripe160 a message or
> something), the unique value to use.
> This would be just as well and would allow the specific hard fork BIPs
> to be updated without having to make code changes (e.g.- for simple
> grammatical updates, post deployment clarification, etc...).
> 
> Perhaps more preferable, in order to keep the coinbases small, or, if
> you're worried about BIP authors using duplicate values, then just
> specify in your proposal that the coinbase message include
> "BIP<NUMBER>" as BIP numbers are not going to be reused. All you are
> trying to achieve is something that can be uniquely pattern matched.
> 
> Are the reasons for your use of the coinbase message 1) to guard
> against one or more hard forks piggy-backing on another's flag block,
> without prior collaboration, and 2) to have a nicer message in the
> alerting system? If so, you might want to spell that out in your
> proposal. At first, I was thinking that the coinbase thing was not
> entirely necessary since the possibility of multiple hard forks taking
> place concurrently is low. Multiple forking changes could be
> coordinated to all use the same possible voting mechanisms, median
> timestamp, and thus flag block. But if the concern is about competing
> hard forks where two or more forks abandon the original chain and
> attempt to use the same checkpoint block, then I can see why you'd
> need it.
> 
> I was reading this proposal alongside another of your messages
> (https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009520.html)
> proposing a version 0 or even a >1MB block in the specific case of a
> block size limit hard fork. I can see the >1MB flag block creating DoS
> banning problems. That's more of a practical issue than a consensus
> issue due to the existing mono-culture of full nodes. So, currently a
>> 1MB flag block is not as appealing as a version 0 or this hard fork
> bit proposal. Besides the DoS ban, the >1MB proposal would mean that
> older nodes do not have the chance to notice a fork is happening (for
> alerting) as they would with a version 0 or hard fork bit.
> 
> A version 0 flag block would not be able to contain any transactions
> unless the flag block version was assumed to be that of the most
> recent version at the time. This is because we'd want to enforce the
> rules of the previous soft forks that had been locked in. Other than
> that, the version 0 idea seems pretty similar to the hard fork bit
> proposal because you need more context than just the block itself to
> determine if it's a valid flag block (and this extra context is
> probably not great, but I don't have a better idea right now).
> 
> Were those reasons part of why you progressed towards this hard fork
> bit proposal?

Re: [bitcoin-dev] BIP draft: Hardfork bit

[Michael Ruddy]

I think your "hardfork bit" proposal is clever.
It addresses the particular valid concern of re-org facing users of a
fork that a small/near/fluctuating majority, or less, of mining power
supported.
While the "economic majority" argument may be enough on its own in
that case, it still has some aspect of being a hand wave.
This proposal adds support to those economic actors, which makes it
easier for them to switch if/when they choose. That is, it provides a
good fallback mechanism that allows them to make a decision and say,
"we're doing this".
Do you have the latest version up on github, or someplace where it
would be easier to collaborate on the specific text?

On Sat, Aug 1, 2015 at 4:23 PM,  <jl2012@xbt.hk> wrote:
> Although the chance is very slim, it is possible to have multiple hardforks
> sharing the same flag block. For example, different proposals may decide the
> flag time based on voting result and 2 proposals may have the same flag time
> just by chance. The coinbase message is to preclude any potential ambiguity.
> It also provides additional info to warning system of non-upgrading nodes.
>
> If we are pretty sure that there won't be other hardfork proposal at the
> same time, the coinbase message may not be necessary. With some prior
> collaboration, this may also be avoided (e.g. no sharing flag block allowed
> as consensus rules of the hardforks)
>
> The "version 0" idea is not compatible with the version bits voting system,
> so I have this hardfork bit BIP after thinking more carefully. Otherwise
> they are technically similar.

Re: [bitcoin-dev] BIP draft: Hardfork bit

[jl2012]

I have put it on the github: 
https://github.com/jl2012/bips/blob/master/hardforkbit.mediawiki

I removed the specification of coinbase message to make it simpler. 
Instead, it requires that a flag block must not be shared by multiple 
hardfork proposals.

I'm not sure whether it is a Standard, Informational, or Process BIP

I'm also thinking whether we should call it "hardfork bit", "hardfork 
flag", or with other name.

Michael Ruddy 於 2015-08-02 06:53 寫到:
> I think your "hardfork bit" proposal is clever.
> It addresses the particular valid concern of re-org facing users of a
> fork that a small/near/fluctuating majority, or less, of mining power
> supported.
> While the "economic majority" argument may be enough on its own in
> that case, it still has some aspect of being a hand wave.
> This proposal adds support to those economic actors, which makes it
> easier for them to switch if/when they choose. That is, it provides a
> good fallback mechanism that allows them to make a decision and say,
> "we're doing this".
> Do you have the latest version up on github, or someplace where it
> would be easier to collaborate on the specific text?