[bitcoin-dev] Are 'soft forks' misnamed? [was: Let's deploy BIP65 CHECKLOCKTIMEVERIFY!]

[Gregory Maxwell <gmaxwell@gmail.com>]

On Mon, Sep 28, 2015 at 10:16 PM, Dave Scotese via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Why are they called soft forks when they are really hidden forks?  Isn't the
> point of a soft fork to prevent old clients from rejecting what they don't
> have the code to validate?  That seems dangerous.

As an aside, this list loses utility if people insist on taking
tangential questions to the list in the middle of threads. It's
preferable to either split the thread or take the message off list.

The naming arose from a series of historical naming-by-comparisons:

The bitcoin network has self-arising forks in state when miners
concurrently create blocks. These are natural, unavoidable, and
self-resolving.

If a nodes enforce different and incompatible rules-- for example,
some decide to require that the subsidy stay at 25 BTC forever, then a
fork may come into existence which is not self resolvable.

Thus the term hardfork arose to talk about rule changes which were
incompatible with the existing network, would require all users to
upgrade, would exclude all non-consenting users from the resulting
system, and which have the power to arbitrarily rewrite rules.  This
is in contrast to "forks" which are boring, natural, and happen every
day.

Its often possible to make critical fixes and powerful improvements to
the Bitcoin consensus rules by using the unavoidable power of miners
to filter transactions according to their own rules.  New features and
fixes can be carved out of existing "do anything" space in the
protocol: like carving a statue out of a block of marble. Doing so
reduces the incidence of flag days which are costly to coordinate and
actively risky for users and avoids forcing constant software churn,
which is bad for decentralization. Such changes are a strict narrowing
of permissible actions. And as such, so long as they have a
super-majority hashpower behind them any network forking that happens
to result from them is automatically self-resolving.

So by contrast with hardfork the term softfork came into use to
describe these _compatible_ protocol rule changes.

There is explicit support for compatible rule changes the bitcoin
protocol in the form of no-op opcodes and free form, non-enforced,
version fields (for example). Every fix or enhancement you've heard
about to Bitcoin's consensus rules (going back to the system's
original author) was performed via some form of this mechanism.

In the modern form, the behavior to be soft-forked out is first made
non-standard (if it wasn't already-- they almost always are) meaning
that participants will not relay, mine, or display unconfirmed txn in
their wallets transactions which violate the new rule.  But if a
violation shows up in a block, the block is still accepted.  After
that the blockchain itself is used to coordinate a vast super-majority
of hashpower (recently 95% has been used) agreeing to enforce the new
rule which results in confidence confidence of low disruption on
account of the enforcement. Then when the threshold is reached, they
enforce (automatically).  Old software continues to enforce all the
old rules they always enforced, the only difference in behavior
relates to non-standard transactions and contests between otherwise
valid blocks.  Even unupgraded participants can tell that the network
is doing something new on account of the block version changing (and,
for example, Bitcoin Core warns users about this).

The primary disadvantage of this approach is that it only allowed you
to carve functionality of of "do anything" space, which is quite
natural for some features (especially since the Bitcoin protocol
includes tons of do anything space)--- e.g. height in coinbase, DER
strictness, transactions that have integer overflow creating a
kazillion coins-- but less natural for others.

Of course, it's always possible for the majority of hashpower to have
hidden transaction exclusion rules that _no one_ but them knows about
and this cannot be prevented, but at least the mechanism proscribed in
modern soft-forks is transparent (the network tells you that its doing
something you don't understand).