[bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Thu, Jul 23, 2015 at 2:49 AM, Eric Voskuil via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> On 07/22/2015 05:13 PM, Eric Lombrozo via bitcoin-dev wrote:
>> Only being partly serious - I strongly am in favor of a sufficiently
> modularized codebase that swapping out consensus rules is fairly
> straightforward and easy to test...
>
> We (libbitcoin) have taken the time to publish and maintain bitcoind's
> "libbitcoinconsensus" source files as an independent C++ library (with
> Java and Python bindings).
>
> https://en.bitcoin.it/wiki/Libbitcoin_Consensus
>
> It can be easily verified against bitcoind sources and in builds of
> libbitcoin-blockchain it can be swapped out for libbitcoin's native
> consensus checks.
>
> https://en.bitcoin.it/wiki/Libbitcoin_Blockchain#Consensus_Validation
>
> So there is really no reason to consider the original client synonymous
> with consensus. I initially argued for this library to be natively
> isolated from bitcoind, but that didn't seem to be in the cards so we
> did it independently.

I think there were some misunderstandings in our previous conversation
about this topic.
I completely agree with having a separated repository for libconsensus
(that's the whole point, alternative implementations can be
consensus-safe by using it, and in the event of a schism fork[1], they
can fork just that smaller project without having to relay on Bitcoin
Core [satoshi] at all).
But I thought you also wanted Bitcoin Core to use libconsensus instead
of just having a subtree/subrepository like it currently does with
libsecp256k1.
I'm not sure if that would ever be accepted, but in any case we're
certainly far away from that goal. Here are some things that need to
happen first:

1) Finish encapsulating consensus code so that it can be built without
any (we've done it only with script-related code so far). Here are
some related PRs (other people havee done other things that help with
this as well):

** MERGED or DELETED
*** MERGED Consensus: Decouple pow from chainparams #5812 [consensuspow]
*** DELETED MOVEONLY: Move constants and globals to consensus.h and
policy.o #5696 [consensus_policy0]
*** DELETED Refactor: Create CCoinsViewEfficient interface for
CCoinsViewCache #5747 [coins]
*** MERGED Chainparams: Refactor: Decouple IsSuperMajority from
Params() #5968 [params_consensus]
*** MERGED Remove redundant getter
CChainParams::SubsidyHalvingInterval() #5996 [params_subsidy]
*** MERGED Separate CValidationState from main #5669 [consensus]
*** DELETED Consensus: Refactor: Separate CheckFinalTx from
main::IsFinalTx #6063 [consensus_finaltx]
*** MERGED Consensus: Decouple ContextualCheckBlockHeader from
checkpoints #5975 [consensus_checkpoints]
*** MERGED Separate Consensus::CheckTxInputs and GetSpendHeight in
CheckInputs #6061 [consensus_inputs]
*** MERGED Bugfix: Don't check the genesis block header before
accepting it #6299 [5975-quick-fix]
** REVIEW Optimizations: Consensus: In AcceptToMemoryPool,
ConnectBlock, and CreateNewBlock #6445 [consensus-txinputs-0.12.99]
** REBASE MOVEONLY: Move most of consensus functions (pre-block) #6051
[consensus_moveonly]
** REBASE Consensus: Refactor: Turn CBlockIndex::GetMedianTimePast
into independent function #6009 [consensus_mediantime]
** DEPENDENT Consensus: Refactor: Consensus version of
CheckBlockHeader() #6035 [consensus_checkblockheader]
** DEPENDENT Consensus: Consensus version of pow functions [consensus_pow2]

2) Finish libconsensus's API: expose more things than VerifyScript, at
the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
Feedback from alternative implementations like libbitcoin is extremely
valuable here. Some related closed-for-now PRs:

** DEPENDENT API: Expose bitcoinconsensus_verify_header() in
libconsensus #5995 [consensus_header]
** DEPENDENT API: Expose bitcoinconsensus_verify_block() in
libconsensus #5946 [consensus_tip]
** REBASE Chainparams: Explicit Consensus::Params arg in consensus
functions #6024 [params_consensus2]

3) Move libconsensus to a separate repository as a
subtree/subrepository of Bitcoin Core.

Only after all that we can discuss whether Bitcoin Core itself should
include libconsensus' code or just use its API directly.
I hope that after all this, libbitcoin also reconsiders whether to
reimplement its own libconsensus or use the "official" one directly
instead.

> In any case I agree with your stated need for this isolation (if not the
> means) for the reasons you state. The community needs to move beyond a
> largely singular and monolithic codebase that is holding that position
> in part due to fear about consensus bug forks.

I completely agree. That's the goal of libconsensus (and an
alternative implementation like libbitcoin being able to use it
without sacrificing any of its current or future design differences
from Bitcoin Core would be a sign of success in this reward).
Unfortunately any changes that touch consensus code are risky and
therefore slow. And when consensus encapsulation changes conflict with
other changes (not because the other changes need to change consensus
but because consensus code is still coupled with policy and other
bitcoind-specific code), refactors are never prioritized. Ironically,
you need to encapsulate the consensus code to avoid such conflicts,
which would make all non-consensus changes far less risky (reducing
the consensus-critical review development bottleneck).

Unfortunately and ironically again, safer, small and incremental
changes are less interesting for reviewers.
For example, I've been trying to move consensus code to the consensus
folder for a long time. The correctness of a MOVEONLY change is
trivial to review for anyone who knows how to copy/paste in its
favorite editor and how to use git diff, but will I ever get answers
to my questions in [1]?
I know there's many people who really care about this, Cory Fields,
Wladimir and Pieter Wuille to name a few have reviewed many of this
changes (I've just got used to publicly whine about lack of review on
this front and policy encapsulation [very related fronts] as an
attempt to get some attention: not always, but begging for review
actually works some times).

Another unfortunate fact is that although a script-only libconsensus
allows you to avoid a big part of all possible consensus fork bugs,
there cannot be users of a finished libconsensus to ask things to util
a finished libconsensus actually exists. At the same time, the future
users (alternative implementations, since bitcoin core is already
"using libconsensus") are the most relevant people to listen when it
comes to the C API. That's why I beg you to comment on [2], even if
#5995 is currently closed. Your input on [1] would be very appreciated
as well (maybe you think it's better to expose verifyTx before
exposing verifyHeader, even if exposing verifyHeader is something that
could be done faster).

 > To make choice regarding consensus an actual choice (and thereby actual
> consensus) the modularity you suggest is essential. One must be able to
> take new developments without having to take consensus changes. The
> option to fork the codebase is not reasonable for most people. At this
> point there is no defensible reason for coupling consensus checks with
> other features.

Would you agree that asking people to fork an independent libconsensus
project instead of having to fork the full Bitcoin-qt is much more
reasonable?
I mean, I agree with your points. If "the specification of the
consensus rules is an implementation", then that implementation
shouldn't be coupled with a bunch of policy and non-consensus
technical choices (storage, dependencies, p2p protocol...). But I
still think that "the specification of the consensus rules should be a
concrete implementation" rather than based purely on a natural
language like English.
I believe that's the only point where we fundamentally disagree, but
it shouldn't be a barrier in our common goal of taking "power" away
from Bitcoin Core development. If we're successful Bitcoin Core won't
have any privileged position with regards to, say, libbitcoin when it
comes to deciding consensus rules changes.
You see, people like Mike Hearn believe that "uncontroversial
acceptance by Bitcoin Core devs" is the same as "uncontroversial
acceptance by all users of the system" (for a libbitcoin developer
like you, obviously a superset of Bitcoin Core's users). He thinks
that Gavin proposal is only a schism consensus fork[3] because the
code is in github/bitcoinxt/bitcoinxt instead of
github/bitcoin/bitcoin, not because PeterTodd-the-user-of-the-system
(he doesn't care about him) opposes it.
But let's imagine a different situation:

1) libconsensus us finished and used by libbitcoin
2) Bitcoin Core was unanimously in favor of Gavin's 32 GB initial
proposal and the changes are applied to bitcoin/bitcoin and
bitcoin/libconsensus (or Bitcoin Core has a dictator like Mike
wants[4] and he accepts it, it doesn't really matter for this
example).

But let's also assume that X% of the users and 10% of the miners are
against that Schism hardfork, and they don't want to be forced to
change the rules by any influential group, mining, economic or user
majority.
Libbitcoin cannot be forced to accept the next, controversial version
of bitcoin/libconsensus, so you guys fork libbitcoin/libconsensus out
of the last ok version.
Centralized-bitcoin and old-bitcoin would become 2 separated
currencies and some people would likely lose money in the transition
from one currency to 2 of them, but the users of old-bitcoin have the
right of keeping the rules they signed up for and the only responsible
people for this likely-catastrophic schism would be the Bitcoin core
developers for trying to impose consensus changes into others against
their will. Trying to impose consensus changes against the will of
some users is wrong, and it is irrelevant if that happens in Bitcoin
Core or Bitcoin Tx (if it is uncontroversial, it's also irrelevant
where it gets implemented first).

I really believe bitcoin needs competitive alternative implementations
and I believe libconsensus is a tool to help that happen and reduce
the "gatekeeping" friction that there's (unfortunately) around Bitcoin
Core. I look forward to any potential collaboration on this front.
Even if you still want to maintain a reimplementation of libconsensus
(which I humbly think it's a mistake, but I don't think there's any
point on keep discussing that, since we know we disagree) we can
collaborate on the future common API of a complete libconsensus (with
verifyBlock and all). I really hope we can do that.

[1] https://github.com/bitcoin/bitcoin/pull/6051#issuecomment-120708121

[2] (kind of outdated, but the API is what matters here)
https://github.com/jtimon/bitcoin/commit/00b9b227afc8669a877984561329dde75d3d8942

[3] https://github.com/jtimon/bips/blob/bip-forks/bip-forks.org#schism1-hardforks

[4] http://lmgtfy.com/?q=mike+hearn+dictator&l=1

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Milly Bitcoin]

On 7/23/2015 10:30 AM, Jorge Timón via bitcoin-dev wrote:

> [4] http://lmgtfy.com/?q=mike+hearn+dictator&l=1
> _______________________________________________


You spend too much time on reddit.  All this drama queen stuff is 
getting ridiculous.


Russ

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Thu, Jul 23, 2015 at 4:57 PM, Milly Bitcoin via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> On 7/23/2015 10:30 AM, Jorge Timón via bitcoin-dev wrote:
>
>> [4] http://lmgtfy.com/?q=mike+hearn+dictator&l=1

Mike has sincerely said that he would like "Bitcoin Core to have a
benevolent dictator like other free software projects", and I wanted
to make clear that I wasn't putting words in his mouth but it's
actually something very easy to find on the internet. But I now
realize that the search can be interpreted as me calling him dictator
or something of the sort. That wasn't my intention. In fact, Mike's
point of view on Bitcoin Core development wasn't even relevant for my
example so I shouldn't even have mentioned him in the first place. I
apologize for both mistakes, but please let's keep this thread focused
on libconsensus.

> You spend too much time on reddit.

I actually don't spend much time on reddit: I don't particularly like
it. But I do spend some time in reddit so, I agree: I spend too much
time on reddit.

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Milly Bitcoin]

 > Mike has sincerely said that he would like "Bitcoin Core to have a
 > benevolent dictator like other free software projects", and I wanted
 > to make clear that I wasn't putting words in his mouth

He is just pointing out reality. Decentralization is really just a 
collection of centralized processes. If you ever want to change anything 
someone, somewhere has to make a decision.

Satoshi was the first dictator and he set the consensus rules for his 
version of Bitcoin and that is what people accepted. Those rules did not 
appear by magic nor did they appear via some unexplained 'consensus' 
mechanism. He made decisions and updated the code or accepted updates 
made by others.

Russ

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 11562 bytes --]

On 07/23/2015 07:30 AM, Jorge Timón wrote:
> On Thu, Jul 23, 2015 at 2:49 AM, Eric Voskuil via bitcoin-dev wrote:
>> On 07/22/2015 05:13 PM, Eric Lombrozo via bitcoin-dev wrote:
>>> Only being partly serious - I strongly am in favor of a sufficiently
>>> modularized codebase that swapping out consensus rules is fairly
>>> straightforward and easy to test...
>>
>> We (libbitcoin) have taken the time to publish and maintain bitcoind's
>> "libbitcoinconsensus" source files as an independent C++ library...

> I think there were some misunderstandings in our previous conversation
> about this topic.
> I completely agree with having a separated repository for libconsensus
> (that's the whole point, alternative implementations can be
> consensus-safe by using it, and in the event of a schism fork[1], they
> can fork just that smaller project without having to relay on Bitcoin
> Core [satoshi] at all).
> But I thought you also wanted Bitcoin Core to use libconsensus instead
> of just having a subtree/subrepository like it currently does with
> libsecp256k1.

libsecp256k1 has it's own repository, libbitcoinconsensus doesn't.  A
separate repository was what I considered as a requirement for us to use it.

> I'm not sure if that would ever be accepted, but in any case we're
> certainly far away from that goal.

If it's not certain whether this would even be accepted, the commitment
to a community consensus library is too weak to take a strong dependency
on. But for us it's moot, as we have made the already accomplished that
goal.

> Here are some things that need to
> happen first:
> 
> 1) Finish encapsulating consensus code so that it can be built without
> any (we've done it only with script-related code so far). Here are
> some related PRs (other people have done other things that help with
> this as well):
...
> 2) Finish libconsensus's API: expose more things than VerifyScript, at
> the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
> Feedback from alternative implementations like libbitcoin is extremely
> valuable here. Some related closed-for-now PRs:

In our earlier discussion I believe you said that the library would not
be undergoing significant change or feature creep. If this is the very
least that's projected it would seem that constraint will not hold.

> 3) Move libconsensus to a separate repository as a
> subtree/subrepository of Bitcoin Core.
> 
> Only after all that we can discuss whether Bitcoin Core itself should
> include libconsensus' code or just use its API directly.

I don't think it's a question of whether it *should* use its own library
as it is published for others - this is a practically self-evident
conclusion.

> I hope that after all this, libbitcoin also reconsiders whether to
> reimplement its own libconsensus or use the "official" one directly
> instead.

We use a fork of libsecp256k1 and would probably do the same with the
consensus library if it was cleanly isolated.

>> In any case I agree with your stated need for this isolation (if not the
>> means) for the reasons you state. The community needs to move beyond a
>> largely singular and monolithic codebase that is holding that position
>> in part due to fear about consensus bug forks.
> 
> I completely agree. That's the goal of libconsensus (and an
> alternative implementation like libbitcoin being able to use it
> without sacrificing any of its current or future design differences
> from Bitcoin Core would be a sign of success in this reward).

It's a performance sacrifice, and then there's the OpenSSL dependency,
but these are both optional within our stack - so the application
developer has the option. So the only downside is that we are
maintaining the conditional compilation.

> Unfortunately any changes that touch consensus code are risky and
> therefore slow. And when consensus encapsulation changes conflict with
> other changes (not because the other changes need to change consensus
> but because consensus code is still coupled with policy and other
> bitcoind-specific code), refactors are never prioritized. Ironically,
> you need to encapsulate the consensus code to avoid such conflicts,
> which would make all non-consensus changes far less risky (reducing
> the consensus-critical review development bottleneck).
> 
> Unfortunately and ironically again, safer, small and incremental
> changes are less interesting for reviewers.
> For example, I've been trying to move consensus code to the consensus
> folder for a long time. The correctness of a MOVEONLY change is
> trivial to review for anyone who knows how to copy/paste in its
> favorite editor and how to use git diff, but will I ever get answers
> to my questions in [1]?

I think it's worthwhile work, especially if you are passionate about the
longer term objectives. I haven't been involved in these reviews as I
spend very little time with the satoshi client

> I know there's many people who really care about this, Cory Fields,
> Wladimir and Pieter Wuille to name a few have reviewed many of this
> changes (I've just got used to publicly whine about lack of review on
> this front and policy encapsulation [very related fronts] as an
> attempt to get some attention: not always, but begging for review
> actually works some times).

Well a cynic might observe that fear of consensus bugs is what keeps
people on the satoshi client, and therefore accelerating the development
of a clean and independent consensus library would be a very low priority.

> Another unfortunate fact is that although a script-only libconsensus
> allows you to avoid a big part of all possible consensus fork bugs,
> there cannot be users of a finished libconsensus to ask things to until
> a finished libconsensus actually exists.

Software is never finished, but this exists and we are using it.

> At the same time, the future
> users (alternative implementations, since bitcoin core is already
> "using libconsensus") 

It is using the same source files, but AFAICT not the library.

> are the most relevant people to listen when it
> comes to the C API. That's why I beg you to comment on [2], even if
> #5995 is currently closed. Your input on [1] would be very appreciated
> as well (maybe you think it's better to expose verifyTx before
> exposing verifyHeader, even if exposing verifyHeader is something that
> could be done faster).

I haven't looked at any of these commits, but I'll make some time to at
least give a cursory review.

>  > To make choice regarding consensus an actual choice (and thereby actual
>> consensus) the modularity you suggest is essential. One must be able to
>> take new developments without having to take consensus changes. The
>> option to fork the codebase is not reasonable for most people. At this
>> point there is no defensible reason for coupling consensus checks with
>> other features.
> 
> Would you agree that asking people to fork an independent libconsensus
> project instead of having to fork the full Bitcoin-qt is much more
> reasonable?

Yes, of course. We've already done it. For each release of the satoshi
client since we made libbitcoin-consensus I've copied the sources. It's
pretty much automated and easy to visually verify that the sources
match. That would be quite a bit more difficult if there wasn't an
independent build.

> I mean, I agree with your points. If "the specification of the
> consensus rules is an implementation", then that implementation
> shouldn't be coupled with a bunch of policy and non-consensus
> technical choices (storage, dependencies, p2p protocol...). But I
> still think that "the specification of the consensus rules should be a
> concrete implementation" rather than based purely on a natural
> language like English.

Useful specifications often have two reference implementations. It's the
idea that there can be only one legitimate implementation that's
problematic.

> I believe that's the only point where we fundamentally disagree, but
> it shouldn't be a barrier in our common goal of taking "power" away
> from Bitcoin Core development. If we're successful Bitcoin Core won't
> have any privileged position with regards to, say, libbitcoin when it
> comes to deciding consensus rules changes.

I don't think we disagree on anything fundamental. My issues with the
library were (1) the lack of isolation, (2) the fact that the satoshi
client wouldn't actually use the library, and (3) backtracking to use
OpenSSL, which we had recently removed from libbitcoin.

..

> 1) libconsensus us finished and used by libbitcoin
> 2) Bitcoin Core was unanimously in favor of Gavin's 32 GB initial
> proposal and the changes are applied to bitcoin/bitcoin and
> bitcoin/libconsensus (or Bitcoin Core has a dictator like Mike
> wants[4] and he accepts it, it doesn't really matter for this
> example).
> 
> But let's also assume that X% of the users and 10% of the miners are
> against that Schism hardfork, and they don't want to be forced to
> change the rules by any influential group, mining, economic or user
> majority.
> Libbitcoin cannot be forced to accept the next, controversial version
> of bitcoin/libconsensus, so you guys fork libbitcoin/libconsensus out
> of the last ok version.

This is already done.

> Centralized-bitcoin and old-bitcoin would become 2 separated
> currencies and some people would likely lose money in the transition
> from one currency to 2 of them, but the users of old-bitcoin have the
> right of keeping the rules they signed up for and the only responsible
> people for this likely-catastrophic schism would be the Bitcoin core
> developers for trying to impose consensus changes into others against
> their will. Trying to impose consensus changes against the will of
> some users is wrong, and it is irrelevant if that happens in Bitcoin
> Core or Bitcoin Tx (if it is uncontroversial, it's also irrelevant
> where it gets implemented first).

I don't disagree, but you previously argued that *everyone* had to agree
on consensus. Above you are making the argument that people can choose
to disagree. Yes, this is important. Yet it's unrealistic for any
alternative consensus to overcome inertia unless there are widely
deployed independent implementations.

> I really believe bitcoin needs competitive alternative implementations
> and I believe libconsensus is a tool to help that happen and reduce
> the "gatekeeping" friction that there's (unfortunately) around Bitcoin
> Core. I look forward to any potential collaboration on this front.
> Even if you still want to maintain a reimplementation of libconsensus
> (which I humbly think it's a mistake, but I don't think there's any
> point on keep discussing that, since we know we disagree) we can
> collaborate on the future common API of a complete libconsensus (with
> verifyBlock and all). I really hope we can do that.

I'm maintain our fork of the consensus sources until they are (1)
properly isolated from the satoshi client and (2) the satoshi client
uses the actual library.

I don't see it as important or even productive to try and wedge the
consensus library into the repository/build for bitcoind and Bitcoin-QT.
It's straightforward to maintain the consensus library independently.
Always willing to work with you on it, although we're all busy, and this
isn't my top priority presently.

e


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Wladimir J. van der Laan]

On Thu, Jul 23, 2015 at 04:30:06PM +0200, Jorge Timón via bitcoin-dev wrote:

> I think there were some misunderstandings in our previous conversation
> about this topic.
> I completely agree with having a separated repository for libconsensus
> (that's the whole point, alternative implementations can be
> consensus-safe by using it, and in the event of a schism fork[1], they
> can fork just that smaller project without having to relay on Bitcoin
> Core [satoshi] at all).

Indeed.

> But I thought you also wanted Bitcoin Core to use libconsensus instead
> of just having a subtree/subrepository like it currently does with
> libsecp256k1.
> I'm not sure if that would ever be accepted, but in any case we're
> certainly far away from that goal. Here are some things that need to
> happen first:

I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such.

Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk.

If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it.

> 2) Finish libconsensus's API: expose more things than VerifyScript, at
> the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
> Feedback from alternative implementations like libbitcoin is extremely
> valuable here. Some related closed-for-now PRs:

Agreed.

> 3) Move libconsensus to a separate repository as a
> subtree/subrepository of Bitcoin Core.

If the rest is done, this is the easy part :)

> Unfortunately and ironically again, safer, small and incremental
> changes are less interesting for reviewers.
> For example, I've been trying to move consensus code to the consensus
> folder for a long time. The correctness of a MOVEONLY change is
> trivial to review for anyone who knows how to copy/paste in its
> favorite editor and how to use git diff, but will I ever get answers
> to my questions in [1]?

Code review capacity is still our greatest bottleneck.
And I don't see any way out of that, unfortunately.

> I know there's many people who really care about this, Cory Fields,
> Wladimir and Pieter Wuille to name a few have reviewed many of this
> changes (I've just got used to publicly whine about lack of review on
> this front and policy encapsulation [very related fronts] as an
> attempt to get some attention: not always, but begging for review
> actually works some times).

I do really care about this.

Wladimir

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Wladimir J. van der Laan]

On Mon, Jul 27, 2015 at 11:40:42PM -0700, Eric Voskuil via bitcoin-dev wrote:

> It's a performance sacrifice, and then there's the OpenSSL dependency,
> but these are both optional within our stack - so the application
> developer has the option. So the only downside is that we are
> maintaining the conditional compilation.

Now that BIP66 became active, and only strict DER signatures are allowed, the OpenSSL dependency can be removed from consensus.

Pieter Wuille will do an announcement on this soon.

Wladimir

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

Ok, I'm going to separate terms: current-libconsensus from theoretical
future-libconsensus (implementing ALL consensus rules).

On Tue, Jul 28, 2015 at 8:40 AM, Eric Voskuil <eric@voskuil.org> wrote:
> libsecp256k1 has it's own repository, libbitcoinconsensus doesn't.  A
> separate repository was what I considered as a requirement for us to use it.

We want to complete future-libconsensus (decouple all the consensus
rules from the rest of the bitcoin core code) first.
Then we can move future-libconsensus to a subrepository/subtree like
libsecp256k1 and I believe everybody wants this to eventually happen.
Separating current-libconsensus now may make completing
future-libconsensus harder.

>> I'm not sure if that would ever be accepted, but in any case we're
>> certainly far away from that goal.
>
> If it's not certain whether this would even be accepted, the commitment
> to a community consensus library is too weak to take a strong dependency
> on. But for us it's moot, as we have made the already accomplished that
> goal.

What I mean is that once it is separated to a subtree, there's one more step:

Make Bitcoin Core use future-libconsensus' API instead of a subtree.
Decoupling future-libconsensus from Bitcoin Core is one thing, and
Decoupling Bitcoin Core from future-libconsensus is another thing: you
need to decouple Bitcoin Core from all future-libconsensus
implementation internals. For example, script/sign (part of Bitcoin
Core) depends on individual non-API-exposed classes in
current-libconsensus.
Moving from a subtree to a completely separated library is what I
don't know will ever happen, but I don't think this is "unfairly
advantageous" for Bitcoin Core or anything like that: other
implementations can also use future-libcosensus as a subtree instead
of the C API as well.

You have accomplished the goal of separating curren-libconsensus, not
future-libconsensus.
In fact, if you complete the equivalent of future-libconsensus in
libbitcoin and separate that, maybe that's a better place to start
drafting a full API.

>> 2) Finish libconsensus's API: expose more things than VerifyScript, at
>> the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
>> Feedback from alternative implementations like libbitcoin is extremely
>> valuable here. Some related closed-for-now PRs:
>
> In our earlier discussion I believe you said that the library would not
> be undergoing significant change or feature creep. If this is the very
> least that's projected it would seem that constraint will not hold.

future-libconsensus will not have significant changes *once it is
completed*. Currently future-libconsensus is spread around many places
inclusing src/main, so that obviously needs to change before it can be
separated to an independent repo.

>> 3) Move libconsensus to a separate repository as a
>> subtree/subrepository of Bitcoin Core.
>>
>> Only after all that we can discuss whether Bitcoin Core itself should
>> include libconsensus' code or just use its API directly.
>
> I don't think it's a question of whether it *should* use its own library
> as it is published for others - this is a practically self-evident
> conclusion.

Well, Bitcoin Core is "currently the only user of future-libconsensus"
since bitcoin core and future-libconsensus are currently mutually
coupled.
Bitcoin Core will always keep using future-libconsensus. The only
question is whether it will use it through the C API or as a
subtree/subrepository (both options are also available to other
implementations). I don't know if decoupling Bitcoin Core from
future-libconsensus' implementation details enough to be able to
directly use the API is worth it or if anyone will be interested in
doing so. For me this last step is not all that interesting: if we
have an independent repo with a full API that other implementations
can use, I don't really care about Bitcoin Core not going through the
API and using including all the code directly instead.

>> I hope that after all this, libbitcoin also reconsiders whether to
>> reimplement its own libconsensus or use the "official" one directly
>> instead.
>
> We use a fork of libsecp256k1 and would probably do the same with the
> consensus library if it was cleanly isolated.

Great.

>>> In any case I agree with your stated need for this isolation (if not the
>>> means) for the reasons you state. The community needs to move beyond a
>>> largely singular and monolithic codebase that is holding that position
>>> in part due to fear about consensus bug forks.
>>
>> I completely agree. That's the goal of libconsensus (and an
>> alternative implementation like libbitcoin being able to use it
>> without sacrificing any of its current or future design differences
>> from Bitcoin Core would be a sign of success in this reward).
>
> It's a performance sacrifice, and then there's the OpenSSL dependency,
> but these are both optional within our stack - so the application
> developer has the option. So the only downside is that we are
> maintaining the conditional compilation.

As I told you before the reason why current-libconsensus is using
OpenSSL instead of libsecp256k1 is that the very authors of
libsecp256k1 warned that using libsecp256k1 for validation was
consensus risky. As Wladimir said, Pieter Wuille will make an
announcement about this soon.
In any case, as I told you in previous conversations, the plan is to
move from OpenSSL to libsecp256k1 for validation too (so libconsensus
wil drop the OpenSSL dependency and this is just a temporary concern).

> I think it's worthwhile work, especially if you are passionate about the
> longer term objectives. I haven't been involved in these reviews as I
> spend very little time with the satoshi client

Well, the questions about the API are just in english, no need to
deeply know Bitcoin Core's (satoshi client) internals.
But maybe we should have an independent mailing list for
consensus-only things. Not all future-libconsensus users will be
interested in Bitcoin Core-specific discussions, and making them
subscribe and filter seems like an unnecessary burden to
participation.

> Well a cynic might observe that fear of consensus bugs is what keeps
> people on the satoshi client, and therefore accelerating the development
> of a clean and independent consensus library would be a very low priority.

I think that's precisely what makes it a high priority in the eyes of
all the people working on it or reviewing related changes.
But, yes, I guess "evil-thinking", maybe that's what make it a low
priority for someone evil that wants Bitcoin Core's implementation
have more importance than it shold forever. I prefer not to evil-think
and just attribute it to having other priorities or just apathy about
it.

>> Another unfortunate fact is that although a script-only libconsensus
>> allows you to avoid a big part of all possible consensus fork bugs,
>> there cannot be users of a finished libconsensus to ask things to until
>> a finished libconsensus actually exists.
>
> Software is never finished, but this exists and we are using it.

By "finished" I mean a future-libconsensus that implements ALL
consensus rules. We don't have that yet.

>> At the same time, the future
>> users (alternative implementations, since bitcoin core is already
>> "using libconsensus")
>
> It is using the same source files, but AFAICT not the library.

Bitcoin Core is the ONLY "user" of future-libconsensus (which actually
only exists inside Bitcoin Core and it's not exposed).
Current-libconsensus is used by Bitcoin Core and also exposed as an
independent build (not a separated repository yet).

Once future-Bitcoin's API is completed and the code in a different
repo, how is Bitcoin Core using the API instead of the sources
directly of any importance to other implementations?
That's really the part that I cannot understand. It will be a problem
Bitcoin Core, but if other implementations want to have (and maybe
solve later) the same problem they can use a subtree too and start
coupling their code with implementations details from
future-libconsensus.
Why would they want to do that? Again, I have no idea. I don't
understand what the complain is here.

>> are the most relevant people to listen when it
>> comes to the C API. That's why I beg you to comment on [2], even if
>> #5995 is currently closed. Your input on [1] would be very appreciated
>> as well (maybe you think it's better to expose verifyTx before
>> exposing verifyHeader, even if exposing verifyHeader is something that
>> could be done faster).
>
> I haven't looked at any of these commits, but I'll make some time to at
> least give a cursory review.

Great. I mean, I wasn't asking about reviewing the commits themselves
(which is also great if you do), but rather on answering the questions
I'm making there, ie: what to expose next (ie VerifyTx or
VerifyHeader)? would this be an acceptable way to expose VerifyHeader
? Which of he step-checks functions is worth exposing too (Bitcoin
Core is currently using some to prevent DoS attacks, for example)?

>> Would you agree that asking people to fork an independent libconsensus
>> project instead of having to fork the full Bitcoin-qt is much more
>> reasonable?
>
> Yes, of course. We've already done it. For each release of the satoshi
> client since we made libbitcoin-consensus I've copied the sources. It's
> pretty much automated and easy to visually verify that the sources
> match. That would be quite a bit more difficult if there wasn't an
> independent build.

Well, neither libbitcoincosnensus nor libbitcoin-consensus implements
all the consensus rules.
That's what makes them different from future-libconsensus.
But great, we're confirming more views that we share.

> Useful specifications often have two reference implementations. It's the
> idea that there can be only one legitimate implementation that's
> problematic.

Well, this is where I fear we will never agree. I think "Bitcoin is
different" in this reward and you disagree.
Maybe Pieter's explanation is more convincing to you:
https://youtu.be/PxW5D9xCIsc?t=769
Otherwise, I think I'll stop trying convincing you.
But one last try:

If there's 2 "specification implementations":

1) widely deployed but containing 1 bug
2) not deployed anywhere but more readable and without the bug.

When the bug is found, is it a consensus rule or not?
If it is, it turns out the second implementation wasn't an specification at all.
If it's not, nobody has been ever following consensus rules!!

>> I believe that's the only point where we fundamentally disagree, but
>> it shouldn't be a barrier in our common goal of taking "power" away
>> from Bitcoin Core development. If we're successful Bitcoin Core won't
>> have any privileged position with regards to, say, libbitcoin when it
>> comes to deciding consensus rules changes.
>
> I don't think we disagree on anything fundamental. My issues with the
> library were (1) the lack of isolation, (2) the fact that the satoshi
> client wouldn't actually use the library, and (3) backtracking to use
> OpenSSL, which we had recently removed from libbitcoin.

1) Working on it
2) The Satohi client has been using all along and it will use it
forever (maybe not through the API, but I don't get what the problem
with that is).
3) There will be an announce about this soon.

>> Libbitcoin cannot be forced to accept the next, controversial version
>> of bitcoin/libconsensus, so you guys fork libbitcoin/libconsensus out
>> of the last ok version.
>
> This is already done.

Once future-libconsensus exists as a separated repository, I don't
think you want to preemptively fork it unless you're actually changing
something.

> I don't disagree, but you previously argued that *everyone* had to agree
> on consensus. Above you are making the argument that people can choose
> to disagree. Yes, this is important. Yet it's unrealistic for any
> alternative consensus to overcome inertia unless there are widely
> deployed independent implementations.

Ideally schism consensus-rules changes will never happen, since they
effectively divide the currency in 2 and force the users to chose
(apart from other complications).
But if someone is trying to impose a schism fork (from Bitcoin Core,
from the new future-lbiconsensus repo or from anywhere else) it
becomes trivial to protect your implementation against forced changes.
Ideally all consensus changes will be uncontroversial or not happen at all.
But I think about non-ideal cases too.

> I'm maintain our fork of the consensus sources until they are (1)
> properly isolated from the satoshi client and (2) the satoshi client
> uses the actual library.

1) Reasonable.
2) You mean use it through the API? Seriously, why do you care?

> I don't see it as important or even productive to try and wedge the
> consensus library into the repository/build for bitcoind and Bitcoin-QT.
> It's straightforward to maintain the consensus library independently.

To be clear, Bitcoin Core's using future-libconsensus through a
subtree instead of the API is not a preference or a goal: it is just
how things will be just after completing and separating
future-libconsensus. Making it use the API instead of the subtree will
be additional work. I'm not sure I will want/have time to do it and I
don't know of anyone planning to do that (which seems very reasonably
given that a separated future-libconsensus is a dependency for such a
change).

> Always willing to work with you on it, although we're all busy, and this
> isn't my top priority presently.

Is it because "fear of consensus bugs is what keeps people on the
satoshi client" and you want to keep things this way?
Sorry, just joking about your previous "cynic" comment. See? Don't
attribute to malice what can be attributed to lack of time.

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Tue, Jul 28, 2015 at 10:43 AM, Wladimir J. van der Laan
<laanwj@gmail.com> wrote:
> On Thu, Jul 23, 2015 at 04:30:06PM +0200, Jorge Timón via bitcoin-dev wrote:
>> But I thought you also wanted Bitcoin Core to use libconsensus instead
>> of just having a subtree/subrepository like it currently does with
>> libsecp256k1.
>> I'm not sure if that would ever be accepted, but in any case we're
>> certainly far away from that goal. Here are some things that need to
>> happen first:
>
> I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such.

As explained to Eric, it's not that I don't want Bitcoin Core to use
future-libconsensu through the API instead of a subtree: it's just
that that's more long-term and more work. And I don't see why other
implementations should really care about it.

> Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk.
>
> If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it.

Well, pure movements will not be enough, parameters will have to
change, incompatible dependencies have to be removed (ie util.h which
contains globals), etc.
But yes, I think we can do it with only low-risk and easy-to-review commits.

>> 3) Move libconsensus to a separate repository as a
>> subtree/subrepository of Bitcoin Core.
>
> If the rest is done, this is the easy part :)

And still, this doesn't require Bitcoin Core to use the API, a subtree
is enough at first.
This "easy step" doesn't guarantee that Bitcoin Core is using
future-libconsensus' API.

> Code review capacity is still our greatest bottleneck.
> And I don't see any way out of that, unfortunately.

I really think these code separations help with this (ie there are
many more people in the world with enough knowledge to review the qt
or even policy parts than there's people able to review consensus
changes).

> I do really care about this.

I know and I said so.

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 6156 bytes --]

On 07/28/2015 02:58 AM, Jorge Timón wrote:
>> I haven't looked at any of these commits, but I'll make some time to at
>> least give a cursory review.
> 
> Great. I mean, I wasn't asking about reviewing the commits themselves
> (which is also great if you do), but rather on answering the questions
> I'm making there, ie: what to expose next (ie VerifyTx or
> VerifyHeader)?

Oh, I misunderstood your ask then. I don't have a preference on
prioritizing VerifyTx vs VerifyHeader.

> would this be an acceptable way to expose
> VerifyHeader?

I'm not sure how you mean to expose it, could you clarify?

> Which of he step-checks functions is worth exposing too (Bitcoin
> Core is currently using some to prevent DoS attacks, for example)?

I don't see any reason to expose checkpoints in this library. They are
trivial to implement and are not part of consensus.

>>> Would you agree that asking people to fork an independent libconsensus
>>> project instead of having to fork the full Bitcoin-qt is much more
>>> reasonable?
>>
>> Yes, of course. We've already done it. For each release of the satoshi
>> client since we made libbitcoin-consensus I've copied the sources. It's
>> pretty much automated and easy to visually verify that the sources
>> match. That would be quite a bit more difficult if there wasn't an
>> independent build.
> 
> Well, neither libbitcoincosnensus nor libbitcoin-consensus implements
> all the consensus rules.
> That's what makes them different from future-libconsensus.
> But great, we're confirming more views that we share.

Nothing can eliminate all consensus risk, not even a common full node
implementation.

>> Useful specifications often have two reference implementations. It's the
>> idea that there can be only one legitimate implementation that's
>> problematic.
> 
> Well, this is where I fear we will never agree. I think "Bitcoin is
> different" in this reward and you disagree.
> Maybe Pieter's explanation is more convincing to you:
> https://youtu.be/PxW5D9xCIsc?t=769
> Otherwise, I think I'll stop trying convincing you.

Maybe I wasn't sufficiently explicit. It is problematic. That is the
core issue we are dealing with. That doesn't mean I disagree with the
objectives of an independent community consensus library.

The premise of the "one true library" idea is that there is *no way* to
sufficiently test for consensus bugs in any software release. That of
course means that each release of the satoshi client poses a significant
risk to the network. This risk is presently greater than that posed by
other implementations simply because of adoption. That is the basis of
the red herring argument:

https://blog.conformal.com/the-bitcoin-consensus-red-herring

The bottom line is that nobody has control over this process. There are,
and will always be, a multitude of consensus implementations that intend
to target the same coin. Presently there are multiple versions of the
satoshi client, and this has produced forks, and will continue to do so.

Isolating the satoshi consensus checks to an independent library serves
not to eliminate that risk, but can reduce it somewhat. Importantly it
will allow various implementations to overcome a perception problem,
which will improve implementation diversity and developer participation.

>>> I believe that's the only point where we fundamentally disagree, but
>>> it shouldn't be a barrier in our common goal of taking "power" away
>>> from Bitcoin Core development. If we're successful Bitcoin Core won't
>>> have any privileged position with regards to, say, libbitcoin when it
>>> comes to deciding consensus rules changes.
>>
>> I don't think we disagree on anything fundamental. My issues with the
>> library were (1) the lack of isolation, (2) the fact that the satoshi
>> client wouldn't actually use the library, and (3) backtracking to use
>> OpenSSL, which we had recently removed from libbitcoin.
> 
> 1) Working on it

For the sake of clarity, this is now a non-issue for us.

> 2) The Satoshi client has been using all along and it will use it
> forever (maybe not through the API, but I don't get what the problem
> with that is).

Again, I consider this a requirement for us to link directly to it as a
library. If the sources are isolated into an independent repo, but the
satoshi client is embedding its own copies, one must continue to diff
the client sources against the library sources. We are doing this
already, so the benefit to having the independent repo is in no longer
having to do this.

There are also differences in the build system that can affect outcome.
Comparing those differences across repos can be more challenging. For
this reason I consider it important to your objective that the satoshi
client actually use the library - as I assume it will at some point.

If the satoshi client folks are to maintain a consensus library for the
community it's also important to show a commitment to its independence.
Dogfooding is of course a software engineering best practice. But there
is also the cynical perspective - the independent library in some ways
works against an advantage of the satoshi client.

I personally don't think the committers are parochial enough to let this
become an issue. We are all after something bigger. But if there was
push-back against using the library it would be a red flag. So until
that point passes I would just maintain our independent library, cloning
the sources from the satoshi client.

> 3) There will be an announce about this soon.

Yes, I've seen this as a temporary setback.

...
>> Always willing to work with you on it, although we're all busy, and this
>> isn't my top priority presently.
> 
> Is it because "fear of consensus bugs is what keeps people on the
> satoshi client" and you want to keep things this way?

No, I see it as less significant to the adoption of libbitcoin-server
than other issues we are working on, especially given the existence of
libbitcoin-consensus. I also trust you will make progress regardless.

e




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Wed, Jul 29, 2015 at 10:38 PM, Eric Voskuil <eric@voskuil.org> wrote:
> On 07/28/2015 02:58 AM, Jorge Timón wrote:
> Oh, I misunderstood your ask then. I don't have a preference on
> prioritizing VerifyTx vs VerifyHeader.

Ok, let's assume we want to expose verifyHeader first (which I think
will be easier).

>> would this be an acceptable way to expose
>> VerifyHeader?
>
> I'm not sure how you mean to expose it, could you clarify?

In https://github.com/bitcoin/bitcoin/pull/5995 I had one (probably
stupid) proposal.
But it had so many preparations commits that I had to close it.
In the last commit
https://github.com/jtimon/bitcoin/commit/00b9b227afc8669a877984561329dde75d3d8942
you can see that I'm adding a new function in
script/bitcoinconsensus.cpp with the following declaration:

int bitcoinconsensus_verify_header(const unsigned char* blockHeader,
unsigned int blockHeaderLen,
 const Consensus::Params& params, int64_t nTime, CBlockIndexBase*
pindexPrev, PrevIndexGetter indexGetter,
 bitcoinconsensus_error* err)

The ugly parts that you may not like are the CBlockIndexBase struct
(or maybe it's not so unreasonable) and the function pointer
PrevIndexGetter.
To see their "ugliness" you can look at:

https://github.com/jtimon/bitcoin/commit/4528ec69617f1b6d6c8f0d73dc4091cded7c216c

The PrevIndexGetter function pointer that Bitcoin Core would use
internally would be:

const CBlockIndexBase* GetPrevIndex(const CBlockIndexBase* pindex)
{
    return ((CBlockIndex*)pindex)->pprev;
}

with an ugly casting. But, well, I guess that's only ugly for Bitcoin
Core, not necessarily for other libconsensus users, which can define
their own function pointer, provided that it's of the form:

typedef const CBlockIndexBase* (*PrevIndexGetter)(const CBlockIndexBase*);

The struct that I think needs more refinement (and I just used what I
considered easier to implement at the time) is the CBlockIndexBase
struct itself:

+struct CBlockIndexBase
+{
+ //! pointer to the hash of the block, if any. Memory is owned by
this CBlockIndexBase
+ const uint256* phashBlock;
+ //! block header
+ int32_t nVersion;
+ uint256 hashMerkleRoot;
+ uint32_t nTime;
+ uint32_t nBits;
+ uint32_t nNonce;
+ //! height of the entry in the chain. The genesis block has height 0
+ int nHeight;
+};

I don't like phashBlock being a pointer instead of just a ref or even an object
Should that struct have a CBlockIndexBase* pprev; field (moving it
down from CBlockIndex)?
That's the kind of question where your feedback seems very important
from other-implementations developers (because you won't necessarily
take into account the difficulty of the refactors required in Bitcoin
Core to expose the right interface, and "libconsensus shouldn't care"
either, all we want is the best interface).

>> Which of he step-checks functions is worth exposing too (Bitcoin
>> Core is currently using some to prevent DoS attacks, for example)?
>
> I don't see any reason to expose checkpoints in this library. They are
> trivial to implement and are not part of consensus.

Agreed, and I would say all of the checkpoint check separation has
been done already.
What I mean by step functions is...look at verfyHeader internals, for example:

https://github.com/jtimon/bitcoin/commit/11ede96f59f611ede596a1335e896b1fef4fb5b2

It internally calls Consensus::CheckBlockHeader (quite cheap with no
context required) and Consensus::ContextualCheckBlockHeader (not so
cheap).
Bitcoin Core never calls (yet) the full verifyHeader at once. It does
the cheap tests first and the expensive later. For example,

call CheckBlockHeader, then CheckBlock (also cheap), then
ContextualCheckBlockHeader and then ContextualCheckBlock.

The question is, will other implementations want access to these
not-full-but-cheap tests?
In other words, apart from exposing VerifyHeader that fully validates
all consensus rules for a header, do we also want to expose
CheckBlockHeader and ContextualCheckBlockHeader to give more
flexibility to libconsensus' users?

I think, yes, other implementations will want this for the same DoS
reasons that Bitcoin Core currently wants them. But it would be nice
to know what a second person thinks about this.

> Nothing can eliminate all consensus risk, not even a common full node
> implementation.

In fact, one thing does: never changing the code again (but the cure
would be worse than the illness).
Agreed, any software changes in the consensus code can cause consensus
forks (and that's why you don't want to touch libconsensus that much
once it's separated).


>>> Useful specifications often have two reference implementations. It's the
>>> idea that there can be only one legitimate implementation that's
>>> problematic.
>>
>> Well, this is where I fear we will never agree. I think "Bitcoin is
>> different" in this reward and you disagree.
>> Maybe Pieter's explanation is more convincing to you:
>> https://youtu.be/PxW5D9xCIsc?t=769
>> Otherwise, I think I'll stop trying convincing you.
>
> Maybe I wasn't sufficiently explicit. It is problematic. That is the
> core issue we are dealing with. That doesn't mean I disagree with the
> objectives of an independent community consensus library.
>
> The premise of the "one true library" idea is that there is *no way* to
> sufficiently test for consensus bugs in any software release. That of
> course means that each release of the satoshi client poses a significant
> risk to the network. This risk is presently greater than that posed by
> other implementations simply because of adoption. That is the basis of
> the red herring argument:

Well, the "one true library" will be much better than the current "one
true full node".
The "one true library" would be the specification of the consensus
rules, but that doesn't mean you can't fork and modify it however you
want.

> The bottom line is that nobody has control over this process. There are,
> and will always be, a multitude of consensus implementations that intend
> to target the same coin. Presently there are multiple versions of the
> satoshi client, and this has produced forks, and will continue to do so.

I get this point, even if the current satoshi client contains the
consensus rules specification (and many other things, obviously), that
doesn't mean is somehow protected from forking with itself if the
consensus code is changed in the wrong way accidentally. But the more
separated libconsensus and Bitcoin Core (satoshi client) are, the less
likely that changes in Bitcoin Core that weren't supposed to change
consensus rules actually do it by accident (like last time with the
migration out of bdb).

> Isolating the satoshi consensus checks to an independent library serves
> not to eliminate that risk, but can reduce it somewhat. Importantly it
> will allow various implementations to overcome a perception problem,
> which will improve implementation diversity and developer participation.

I think alternative implementations using a full libconsensus can
increase their adoption a lot, since they become just as vulnerable to
consensus forks as Bitcoin Core (instead of more vulnerable like now).

>>>> I believe that's the only point where we fundamentally disagree, but
>>>> it shouldn't be a barrier in our common goal of taking "power" away
>>>> from Bitcoin Core development. If we're successful Bitcoin Core won't
>>>> have any privileged position with regards to, say, libbitcoin when it
>>>> comes to deciding consensus rules changes.
>>>
>>> I don't think we disagree on anything fundamental. My issues with the
>>> library were (1) the lack of isolation, (2) the fact that the satoshi
>>> client wouldn't actually use the library, and (3) backtracking to use
>>> OpenSSL, which we had recently removed from libbitcoin.
>>
>> 1) Working on it
>
> For the sake of clarity, this is now a non-issue for us.

You mean libbitcoin's code is better organized than Bitcoin Core's?
I don't doubt it. Maybe we can create a full-libbitcoin-libconsensus
first and work on the API there.

>> 2) The Satoshi client has been using all along and it will use it
>> forever (maybe not through the API, but I don't get what the problem
>> with that is).
>
> Again, I consider this a requirement for us to link directly to it as a
> library. If the sources are isolated into an independent repo, but the
> satoshi client is embedding its own copies, one must continue to diff
> the client sources against the library sources. We are doing this
> already, so the benefit to having the independent repo is in no longer
> having to do this.

Oh, I see, you don't like that libsecp256k1 is currently a subtree of
Bitcoin Core either for the same reasons, right?
To not need to know when the changes in libconsensus are applied in
Bitcoin Core.
Mhmm, once libconsensus is complete, why would you care about it?
You just care about the libconsensus version (which doesn't have to
coincide with Bitcoin Core versions anymore).

> There are also differences in the build system that can affect outcome.
> Comparing those differences across repos can be more challenging. For
> this reason I consider it important to your objective that the satoshi
> client actually use the library - as I assume it will at some point.

For the sake of clarity, please say "use the library's API". It's
going to use the library one way or another.

> If the satoshi client folks are to maintain a consensus library for the
> community it's also important to show a commitment to its independence.
> Dogfooding is of course a software engineering best practice. But there
> is also the cynical perspective - the independent library in some ways
> works against an advantage of the satoshi client.
>
> I personally don't think the committers are parochial enough to let this
> become an issue. We are all after something bigger. But if there was
> push-back against using the library it would be a red flag. So until
> that point passes I would just maintain our independent library, cloning
> the sources from the satoshi client.

To be clear, I don't oppose to "dogfooding", it's just clear to me
that it will take even longer.
So what I don't understand is "once libbitcoin is complete and ready
for us to use, we will keep using our reimplementation of consensus
until Bitcoin Core uses the API as well. If Bitcoin core doesn't use
the API, we prefer not to use the library at all and keep having the
same consensus risk. We will do what we think it's worse for us until
Bitcoin Core uses the library through the API".

>> 3) There will be an announce about this soon.
>
> Yes, I've seen this as a temporary setback.

And we will hopefully migrate the current libconsensus from openSSL to
libsecp256k1 soon. So we will be able to enjoy libsecp256k1's
performance improvements without risking consensus. One problem less.

>>> Always willing to work with you on it, although we're all busy, and this
>>> isn't my top priority presently.
>>
>> Is it because "fear of consensus bugs is what keeps people on the
>> satoshi client" and you want to keep things this way?
>
> No, I see it as less significant to the adoption of libbitcoin-server
> than other issues we are working on, especially given the existence of
> libbitcoin-consensus. I also trust you will make progress regardless.

This was just a joke because you said something similar earlier.
Don't take it seriously.

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

Moving it here from the other thread.

On Thu, Aug 20, 2015 at 2:08 AM, Eric Voskuil <eric@voskuil.org> wrote:
> On 08/19/2015 04:27 PM, Jorge Timón wrote:
>> No, as previously explained, once libconsensus is complete it can be
>> moved to a separate repository like libsecp256k1.
>
> I don't see this happening any time soon, and I'm not sure why we should
> wait for it.

Yes, unfortunately I don't see this happening any time soon either, at
least not with the amount of review I'm getting.
My initial hope was to complete libconsensus by 0.12 (one year should
be enough time, right?) but I was being too optimistic.
By "wait for it" I assume you mean waiting for libconsensus to be
complete before we separate it to a different repository.
The reason is just simplicity.

>>> In our discussion leading up to libbitcoin building libbitcoin-consensus
>>> we disagreed on whether intentional hard forks would (or even could)
>>> happen. I think that issue is now settled. So my question remains how do
>>> stakeholders (users/miners) maintain consensus when it's their
>>> individual intent (the first objective of libconsensus), and diverge
>>> when intended (which a direct dependency on libconsensus makes harder)?
>>> IMO it's unreasonable to operate as if this won't happen, given that it has.
>>
>> I believe the simplest option...
>
> You might consider this as feedback from your customer base.

Mhmm, not sure I understand this point.

>> would be to fork the libconsensus
>> project and do the schism/controversial/contentious hardfork there.
>> But of course modifying libconsensus will be much easier than
>> modifying Bitcoin Core (if anything, because the amount of code is
>> much smaller).
>
> That's a false dichotomy. We never would have considered forking Bitcoin
> Core, and still wouldn't. Why would we set ourselves up for this
> disruption, which would inevitably lead to us factoring the consensus
> portions of libconsensus out of /bitcoin at the 11th hour?
>
> We have to operate as if it can happen at any time. Otherwise we have
> relinquished control of this vote and failed our users. Given that
> operating assumption, it is much safer for us to have already done this
> work (which we did). [It also provides a forcing function for us to
> review in detail any consensus changes that get pushed out.]

Yes, you need to operate as if it can happen at any time. I now
understandbetter  your position of having your own repository until a
complete libconsensus is separated.
In the meantime you will have to keep using your re-implementation of
the rest of the consensus rules (besides the script checks), but
fortunately the most risky and harder reimplementation is the part of
the script validation.

> My question is why you would not embrace an independent consensus
> repository? Your work to evolve it doesn't change.

Yes, I want a separated repository. I just wanted to start with a
separated folder first. Right now there's consensus code all over the
place, specially in main.cpp.
I think changing the order (separated repository first, moving code
from Bitcoin Core to libconsensus later) would increase the total
amount of work.

Here's another option that has recently crossed my mind:

1) Finish the libconsensus separation in an independent branch on top
of a given version, for example 0.11.
2) Separate a repository from that. Alternative implementations can
start using a full libconsensus
3) Rebase that branch on top of bitcoin/master and start to PR small
groups of commits. Once the whole branch has been merged, Bitcoin Core
can replace the consensus folder with the libconsensus subtree, so
that Bitcoin Core itself can start using a full libconsensus.

Ironically with this plan Bitcoin Core may not be the full node first
implementation to use a full libconsensus.
There will be some consensus fork bug risks during 3 (which at the
current speed I estimate it could easily take 3 or 4 years) and there
would be some redundant work (replicating every consensus change in
both Bitcoin Core and libconsensus).
On the bright side, we may be able to have a full libconsensus this
year (which was my goal after we exposed VerifyScript in the first
libconsensus).

Thoughts?

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 2510 bytes --]

Jorge,

separating script engine into libconsensus was very helpful, since wrapped the piece of consensus
that would least likely to be captured exactly with an implementation from scratch. Thank you for your
effort there.  Bits of Proof now uses its own or alternatively libconsensus for full validation.

I am sceptical however that a “full” consensus lib extracted from satoshi’s code is worth trying.
Not because it was impossible, but because the result would not be higher quality, if measured on agreement
with satoshi, than other re-implementations. It would actually be lower quality because of the antique tool set.

The rules outside script engine are simpler, therefore much easier to capture exactly. They are however
scattered around in the spaghetti and are often just a single if statement, also repeated elsewhere.

You would either have to very extensively refactor the code, that unlikely goes through as a PR, or
do what me and others did. Read satoshi code and rewrite the same. You have
a slight advantage of copy-paste small fragments, but I doubt the consensus relevant advantage of that.

Tamas Blummer

> On Aug 20, 2015, at 02:53, Jorge Timón via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> 
> 1) Finish the libconsensus separation in an independent branch on top
> of a given version, for example 0.11.
> 2) Separate a repository from that. Alternative implementations can
> start using a full libconsensus
> 3) Rebase that branch on top of bitcoin/master and start to PR small
> groups of commits. Once the whole branch has been merged, Bitcoin Core
> can replace the consensus folder with the libconsensus subtree, so
> that Bitcoin Core itself can start using a full libconsensus.
> 
> Ironically with this plan Bitcoin Core may not be the full node first
> implementation to use a full libconsensus.
> There will be some consensus fork bug risks during 3 (which at the
> current speed I estimate it could easily take 3 or 4 years) and there
> would be some redundant work (replicating every consensus change in
> both Bitcoin Core and libconsensus).
> On the bright side, we may be able to have a full libconsensus this
> year (which was my goal after we exposed VerifyScript in the first
> libconsensus).
> 
> Thoughts?
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[-- Attachment #1.2: Type: text/html, Size: 4394 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Thu, Aug 20, 2015 at 9:14 AM, Tamas Blummer <tamas@bitsofproof.com> wrote:
> Jorge,
>
> separating script engine into libconsensus was very helpful, since wrapped
> the piece of consensus
> that would least likely to be captured exactly with an implementation from
> scratch. Thank you for your
> effort there.  Bits of Proof now uses its own or alternatively libconsensus
> for full validation.

It wasn't just me: I didn't had the idea of creating a libconsensus
with a C API (thank Matt Corallo for that), I didn't removed all the
undesired dependencies or prepared the building part (thank Cory
Fields) and also thank at least Wladimir and Pieter who also
contributed in some ways I don't remember.
And of course also thank all the reviewers that made the PR merges possible.

I'm really happy to hear that libconsensus is being used, thank you
for your effort there too.

> I am sceptical however that a “full” consensus lib extracted from satoshi’s
> code is worth trying.
> Not because it was impossible, but because the result would not be higher
> quality, if measured on agreement
> with satoshi, than other re-implementations. It would actually be lower
> quality because of the antique tool set.

I don't understand what you mean by "quality" in this context. One of
the goals is to have as little dependencies as possible (so "more
modern tool sets" may not be suitable for libconsensus). libsecp256k1
will keep on being a dependency (highly optimized C code) and that's
about it.
Ideally I would like to slowly move libconsensus from C++ to C too,
but it seems other people would prefer to move to C++11 instead.

> The rules outside script engine are simpler, therefore much easier to
> capture exactly. They are however
> scattered around in the spaghetti and are often just a single if statement,
> also repeated elsewhere.

Yes, they are simpler and thus there's less risks of consensus fork
bugs, but it still exists.
It is true that the consensus code is currently spread all around
(specially in main.cpp), but completing libconsensus would solve that.
Lastly, since for consensus rules "the code is the specification", it
is unfortunate that the specification is coupled with a concrete
implementation (Bitcoin Core) and we should fix that.

> You would either have to very extensively refactor the code, that unlikely
> goes through as a PR, or
> do what me and others did. Read satoshi code and rewrite the same. You have
> a slight advantage of copy-paste small fragments, but I doubt the consensus
> relevant advantage of that.

But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
It is not only exposing it but also separating it from Bitcoin Core so
that they can be changed without having to also change/take into
account non-consensus Bitcoin Core specific things.
A single PR would certainly be unacceptable, I was making many little
more acceptable ones (some of them already merged):

* [1/9] Consensus
** MERGED or DELETED
*** MERGED Consensus: Decouple pow from chainparams #5812 [consensuspow]
*** MERGED MOVEONLY: Move constants and globals to consensus.h #5696
[consensus_policy0]
*** DELETED Refactor: Create CCoinsViewEfficient interface for
CCoinsViewCache #5747 [coins]
*** MERGED Chainparams: Refactor: Decouple IsSuperMajority from
Params() #5968 [params_consensus]
*** MERGED Remove redundant getter
CChainParams::SubsidyHalvingInterval() #5996 [params_subsidy]
*** MERGED Separate CValidationState from main #5669 [consensus]
*** DELETED Consensus: Refactor: Separate CheckFinalTx from
main::IsFinalTx #6063 [consensus_finaltx]
*** MERGED Consensus: Decouple ContextualCheckBlockHeader from
checkpoints #5975 [consensus_checkpoints]
*** MERGED Separate Consensus::CheckTxInputs and GetSpendHeight in
CheckInputs #6061 [consensus_inputs]
*** MERGED Bugfix: Don't check the genesis block header before
accepting it #6299 [5975-quick-fix]
** REBASE Chainparams: Explicit Consensus::Params arg in consensus
functions #6024 [params_consensus2]
** REBASE Optimizations: Consensus: In AcceptToMemoryPool,
ConnectBlock, and CreateNewBlock #6445 [consensus-txinputs-0.12.99]
** REBASE MOVEONLY: Move most of consensus functions (pre-block) #6051
[consensus_moveonly]
** REBASE Consensus: Refactor: Turn CBlockIndex::GetMedianTimePast
into independent function #6009 [consensus_mediantime]
** DEPENDENT Consensus: Refactor: Consensus version of
CheckBlockHeader() #6035 [consensus_checkblockheader]
** DEPENDENT Consensus: Consensus version of pow functions [consensus_pow2]
** DEPENDENT API: Expose bitcoinconsensus_verify_header() in
libconsensus #5995 [consensus_header]
** DEPENDENT API: Expose bitcoinconsensus_verify_block() in
libconsensus #5946 [consensus_tip]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Tamas Blummer]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.

> On Aug 20, 2015, at 10:06, Jorge Timón <jtimon@jtimon.cc> wrote:
> 
> 
> But the goal is not reimplementing the consensus rules but rather
> extract them from Bitcoin Core so that nobody needs to re-implement
> them again.



My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
but it is also imperative to be able to create and serve other block chains with other rules and for those
I do not want to carry on the legacy of an antique tool set and a spaghetti style.

Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
made because each piece appears best in what they do.

Tamas Blummer


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Matt Corallo]

I dont think a libconsensus would have any kind of networking layer, nor
is C++ an antique tool set (hopefully libconsensus can avoid a boost
dependency, though thats not antique either). Ideally it would have a
simple API to give it blocks and a simple API for it to inform you of
what the current chain is. If you really want to get fancy maybe it has
pluggable block storage, too, but I dont see why you couldnt use this in
~any client?

On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote:
> Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
> but also open the chance of doing the work better, in the sense of software engineering.
> 
>> On Aug 20, 2015, at 10:06, Jorge Timón <jtimon@jtimon.cc> wrote:
>>
>>
>> But the goal is not reimplementing the consensus rules but rather
>> extract them from Bitcoin Core so that nobody needs to re-implement
>> them again.
> 
> 
> 
> My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
> but it is also imperative to be able to create and serve other block chains with other rules and for those
> I do not want to carry on the legacy of an antique tool set and a spaghetti style.
> 
> Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
> and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
> made because each piece appears best in what they do.
> 
> Tamas Blummer
> 
> 
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 2679 bytes --]

I know what you mean as I already have such a component with pluggable block store and networking.
While you are at it you could aim for isolation of bitcoin specific decisions and algos from generic block chain code.

The magnitude of refactoring you would have to do to get there from main.cpp and the rest of the hairball
is harder than a re-write from scratch, and the result will not be impressive, just hopefully working.
I think a slim API server was a lower hanging fruit in Core’s case.

BTW, support for refactoring is an example where you see if your tool set is modern.

Tamas Blummer

> On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> I dont think a libconsensus would have any kind of networking layer, nor
> is C++ an antique tool set (hopefully libconsensus can avoid a boost
> dependency, though thats not antique either). Ideally it would have a
> simple API to give it blocks and a simple API for it to inform you of
> what the current chain is. If you really want to get fancy maybe it has
> pluggable block storage, too, but I dont see why you couldnt use this in
> ~any client?
> 
> On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote:
>> Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
>> but also open the chance of doing the work better, in the sense of software engineering.
>> 
>>> On Aug 20, 2015, at 10:06, Jorge Timón <jtimon@jtimon.cc> wrote:
>>> 
>>> 
>>> But the goal is not reimplementing the consensus rules but rather
>>> extract them from Bitcoin Core so that nobody needs to re-implement
>>> them again.
>> 
>> 
>> 
>> My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
>> but it is also imperative to be able to create and serve other block chains with other rules and for those
>> I do not want to carry on the legacy of an antique tool set and a spaghetti style.
>> 
>> Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
>> and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
>> made because each piece appears best in what they do.
>> 
>> Tamas Blummer
>> 
>> 
>> 
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 


[-- Attachment #1.2: Type: text/html, Size: 4723 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Matt Corallo]

On 08/20/15 21:26, Tamas Blummer wrote:
> I know what you mean as I already have such a component with pluggable
> block store and networking.

I'm not suggesting pluggable networking, I'm suggesting (and I think
everyone thinks the design should be) NO networking. The API is
ValidationResult libconsensus.HeyIFoundABlock(Block) and
ListOfBlocksToDownloadNext libconsensus.HeyIFoundAHeaderList(ListOfHeaders).

> While you are at it you could aim for isolation of bitcoin specific
> decisions and algos from generic block chain code. 

Are you suggesting to support altcoins? I dont think anyone cares about
supporting that.

> The magnitude of refactoring you would have to do to get there from
> main.cpp and the rest of the hairball
> is harder than a re-write from scratch,

I think you'd be very pleasantly surprised. It sounds like you havent
dug into Bitcoin Core validation code in years.

> and the result will not be
> impressive, just hopefully working.

Hmm? The result would be an obviously correct consensus implementation
that everyone could use, instead of everyone going off and writing their
own and either being wrong, or never updating in the case of forks. Its
a huge deal to allow people to focus on making their libraries have good
APIs/Wallets/etc instead of focusing on making a working validation
engine (though maybe for that the p2p layer needs to also be in a library).

> I think a slim API server was a lower hanging fruit in Core’s case.

We have one, it just needs a few already obvious performance improvements.

> BTW, support for refactoring is an example where you see if your tool
> set is modern.

There are a number of good development tools for C++ that allow this....

> Tamas Blummer
> 
>> On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>
>> I dont think a libconsensus would have any kind of networking layer, nor
>> is C++ an antique tool set (hopefully libconsensus can avoid a boost
>> dependency, though thats not antique either). Ideally it would have a
>> simple API to give it blocks and a simple API for it to inform you of
>> what the current chain is. If you really want to get fancy maybe it has
>> pluggable block storage, too, but I dont see why you couldnt use this in
>> ~any client?
>>
>> On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote:
>>> Every re-implementation, re-factoring even copy-paste introduces a
>>> risk of disagreement,
>>> but also open the chance of doing the work better, in the sense of
>>> software engineering.
>>>
>>>> On Aug 20, 2015, at 10:06, Jorge Timón <jtimon@jtimon.cc
>>>> <mailto:jtimon@jtimon.cc>> wrote:
>>>>
>>>>
>>>> But the goal is not reimplementing the consensus rules but rather
>>>> extract them from Bitcoin Core so that nobody needs to re-implement
>>>> them again.
>>>
>>>
>>>
>>> My goal is different. Compatibility with Bitcoin is important as I
>>> also want to deal with Bitcoins,
>>> but it is also imperative to be able to create and serve other block
>>> chains with other rules and for those
>>> I do not want to carry on the legacy of an antique tool set and a
>>> spaghetti style.
>>>
>>> Bits of Proof uses scala (akka networking), java (api service), c++
>>> (leveledb and now libconsensus)
>>> and I am eager to integrate secp256k1 (c) as soon as part of
>>> consensus. The choices were
>>> made because each piece appears best in what they do.
>>>
>>> Tamas Blummer
>>>
>>>
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
> 

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 4488 bytes --]

Thinking in Bitcoins only on the level of technology unnecessarily narrows your view.

OK, I hope to be pleasantly surprised.

Tamas Blummer

> On Aug 20, 2015, at 23:35, Matt Corallo <lf-lists@mattcorallo.com> wrote:
> 
> 
> 
> On 08/20/15 21:26, Tamas Blummer wrote:
>> I know what you mean as I already have such a component with pluggable
>> block store and networking.
> 
> I'm not suggesting pluggable networking, I'm suggesting (and I think
> everyone thinks the design should be) NO networking. The API is
> ValidationResult libconsensus.HeyIFoundABlock(Block) and
> ListOfBlocksToDownloadNext libconsensus.HeyIFoundAHeaderList(ListOfHeaders).
> 
>> While you are at it you could aim for isolation of bitcoin specific
>> decisions and algos from generic block chain code.
> 
> Are you suggesting to support altcoins? I dont think anyone cares about
> supporting that.
> 
>> The magnitude of refactoring you would have to do to get there from
>> main.cpp and the rest of the hairball
>> is harder than a re-write from scratch,
> 
> I think you'd be very pleasantly surprised. It sounds like you havent
> dug into Bitcoin Core validation code in years.
> 
>> and the result will not be
>> impressive, just hopefully working.
> 
> Hmm? The result would be an obviously correct consensus implementation
> that everyone could use, instead of everyone going off and writing their
> own and either being wrong, or never updating in the case of forks. Its
> a huge deal to allow people to focus on making their libraries have good
> APIs/Wallets/etc instead of focusing on making a working validation
> engine (though maybe for that the p2p layer needs to also be in a library).
> 
>> I think a slim API server was a lower hanging fruit in Core’s case.
> 
> We have one, it just needs a few already obvious performance improvements.
> 
>> BTW, support for refactoring is an example where you see if your tool
>> set is modern.
> 
> There are a number of good development tools for C++ that allow this....
> 
>> Tamas Blummer
>> 
>>> On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev
>>> <bitcoin-dev@lists.linuxfoundation.org
>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>> 
>>> I dont think a libconsensus would have any kind of networking layer, nor
>>> is C++ an antique tool set (hopefully libconsensus can avoid a boost
>>> dependency, though thats not antique either). Ideally it would have a
>>> simple API to give it blocks and a simple API for it to inform you of
>>> what the current chain is. If you really want to get fancy maybe it has
>>> pluggable block storage, too, but I dont see why you couldnt use this in
>>> ~any client?
>>> 
>>> On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote:
>>>> Every re-implementation, re-factoring even copy-paste introduces a
>>>> risk of disagreement,
>>>> but also open the chance of doing the work better, in the sense of
>>>> software engineering.
>>>> 
>>>>> On Aug 20, 2015, at 10:06, Jorge Timón <jtimon@jtimon.cc
>>>>> <mailto:jtimon@jtimon.cc>> wrote:
>>>>> 
>>>>> 
>>>>> But the goal is not reimplementing the consensus rules but rather
>>>>> extract them from Bitcoin Core so that nobody needs to re-implement
>>>>> them again.
>>>> 
>>>> 
>>>> 
>>>> My goal is different. Compatibility with Bitcoin is important as I
>>>> also want to deal with Bitcoins,
>>>> but it is also imperative to be able to create and serve other block
>>>> chains with other rules and for those
>>>> I do not want to carry on the legacy of an antique tool set and a
>>>> spaghetti style.
>>>> 
>>>> Bits of Proof uses scala (akka networking), java (api service), c++
>>>> (leveledb and now libconsensus)
>>>> and I am eager to integrate secp256k1 (c) as soon as part of
>>>> consensus. The choices were
>>>> made because each piece appears best in what they do.
>>>> 
>>>> Tamas Blummer
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> bitcoin-dev mailing list
>>>> bitcoin-dev@lists.linuxfoundation.org
>>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>> 
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>> 
>> 
> 


[-- Attachment #1.2: Type: text/html, Size: 7382 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com> wrote:
> Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
> but also open the chance of doing the work better, in the sense of software engineering.

But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
the specification" and the reasons to separate libconsensus:
https://youtu.be/l3O4nh79CUU?t=764

>> On Aug 20, 2015, at 10:06, Jorge Timón <jtimon@jtimon.cc> wrote:
>>
>>
>> But the goal is not reimplementing the consensus rules but rather
>> extract them from Bitcoin Core so that nobody needs to re-implement
>> them again.
>
>
>
> My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
> but it is also imperative to be able to create and serve other block chains with other rules and for those
> I do not want to carry on the legacy of an antique tool set and a spaghetti style.
>
> Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
> and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
> made because each piece appears best in what they do.

Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Eric Lombrozo]

[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]

Unfortunately we have no way of rigorously proving functional equivalence
other than code review and unit testing. The simpler the consensus code
(and the more we can write it in a style that affords provability of
correctness) the easier it will be in the future to compare implementations.

Prior to swapping out implementations, we should at the least run it
through the gauntlet and perhaps run both implementations side-by-side.

All I/O should be treated abstractly in the API.

In C++ I really like using a nearly bare-bones signal template for most
async message handling, i.e.
https://github.com/ciphrex/mSIGNA/blob/master/deps/Signals/src/Signals.h

This greatly facilitates support for async bidirectional I/O, etc...with
minimal overhead.

But others might have other stylistic preferences.

- Eric

On Fri, Aug 21, 2015, 12:46 PM Jorge Timón <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com>
> wrote:
> > Every re-implementation, re-factoring even copy-paste introduces a risk
> of disagreement,
> > but also open the chance of doing the work better, in the sense of
> software engineering.
>
> But you don't want something better, you want something functionally
> identical.
> You may want to watch sipa's explanation on why "the implementation is
> the specification" and the reasons to separate libconsensus:
> https://youtu.be/l3O4nh79CUU?t=764
>
> >> On Aug 20, 2015, at 10:06, Jorge Timón <jtimon@jtimon.cc> wrote:
> >>
> >>
> >> But the goal is not reimplementing the consensus rules but rather
> >> extract them from Bitcoin Core so that nobody needs to re-implement
> >> them again.
> >
> >
> >
> > My goal is different. Compatibility with Bitcoin is important as I also
> want to deal with Bitcoins,
> > but it is also imperative to be able to create and serve other block
> chains with other rules and for those
> > I do not want to carry on the legacy of an antique tool set and a
> spaghetti style.
> >
> > Bits of Proof uses scala (akka networking), java (api service), c++
> (leveledb and now libconsensus)
> > and I am eager to integrate secp256k1 (c) as soon as part of consensus.
> The choices were
> > made because each piece appears best in what they do.
>
> Since you already depend on libconsensus for VerifyScript, wouldn't it
> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
> You would still have complete control over storage, concurrency,
> networking, policy...
> My plan is for the C API to interface with the external storage by
> passing a function pointer to it.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 3766 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 3398 bytes --]

> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon.cc> wrote:
> 
> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com <mailto:tamas@bitsofproof.com>> wrote:
>> Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
>> but also open the chance of doing the work better, in the sense of software engineering.
> 
> But you don't want something better, you want something functionally identical.
> You may want to watch sipa's explanation on why "the implementation is
> the specification" and the reasons to separate libconsensus:
> https://youtu.be/l3O4nh79CUU?t=764 <https://youtu.be/l3O4nh79CUU?t=764>

I do want something better, but not for the focus you have.

Not because what you produce was not high quality, but because quality is achieved at a very
high cost and is hard to uphold over generations of developer. You focus on a single use case
while there are many out there for distributed ledgers.

I think in an infrastructure for enterprise applications, building consensus on the ledger is a
cornerstone there, but is only a piece of the solution. I built several commercially successful
deployments where I delegated the consensus building to a border router, a Bitcoin Core,
then interfaced that trusted peer with my  implementation that accepted Core’s decisions
in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices”
therefore an example of centralization people here try to avoid.

Enterprises have sufficient resources. Solving the business problem is valuable to them even at
magnitudes higher cost than a hobbyist would bear.

For mainstream adoption you need to get enterprises on board too, and  that is what I care of.
Enterprises want code that is not only high quality, but is easy to maintain with a development
team with high attrition. One has to take whatever help is offered for that, and one is modern
languages and runtimes.

Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially
successful deployments, because of the use of a border router, but it helped development,
enabling easier debug and precise error feedback esp. end even after Core had a reject message.

I integrated libconsensus only for the hope that is significantly fastens application side tx verification,
 which it has turned out it does not, until secp265k1 is integrated.

I would likely use an other extended libconsensus too, but do not think there was a dependency on
that for enterprise development.

It would help there more to have a slim protocol server, no wallet, no rpc, no qt but a high
performance remoting API.

> Since you already depend on libconsensus for VerifyScript, wouldn't it
> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
> You would still have complete control over storage, concurrency,
> networking, policy...
> My plan is for the C API to interface with the external storage by
> passing a function pointer to it.


Storage and validation is non-trivially interconnected, but I now the separation can be done,
since I did it.

Excuse me, but function pointers is a pattern I used in the 80’s. I know that they are behind
the curtain of modern abstractions with similar use, I still prefer not to see them again.

Tamas Blummer


[-- Attachment #1.2: Type: text/html, Size: 13258 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Eric Lombrozo]

[-- Attachment #1: Type: text/plain, Size: 6470 bytes --]

I've been pushing for greater modularization since I first got into
bitcoin. I got quickly frustrated when I was only able to get through very
few things (i.e. moving core structure serialization classes to a separate
unit not called main). Working on Bitcoin has an added layer of frustration
that goes beyond most open source projects: even though we're clearly in
userland working at the application layer, a good layered protocol design
is still lacking. We have no standards process separate from what basically
amount to updates to one specific reference implementation. And we all need
to agree on any major change, since a blockchain that is easily forked in
contentious ways pretty much defeats its own purpose.

I went off to develop my own stack, where I could more easily avoid
politics and focus on engineering. But I now understand the politics are
inevitable. Bitcoin is inherently a cooperative project. Several people
have poured themselves passionately into the reference codebase, most of
whom did it (at least initially) purely as unpaid volunteers. There's a lot
of love that's gone into this. But it's become pretty clear that the
modularization is no longer merely a matter of good engineering - it is
essential to resolving serious political challenges.

Perhaps the most frustrating thing of all is watching people pushing for
relatively superficial yet highly controversial changes while we still lack
the proper infrastructure to handle these kinds of divergences of opinion
without either stagnating or becoming polarized.

I could continue working to reimplement an entire stack from scratch, as
several others have also done - but besides the serious effort duplication
this entails, it doesn't really seem like it will ultimately be a
convergent process. It's too easy to let ego and habit dictate one's
preferences rather than rational engineering considerations.

I know that some might feel I'm just preaching to the choir, but we should
probably take a step back from implementation hackery and try to specify
some core protocol layers, focusing on interfaces. Specifically, we need a
consensus layer that doesn't try to specify networking, storage, wallets,
UI, etc. Let different people improve upon these things independently in
their own implementations. What matters is that we all converge on a common
history and state. At the same time, let's open up more competition on all
these other things that are separate from the consensus layer.

If only we were to dedicate a fraction of the effort we've put into this
whole block size circus into what's actually important...and I blame myself
as well...

On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon.cc> wrote:
>
> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com>
> wrote:
>
> Every re-implementation, re-factoring even copy-paste introduces a risk of
> disagreement,
> but also open the chance of doing the work better, in the sense of
> software engineering.
>
>
> But you don't want something better, you want something functionally
> identical.
> You may want to watch sipa's explanation on why "the implementation is
> the specification" and the reasons to separate libconsensus:
> https://youtu.be/l3O4nh79CUU?t=764
>
>
> I do want something better, but not for the focus you have.
>
> Not because what you produce was not high quality, but because quality is
> achieved at a very
> high cost and is hard to uphold over generations of developer. You focus
> on a single use case
> while there are many out there for distributed ledgers.
>
> I think in an infrastructure for enterprise applications, building
> consensus on the ledger is a
> cornerstone there, but is only a piece of the solution. I built several
> commercially successful
> deployments where I delegated the consensus building to a border router, a
> Bitcoin Core,
> then interfaced that trusted peer with my  implementation that accepted
> Core’s decisions
> in an SPV manner. One might think of this setup as wasteful and unsuitable
> for “small devices”
> therefore an example of centralization people here try to avoid.
>
> Enterprises have sufficient resources. Solving the business problem is
> valuable to them even at
> magnitudes higher cost than a hobbyist would bear.
>
> For mainstream adoption you need to get enterprises on board too, and
>  that is what I care of.
> Enterprises want code that is not only high quality, but is easy to
> maintain with a development
> team with high attrition. One has to take whatever help is offered for
> that, and one is modern
> languages and runtimes.
>
> Bits of Proof’s own implementation of the scripts was not practically
> relevant in my commercially
> successful deployments, because of the use of a border router, but it
> helped development,
> enabling easier debug and precise error feedback esp. end even after Core
> had a reject message.
>
> I integrated libconsensus only for the hope that is significantly fastens
> application side tx verification,
>  which it has turned out it does not, until secp265k1 is integrated.
>
> I would likely use an other extended libconsensus too, but do not think
> there was a dependency on
> that for enterprise development.
>
> It would help there more to have a slim protocol server, no wallet, no
> rpc, no qt but a high
> performance remoting API.
>
> Since you already depend on libconsensus for VerifyScript, wouldn't it
> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
> You would still have complete control over storage, concurrency,
> networking, policy...
> My plan is for the C API to interface with the external storage by
> passing a function pointer to it.
>
>
> Storage and validation is non-trivially interconnected, but I now the
> separation can be done,
> since I did it.
>
> Excuse me, but function pointers is a pattern I used in the 80’s. I know
> that they are behind
> the curtain of modern abstractions with similar use, I still prefer not to
> see them again.
>
> Tamas Blummer
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 14121 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Eric Lombrozo]

[-- Attachment #1: Type: text/plain, Size: 7281 bytes --]

One thing it occurs to me (and I don't know if this has been suggested
before) we could do is separate the BIP process into at several distinct
areas:

1) Commit structure changes/consensus rule change proposals
- Consensus-building process (how are proposals debated, improved, vetted,
and selected)
- Update/deployment mechanisms for rule changes
- Specific hard fork proposals
- Specific soft fork proposals

2) Peer policies
- Seeding and discovery mechanisms
- Relay policies
- p2p message support

3) RPC

4) Everything else

On Sat, Aug 22, 2015, 6:28 PM Eric Lombrozo <elombrozo@gmail.com> wrote:

> I've been pushing for greater modularization since I first got into
> bitcoin. I got quickly frustrated when I was only able to get through very
> few things (i.e. moving core structure serialization classes to a separate
> unit not called main). Working on Bitcoin has an added layer of frustration
> that goes beyond most open source projects: even though we're clearly in
> userland working at the application layer, a good layered protocol design
> is still lacking. We have no standards process separate from what basically
> amount to updates to one specific reference implementation. And we all need
> to agree on any major change, since a blockchain that is easily forked in
> contentious ways pretty much defeats its own purpose.
>
> I went off to develop my own stack, where I could more easily avoid
> politics and focus on engineering. But I now understand the politics are
> inevitable. Bitcoin is inherently a cooperative project. Several people
> have poured themselves passionately into the reference codebase, most of
> whom did it (at least initially) purely as unpaid volunteers. There's a lot
> of love that's gone into this. But it's become pretty clear that the
> modularization is no longer merely a matter of good engineering - it is
> essential to resolving serious political challenges.
>
> Perhaps the most frustrating thing of all is watching people pushing for
> relatively superficial yet highly controversial changes while we still lack
> the proper infrastructure to handle these kinds of divergences of opinion
> without either stagnating or becoming polarized.
>
> I could continue working to reimplement an entire stack from scratch, as
> several others have also done - but besides the serious effort duplication
> this entails, it doesn't really seem like it will ultimately be a
> convergent process. It's too easy to let ego and habit dictate one's
> preferences rather than rational engineering considerations.
>
> I know that some might feel I'm just preaching to the choir, but we should
> probably take a step back from implementation hackery and try to specify
> some core protocol layers, focusing on interfaces. Specifically, we need a
> consensus layer that doesn't try to specify networking, storage, wallets,
> UI, etc. Let different people improve upon these things independently in
> their own implementations. What matters is that we all converge on a common
> history and state. At the same time, let's open up more competition on all
> these other things that are separate from the consensus layer.
>
> If only we were to dedicate a fraction of the effort we've put into this
> whole block size circus into what's actually important...and I blame myself
> as well...
>
> On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon.cc> wrote:
>>
>> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com>
>> wrote:
>>
>> Every re-implementation, re-factoring even copy-paste introduces a risk
>> of disagreement,
>> but also open the chance of doing the work better, in the sense of
>> software engineering.
>>
>>
>> But you don't want something better, you want something functionally
>> identical.
>> You may want to watch sipa's explanation on why "the implementation is
>> the specification" and the reasons to separate libconsensus:
>> https://youtu.be/l3O4nh79CUU?t=764
>>
>>
>> I do want something better, but not for the focus you have.
>>
>> Not because what you produce was not high quality, but because quality is
>> achieved at a very
>> high cost and is hard to uphold over generations of developer. You focus
>> on a single use case
>> while there are many out there for distributed ledgers.
>>
>> I think in an infrastructure for enterprise applications, building
>> consensus on the ledger is a
>> cornerstone there, but is only a piece of the solution. I built several
>> commercially successful
>> deployments where I delegated the consensus building to a border router,
>> a Bitcoin Core,
>> then interfaced that trusted peer with my  implementation that accepted
>> Core’s decisions
>> in an SPV manner. One might think of this setup as wasteful and
>> unsuitable for “small devices”
>> therefore an example of centralization people here try to avoid.
>>
>> Enterprises have sufficient resources. Solving the business problem is
>> valuable to them even at
>> magnitudes higher cost than a hobbyist would bear.
>>
>> For mainstream adoption you need to get enterprises on board too, and
>>  that is what I care of.
>> Enterprises want code that is not only high quality, but is easy to
>> maintain with a development
>> team with high attrition. One has to take whatever help is offered for
>> that, and one is modern
>> languages and runtimes.
>>
>> Bits of Proof’s own implementation of the scripts was not practically
>> relevant in my commercially
>> successful deployments, because of the use of a border router, but it
>> helped development,
>> enabling easier debug and precise error feedback esp. end even after Core
>> had a reject message.
>>
>> I integrated libconsensus only for the hope that is significantly fastens
>> application side tx verification,
>>  which it has turned out it does not, until secp265k1 is integrated.
>>
>> I would likely use an other extended libconsensus too, but do not think
>> there was a dependency on
>> that for enterprise development.
>>
>> It would help there more to have a slim protocol server, no wallet, no
>> rpc, no qt but a high
>> performance remoting API.
>>
>> Since you already depend on libconsensus for VerifyScript, wouldn't it
>> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
>> You would still have complete control over storage, concurrency,
>> networking, policy...
>> My plan is for the C API to interface with the external storage by
>> passing a function pointer to it.
>>
>>
>> Storage and validation is non-trivially interconnected, but I now the
>> separation can be done,
>> since I did it.
>>
>> Excuse me, but function pointers is a pattern I used in the 80’s. I know
>> that they are behind
>> the curtain of modern abstractions with similar use, I still prefer not
>> to see them again.
>>
>> Tamas Blummer
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

[-- Attachment #2: Type: text/html, Size: 15088 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 7745 bytes --]

I see the huge amount of sweat and love that went into core and it actually hurts to see that most is expended in friction and lack of a vision for the software architecture.

To be concrete, this was my plan if dealing with the Core code base:

1) I'd consider the separation of networking and storage as suggested for a future extended libconsensus low priority, as their design should be (are) dominated by the need of the consensus logic only.

2) create an API to the consensus+networking+storage service that is not at the C++ language level but some scaleable cross-platform remoting, like eg. ZeroMQ.
This API should be minimal and simple, assuming that one fully trusts the node answering it. This API would unlock user land development by distinct teams with diverse technologies.

3) move the wallet, QT and RPC and other backward compatibility stuff (if e.g. there is some mining support) in-top of the new API and into distinct source code repositories.


Tamas Blummer

> On Aug 23, 2015, at 03:23, Eric Lombrozo <elombrozo@gmail.com> wrote:
> 
> I've been pushing for greater modularization since I first got into bitcoin. I got quickly frustrated when I was only able to get through very few things (i.e. moving core structure serialization classes to a separate unit not called main). Working on Bitcoin has an added layer of frustration that goes beyond most open source projects: even though we're clearly in userland working at the application layer, a good layered protocol design is still lacking. We have no standards process separate from what basically amount to updates to one specific reference implementation. And we all need to agree on any major change, since a blockchain that is easily forked in contentious ways pretty much defeats its own purpose.
> 
> I went off to develop my own stack, where I could more easily avoid politics and focus on engineering. But I now understand the politics are inevitable. Bitcoin is inherently a cooperative project. Several people have poured themselves passionately into the reference codebase, most of whom did it (at least initially) purely as unpaid volunteers. There's a lot of love that's gone into this. But it's become pretty clear that the modularization is no longer merely a matter of good engineering - it is essential to resolving serious political challenges.
> 
> Perhaps the most frustrating thing of all is watching people pushing for relatively superficial yet highly controversial changes while we still lack the proper infrastructure to handle these kinds of divergences of opinion without either stagnating or becoming polarized.
> 
> I could continue working to reimplement an entire stack from scratch, as several others have also done - but besides the serious effort duplication this entails, it doesn't really seem like it will ultimately be a convergent process. It's too easy to let ego and habit dictate one's preferences rather than rational engineering considerations.
> 
> I know that some might feel I'm just preaching to the choir, but we should probably take a step back from implementation hackery and try to specify some core protocol layers, focusing on interfaces. Specifically, we need a consensus layer that doesn't try to specify networking, storage, wallets, UI, etc. Let different people improve upon these things independently in their own implementations. What matters is that we all converge on a common history and state. At the same time, let's open up more competition on all these other things that are separate from the consensus layer.
> 
> If only we were to dedicate a fraction of the effort we've put into this whole block size circus into what's actually important...and I blame myself as well...
> 
> 
> On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon.cc <mailto:jtimon@jtimon.cc>> wrote:
>> 
>> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com <mailto:tamas@bitsofproof.com>> wrote:
>>> Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
>>> but also open the chance of doing the work better, in the sense of software engineering.
>> 
>> But you don't want something better, you want something functionally identical.
>> You may want to watch sipa's explanation on why "the implementation is
>> the specification" and the reasons to separate libconsensus:
>> https://youtu.be/l3O4nh79CUU?t=764 <https://youtu.be/l3O4nh79CUU?t=764>
> 
> I do want something better, but not for the focus you have.
> 
> Not because what you produce was not high quality, but because quality is achieved at a very
> high cost and is hard to uphold over generations of developer. You focus on a single use case
> while there are many out there for distributed ledgers.
> 
> I think in an infrastructure for enterprise applications, building consensus on the ledger is a
> cornerstone there, but is only a piece of the solution. I built several commercially successful
> deployments where I delegated the consensus building to a border router, a Bitcoin Core,
> then interfaced that trusted peer with my  implementation that accepted Core’s decisions
> in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices”
> therefore an example of centralization people here try to avoid.
> 
> Enterprises have sufficient resources. Solving the business problem is valuable to them even at
> magnitudes higher cost than a hobbyist would bear.
> 
> For mainstream adoption you need to get enterprises on board too, and  that is what I care of.
> Enterprises want code that is not only high quality, but is easy to maintain with a development
> team with high attrition. One has to take whatever help is offered for that, and one is modern
> languages and runtimes.
> 
> Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially
> successful deployments, because of the use of a border router, but it helped development,
> enabling easier debug and precise error feedback esp. end even after Core had a reject message.
> 
> I integrated libconsensus only for the hope that is significantly fastens application side tx verification,
>  which it has turned out it does not, until secp265k1 is integrated.
> 
> I would likely use an other extended libconsensus too, but do not think there was a dependency on
> that for enterprise development.
> 
> It would help there more to have a slim protocol server, no wallet, no rpc, no qt but a high
> performance remoting API.
> 
>> Since you already depend on libconsensus for VerifyScript, wouldn't it
>> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
>> You would still have complete control over storage, concurrency,
>> networking, policy...
>> My plan is for the C API to interface with the external storage by
>> passing a function pointer to it.
> 
> 
> Storage and validation is non-trivially interconnected, but I now the separation can be done,
> since I did it.
> 
> Excuse me, but function pointers is a pattern I used in the 80’s. I know that they are behind
> the curtain of modern abstractions with similar use, I still prefer not to see them again.
> 
> Tamas Blummer
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org <mailto:bitcoin-dev@lists.linuxfoundation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>


[-- Attachment #1.2: Type: text/html, Size: 18120 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Sat, Aug 22, 2015 at 1:04 PM, Tamas Blummer <tamas@bitsofproof.com> wrote:
> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon.cc> wrote:
>
> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com>
> wrote:
>
> Every re-implementation, re-factoring even copy-paste introduces a risk of
> disagreement,
> but also open the chance of doing the work better, in the sense of software
> engineering.
>
>
> But you don't want something better, you want something functionally
> identical.
> You may want to watch sipa's explanation on why "the implementation is
> the specification" and the reasons to separate libconsensus:
> https://youtu.be/l3O4nh79CUU?t=764
>
>
> I do want something better, but not for the focus you have.
>
> Not because what you produce was not high quality, but because quality is
> achieved at a very
> high cost and is hard to uphold over generations of developer. You focus on
> a single use case
> while there are many out there for distributed ledgers.
>
> I think in an infrastructure for enterprise applications, building consensus
> on the ledger is a
> cornerstone there, but is only a piece of the solution. I built several
> commercially successful
> deployments where I delegated the consensus building to a border router, a
> Bitcoin Core,
> then interfaced that trusted peer with my  implementation that accepted
> Core’s decisions
> in an SPV manner. One might think of this setup as wasteful and unsuitable
> for “small devices”
> therefore an example of centralization people here try to avoid.
>
> Enterprises have sufficient resources. Solving the business problem is
> valuable to them even at
> magnitudes higher cost than a hobbyist would bear.
>
> For mainstream adoption you need to get enterprises on board too, and  that
> is what I care of.
> Enterprises want code that is not only high quality, but is easy to maintain
> with a development
> team with high attrition. One has to take whatever help is offered for that,
> and one is modern
> languages and runtimes.
>
> Bits of Proof’s own implementation of the scripts was not practically
> relevant in my commercially
> successful deployments, because of the use of a border router, but it helped
> development,
> enabling easier debug and precise error feedback esp. end even after Core
> had a reject message.

In fact I have been accused in the past (by at least Peter Todd) of
having "too many cases in mind" or "doing refactors that are good for
altchains".
That's why I'm very cautious about proposing changes that are not
strict improvments in maintainability to bitcoin itself.
But I actually have freicoin, sidechains and private chains (defined
in freimarkets, used in elements alpha as "block signing") in mind.
Some of the consensus changes I have in mind are support for multiple
assets or interest-bearing assets, for example.
But if you need to change the consensus rules you need to change the
code, there's no way around that.
It will be much simpler to only adapt libconsensus to other chains
than it is to adapt the whole Bitcoin Core code base.
Libconsensus can free you from the need of running "border routers"
(which you need to adapt if you depend on them and are supporting
chains with different rules).
When libconsensus has it's own independent repository, will I fork the
project to have a multi-consensus library supporting multiple
different chains (apart from bitcoin and its testchains)? Maybe, I'm
not sure it makes sense, maybe it's just simpler to maintain a
different project for each different chain (ie libfreicoinconsensus,
libbetaconsensus, etc).

> I integrated libconsensus only for the hope that is significantly fastens
> application side tx verification,
>  which it has turned out it does not, until secp265k1 is integrated.

That is very sad to hear. The main reason to integrate libconsensus is
to avoid consensus fork bugs (or to not depend on the "border routers"
to avoid those bugs).

> I would likely use an other extended libconsensus too, but do not think
> there was a dependency on
> that for enterprise development.
>
> It would help there more to have a slim protocol server, no wallet, no rpc,
> no qt but a high
> performance remoting API.

That's out of scope for libconsensus which will be stateless and whose
only API would be in C.
But the refactors in Bitcoin Core will hopefully make it easier to
support such a minimal node in it (you know you can "./configure
--disable-wallet --without-gui" already, right?, about RPC, that's the
remaining API!).

> Since you already depend on libconsensus for VerifyScript, wouldn't it
> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
> You would still have complete control over storage, concurrency,
> networking, policy...
> My plan is for the C API to interface with the external storage by
> passing a function pointer to it.
>
>
> Storage and validation is non-trivially interconnected, but I now the
> separation can be done,
> since I did it.
>
> Excuse me, but function pointers is a pattern I used in the 80’s. I know
> that they are behind
> the curtain of modern abstractions with similar use, I still prefer not to
> see them again.

Yes, and the wheel it's an invention used in pre-historic times: that
doesn't make it less useful.
Do you have any other suggestion for interfacing with external storage
using a C API?

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

I completely agree and I share your frustration.
The importance of modularization is often disregarded but in my
opinion it has a deep positive impact in the long term: more people
are able to contribute with code and review (in the areas they know
better), the risks associated with each change become more clear
(there was a time when almost any change implied consensus risks),
more alternative code bases can be implemented on top of the basic
ones without fear of consensus bugs, etc.

When I first read some of the code in 2011, I concluded that almost
everything was in main.cpp (which I found ridiculous from a software
engineering perspective). When I started to contribute with code in
2014, main.cpp was still (and still is in my opinion) giant, but the
modularization had greatly improved thanks to changes like moving the
serialization code out of main (thank you very much for that). We
still have a lot of work ahead, but we've certainly advanced a lot.

Unfortunately we cannot force reviewers to pay more attention to
modularization PRs, many of them are usually more interested in
changes that add or remove functionality in the short term. This
problem gets exacerbated when modularization changes are required to
be done in small increments to make them more easily reviewable and
less disruptive to other open PRs, since it's harder for people to see
the big picture and the rationale for those small changes (that often
don't hcange functionality or performance at all).

I know we are not alone on this and people like Wladimir, Pieter, Cory
and Jonas Schnelli (at least, probably more people do) deeply care
about modularization, even if I subjectively and selfishly interpret
the lack of review on some of my PRs as a symptom of the opposite.
So I suggest that people who think this is a high priority join and
review each other's PRs on the subject.

Currently I focus on 3 modularization areas:

1) Chainparams: supporting multiple chains (ie multiple testchains is
all what Bitcoin cares about) is a great goal but there's still many
barriers to create a new testchain. I started this work with #3824,
but even after #6382 there are still more things to do.

2) Consensus: separating the consensus code, Matt Corallo had the idea
of also exposing it in libconsensus. I started with #3839, the latest
things I still have open are #6591 and #6445, please review.

3) Policy: separate node local policy code. Luke Dashjr started with
#5071, I started with #6335 (after several failed attempts), the next
little step blocking many other changes I have ready for way too long
is #6068 (#6424 also helps), please review.

I know Jonas Schnelli is focusing on the wallet. Cory Fields has
recently focused on checkpoints and chainparams.

Now that I know that you also care about modularization I will ask you
for review as well, I hope not to be too annoying like I've been with
Wladimir and Cory some times (and I usually am with some of my
coworkers at blockstream). Please do the same with me: point me to any
modularization PR you have opened.

Regarding your next post, I agree that an additional "Layer" field in
BIPs could be useful. Maybe you should start a BIP for that?

On Sun, Aug 23, 2015 at 3:23 AM, Eric Lombrozo <elombrozo@gmail.com> wrote:
> I've been pushing for greater modularization since I first got into bitcoin.
> I got quickly frustrated when I was only able to get through very few things
> (i.e. moving core structure serialization classes to a separate unit not
> called main). Working on Bitcoin has an added layer of frustration that goes
> beyond most open source projects: even though we're clearly in userland
> working at the application layer, a good layered protocol design is still
> lacking. We have no standards process separate from what basically amount to
> updates to one specific reference implementation. And we all need to agree
> on any major change, since a blockchain that is easily forked in contentious
> ways pretty much defeats its own purpose.
>
> I went off to develop my own stack, where I could more easily avoid politics
> and focus on engineering. But I now understand the politics are inevitable.
> Bitcoin is inherently a cooperative project. Several people have poured
> themselves passionately into the reference codebase, most of whom did it (at
> least initially) purely as unpaid volunteers. There's a lot of love that's
> gone into this. But it's become pretty clear that the modularization is no
> longer merely a matter of good engineering - it is essential to resolving
> serious political challenges.
>
> Perhaps the most frustrating thing of all is watching people pushing for
> relatively superficial yet highly controversial changes while we still lack
> the proper infrastructure to handle these kinds of divergences of opinion
> without either stagnating or becoming polarized.
>
> I could continue working to reimplement an entire stack from scratch, as
> several others have also done - but besides the serious effort duplication
> this entails, it doesn't really seem like it will ultimately be a convergent
> process. It's too easy to let ego and habit dictate one's preferences rather
> than rational engineering considerations.
>
> I know that some might feel I'm just preaching to the choir, but we should
> probably take a step back from implementation hackery and try to specify
> some core protocol layers, focusing on interfaces. Specifically, we need a
> consensus layer that doesn't try to specify networking, storage, wallets,
> UI, etc. Let different people improve upon these things independently in
> their own implementations. What matters is that we all converge on a common
> history and state. At the same time, let's open up more competition on all
> these other things that are separate from the consensus layer.
>
> If only we were to dedicate a fraction of the effort we've put into this
> whole block size circus into what's actually important...and I blame myself
> as well...
>
>
> On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon.cc> wrote:
>>
>> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com>
>> wrote:
>>
>> Every re-implementation, re-factoring even copy-paste introduces a risk of
>> disagreement,
>> but also open the chance of doing the work better, in the sense of
>> software engineering.
>>
>>
>> But you don't want something better, you want something functionally
>> identical.
>> You may want to watch sipa's explanation on why "the implementation is
>> the specification" and the reasons to separate libconsensus:
>> https://youtu.be/l3O4nh79CUU?t=764
>>
>>
>> I do want something better, but not for the focus you have.
>>
>> Not because what you produce was not high quality, but because quality is
>> achieved at a very
>> high cost and is hard to uphold over generations of developer. You focus
>> on a single use case
>> while there are many out there for distributed ledgers.
>>
>> I think in an infrastructure for enterprise applications, building
>> consensus on the ledger is a
>> cornerstone there, but is only a piece of the solution. I built several
>> commercially successful
>> deployments where I delegated the consensus building to a border router, a
>> Bitcoin Core,
>> then interfaced that trusted peer with my  implementation that accepted
>> Core’s decisions
>> in an SPV manner. One might think of this setup as wasteful and unsuitable
>> for “small devices”
>> therefore an example of centralization people here try to avoid.
>>
>> Enterprises have sufficient resources. Solving the business problem is
>> valuable to them even at
>> magnitudes higher cost than a hobbyist would bear.
>>
>> For mainstream adoption you need to get enterprises on board too, and
>> that is what I care of.
>> Enterprises want code that is not only high quality, but is easy to
>> maintain with a development
>> team with high attrition. One has to take whatever help is offered for
>> that, and one is modern
>> languages and runtimes.
>>
>> Bits of Proof’s own implementation of the scripts was not practically
>> relevant in my commercially
>> successful deployments, because of the use of a border router, but it
>> helped development,
>> enabling easier debug and precise error feedback esp. end even after Core
>> had a reject message.
>>
>> I integrated libconsensus only for the hope that is significantly fastens
>> application side tx verification,
>>  which it has turned out it does not, until secp265k1 is integrated.
>>
>> I would likely use an other extended libconsensus too, but do not think
>> there was a dependency on
>> that for enterprise development.
>>
>> It would help there more to have a slim protocol server, no wallet, no
>> rpc, no qt but a high
>> performance remoting API.
>>
>> Since you already depend on libconsensus for VerifyScript, wouldn't it
>> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
>> You would still have complete control over storage, concurrency,
>> networking, policy...
>> My plan is for the C API to interface with the external storage by
>> passing a function pointer to it.
>>
>>
>> Storage and validation is non-trivially interconnected, but I now the
>> separation can be done,
>> since I did it.
>>
>> Excuse me, but function pointers is a pattern I used in the 80’s. I know
>> that they are behind
>> the curtain of modern abstractions with similar use, I still prefer not to
>> see them again.
>>
>> Tamas Blummer
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

[Jorge Timón]

On Sun, Aug 23, 2015 at 8:42 AM, Tamas Blummer <tamas@bitsofproof.com> wrote:
> I see the huge amount of sweat and love that went into core and it actually
> hurts to see that most is expended in friction and lack of a vision for the
> software architecture.
>
> To be concrete, this was my plan if dealing with the Core code base:
>
> 1) I'd consider the separation of networking and storage as suggested for a
> future extended libconsensus low priority, as their design should be (are)
> dominated by the need of the consensus logic only.
>
> 2) create an API to the consensus+networking+storage service that is not at
> the C++ language level but some scaleable cross-platform remoting, like eg.
> ZeroMQ.
> This API should be minimal and simple, assuming that one fully trusts the
> node answering it. This API would unlock user land development by distinct
> teams with diverse technologies.

I plan to replicate the RPC API (or a subset of it) using ZMQ's
req/rep pattern, but #6103 comes first.

> 3) move the wallet, QT and RPC and other backward compatibility stuff (if
> e.g. there is some mining support) in-top of the new API and into distinct
> source code repositories.

Well, the RPC is the API. For libconsensus, its C API is the API.
We've been talking about separating the wallet and qt to a different
repository for long, but modularization is a prerequisite.