Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

["Jorge Timón" <jtimon@jtimon.cc>]

I completely agree and I share your frustration.
The importance of modularization is often disregarded but in my
opinion it has a deep positive impact in the long term: more people
are able to contribute with code and review (in the areas they know
better), the risks associated with each change become more clear
(there was a time when almost any change implied consensus risks),
more alternative code bases can be implemented on top of the basic
ones without fear of consensus bugs, etc.

When I first read some of the code in 2011, I concluded that almost
everything was in main.cpp (which I found ridiculous from a software
engineering perspective). When I started to contribute with code in
2014, main.cpp was still (and still is in my opinion) giant, but the
modularization had greatly improved thanks to changes like moving the
serialization code out of main (thank you very much for that). We
still have a lot of work ahead, but we've certainly advanced a lot.

Unfortunately we cannot force reviewers to pay more attention to
modularization PRs, many of them are usually more interested in
changes that add or remove functionality in the short term. This
problem gets exacerbated when modularization changes are required to
be done in small increments to make them more easily reviewable and
less disruptive to other open PRs, since it's harder for people to see
the big picture and the rationale for those small changes (that often
don't hcange functionality or performance at all).

I know we are not alone on this and people like Wladimir, Pieter, Cory
and Jonas Schnelli (at least, probably more people do) deeply care
about modularization, even if I subjectively and selfishly interpret
the lack of review on some of my PRs as a symptom of the opposite.
So I suggest that people who think this is a high priority join and
review each other's PRs on the subject.

Currently I focus on 3 modularization areas:

1) Chainparams: supporting multiple chains (ie multiple testchains is
all what Bitcoin cares about) is a great goal but there's still many
barriers to create a new testchain. I started this work with #3824,
but even after #6382 there are still more things to do.

2) Consensus: separating the consensus code, Matt Corallo had the idea
of also exposing it in libconsensus. I started with #3839, the latest
things I still have open are #6591 and #6445, please review.

3) Policy: separate node local policy code. Luke Dashjr started with
#5071, I started with #6335 (after several failed attempts), the next
little step blocking many other changes I have ready for way too long
is #6068 (#6424 also helps), please review.

I know Jonas Schnelli is focusing on the wallet. Cory Fields has
recently focused on checkpoints and chainparams.

Now that I know that you also care about modularization I will ask you
for review as well, I hope not to be too annoying like I've been with
Wladimir and Cory some times (and I usually am with some of my
coworkers at blockstream). Please do the same with me: point me to any
modularization PR you have opened.

Regarding your next post, I agree that an additional "Layer" field in
BIPs could be useful. Maybe you should start a BIP for that?

On Sun, Aug 23, 2015 at 3:23 AM, Eric Lombrozo <elombrozo@gmail.com> wrote:
> I've been pushing for greater modularization since I first got into bitcoin.
> I got quickly frustrated when I was only able to get through very few things
> (i.e. moving core structure serialization classes to a separate unit not
> called main). Working on Bitcoin has an added layer of frustration that goes
> beyond most open source projects: even though we're clearly in userland
> working at the application layer, a good layered protocol design is still
> lacking. We have no standards process separate from what basically amount to
> updates to one specific reference implementation. And we all need to agree
> on any major change, since a blockchain that is easily forked in contentious
> ways pretty much defeats its own purpose.
>
> I went off to develop my own stack, where I could more easily avoid politics
> and focus on engineering. But I now understand the politics are inevitable.
> Bitcoin is inherently a cooperative project. Several people have poured
> themselves passionately into the reference codebase, most of whom did it (at
> least initially) purely as unpaid volunteers. There's a lot of love that's
> gone into this. But it's become pretty clear that the modularization is no
> longer merely a matter of good engineering - it is essential to resolving
> serious political challenges.
>
> Perhaps the most frustrating thing of all is watching people pushing for
> relatively superficial yet highly controversial changes while we still lack
> the proper infrastructure to handle these kinds of divergences of opinion
> without either stagnating or becoming polarized.
>
> I could continue working to reimplement an entire stack from scratch, as
> several others have also done - but besides the serious effort duplication
> this entails, it doesn't really seem like it will ultimately be a convergent
> process. It's too easy to let ego and habit dictate one's preferences rather
> than rational engineering considerations.
>
> I know that some might feel I'm just preaching to the choir, but we should
> probably take a step back from implementation hackery and try to specify
> some core protocol layers, focusing on interfaces. Specifically, we need a
> consensus layer that doesn't try to specify networking, storage, wallets,
> UI, etc. Let different people improve upon these things independently in
> their own implementations. What matters is that we all converge on a common
> history and state. At the same time, let's open up more competition on all
> these other things that are separate from the consensus layer.
>
> If only we were to dedicate a fraction of the effort we've put into this
> whole block size circus into what's actually important...and I blame myself
> as well...
>
>
> On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon.cc> wrote:
>>
>> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com>
>> wrote:
>>
>> Every re-implementation, re-factoring even copy-paste introduces a risk of
>> disagreement,
>> but also open the chance of doing the work better, in the sense of
>> software engineering.
>>
>>
>> But you don't want something better, you want something functionally
>> identical.
>> You may want to watch sipa's explanation on why "the implementation is
>> the specification" and the reasons to separate libconsensus:
>> https://youtu.be/l3O4nh79CUU?t=764
>>
>>
>> I do want something better, but not for the focus you have.
>>
>> Not because what you produce was not high quality, but because quality is
>> achieved at a very
>> high cost and is hard to uphold over generations of developer. You focus
>> on a single use case
>> while there are many out there for distributed ledgers.
>>
>> I think in an infrastructure for enterprise applications, building
>> consensus on the ledger is a
>> cornerstone there, but is only a piece of the solution. I built several
>> commercially successful
>> deployments where I delegated the consensus building to a border router, a
>> Bitcoin Core,
>> then interfaced that trusted peer with my  implementation that accepted
>> Core’s decisions
>> in an SPV manner. One might think of this setup as wasteful and unsuitable
>> for “small devices”
>> therefore an example of centralization people here try to avoid.
>>
>> Enterprises have sufficient resources. Solving the business problem is
>> valuable to them even at
>> magnitudes higher cost than a hobbyist would bear.
>>
>> For mainstream adoption you need to get enterprises on board too, and
>> that is what I care of.
>> Enterprises want code that is not only high quality, but is easy to
>> maintain with a development
>> team with high attrition. One has to take whatever help is offered for
>> that, and one is modern
>> languages and runtimes.
>>
>> Bits of Proof’s own implementation of the scripts was not practically
>> relevant in my commercially
>> successful deployments, because of the use of a border router, but it
>> helped development,
>> enabling easier debug and precise error feedback esp. end even after Core
>> had a reject message.
>>
>> I integrated libconsensus only for the hope that is significantly fastens
>> application side tx verification,
>>  which it has turned out it does not, until secp265k1 is integrated.
>>
>> I would likely use an other extended libconsensus too, but do not think
>> there was a dependency on
>> that for enterprise development.
>>
>> It would help there more to have a slim protocol server, no wallet, no
>> rpc, no qt but a high
>> performance remoting API.
>>
>> Since you already depend on libconsensus for VerifyScript, wouldn't it
>> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
>> You would still have complete control over storage, concurrency,
>> networking, policy...
>> My plan is for the C API to interface with the external storage by
>> passing a function pointer to it.
>>
>>
>> Storage and validation is non-trivially interconnected, but I now the
>> separation can be done,
>> since I did it.
>>
>> Excuse me, but function pointers is a pattern I used in the 80’s. I know
>> that they are behind
>> the curtain of modern abstractions with similar use, I still prefer not to
>> see them again.
>>
>> Tamas Blummer
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev