From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 466581403 for ; Tue, 15 Sep 2015 04:10:40 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wi0-f177.google.com (mail-wi0-f177.google.com [209.85.212.177]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 523EB19C for ; Tue, 15 Sep 2015 04:10:39 +0000 (UTC) Received: by wicgb1 with SMTP id gb1so10000078wic.1 for ; Mon, 14 Sep 2015 21:10:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=L3pjBRirKQP4+8UquVD8Vv4Xb2NOo5/ldG8fMqRw/Is=; b=fQvptslj0jSh+v8h29p5P53uoLM70LMTQz41+7FdHTFjQk9bMcfW9jCNY/7CJ8UOwn kAvnbf0T0SjsAXN14M9blH2LaNy68YJkr956CfuZGj3iz7c5qHlnGENtJ2dPtCYjRBo/ 4YjwHX6FrFXA64NlCyGSJhWZfX6khtLAbjKg5Kcbj0qWmoNOxjyFh91FbcapqOX748nx OqY6V6mFa9cvkSb3/jTDgavyzPciaODjJLQ1/WBPqBRHgcoUDpEuTiZ/paAgThu/WQ4x dayWSxqzrZPJ0YyXBLYblX8g41NFgchcTst5fxvxurmC+4nRPynPKnJM59TWVv3jV8U8 NFWg== MIME-Version: 1.0 X-Received: by 10.180.76.170 with SMTP id l10mr3041025wiw.68.1442290237969; Mon, 14 Sep 2015 21:10:37 -0700 (PDT) Received: by 10.28.158.9 with HTTP; Mon, 14 Sep 2015 21:10:37 -0700 (PDT) Date: Tue, 15 Sep 2015 00:10:37 -0400 Message-ID: From: Jeff Garzik To: Bitcoin development mailing list Content-Type: text/plain; charset=UTF-8 X-Spam-Status: No, score=-0.8 required=5.0 tests=BAYES_20,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [bitcoin-dev] libconsensus and bitcoin development process X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2015 04:10:40 -0000 [collating a private mail and a github issue comment, moving it to a better forum] On libconsensus --------------- In general there exists the reasonable goal to move consensus state and code to a specific, separate lib. To someone not closely reviewing the seemingly endless stream of libconsensus refactoring PRs, the 10,000 foot view is that there is a rather random stream of refactors that proceed in fits and starts without apparent plan or end other than a one sentence "isolate consensus state and code" summary. I am hoping that * There is some plan * We will not see a five year stream of random consensus code movement patches causing lots of downstream developer headaches. I read every code change in every pull request that comes into github/bitcoin/bitcoin with three exceptions: * consensus code movement changes - too big, too chaotic, too frequent, too unfocused, laziness guarantees others will inevitably ACK it without me. * some non-code changes (docs) * ignore 80% of the Qt changes As with any sort of refactoring, they are easy to prove correct, easy to reason, and therefore quick and easy to ACK and merge. Refactors however have a very real negative impact. bitcoin/bitcoin.git is not only the source tree in the universe. Software engineers at home, at startups, and at major companies are maintaining branches of their own. It is very very easy to fall into a trap where a project is merging lots of cosmetic changes and not seeing the downstream ripple effects. Several people complained to me at the conference about all the code movement changes breaking their own work, causing them to stay on older versions of bitcoin due to the effort required to rebase to each new release version - and I share those complaints. Complex code changes with longer development cycles than simple code movement patches keep breaking. It is very frustrating, and causes folks to get trapped between a rock and a hard place: - Trying to push non-trivial changes upstream is difficult, for normal and reasonable reasons (big important changes need review etc.). - Maintaining non-trivial changes out of tree is also painful, for the aforementioned reasons. Reasonable work languishes in constant-rebase hell, and incentivizes against keeping up with the latest tree. Aside from the refactor, libconsensus appears to be engineering in the dark. Where is any sort of plan? I have low standards - a photo of a whiteboard or youtube clip will do. The general goal is good. But we must not stray into unfocused engineering for a non-existent future library user. The higher priority must be given to having a source code base that maximizes the collective developers' ability to maintain The Router -- the core bitcoin full node P2P engine. I recommend time-based bursts of code movement changes. See below; for example, just submit & merge code movement changes on the first week of every 2nd month. Code movement changes are easy to create from scratch once a concrete goal is known. The coding part is trivial and takes no time. As we saw in the Linux kernel - battle lessons hard learned - code movement and refactors have often unseen negative impact on downstream developers working on more complicated changes that have more positive impact to our developers and users. On Bitcoin development release cycles & process ------------------------------------------------------------------ As I've outlined in the past, the Linux kernel maintenance phases address some of these problems. The merge window into git master opens for 1 week, a very chaotic week full of merging (and rebasing), and then the merge window closes. Several weeks follow as the "dust settles" -- testing, bug fixing, moving in parallel OOB with not-yet-ready development. Release candidates follow, then the release, then the cycle repeats. IMO a merge window approach fixes some of the issues with refactoring, as well as introduces some useful -developer discipline- into the development process. Bitcoin Core still needs rapid iteration -- another failing of the current project -- and so something of a more rapid pace is needed: - 1st week of each month, merge changes. Lots of rebasing during this week. - remaining days of the month, test, bug fix - release at end of month If changes are not ready for merging, then so be it, they wait until next month's release. Some releases have major features, some releases are completely boring and offer little of note. That is the nature of time-based development iteration. It's like dollar cost averaging, a bit. And frankly, I would like to close all github pull requests that are not ready to merge That Week. I'm as guilty of this as any, but that stuff just languishes. Excluding a certain category of obvious-crap, pull requests tend to default to a state of either (a) rapid merging, (b) months-long issues/projects, (c) limbo. Under a more time-based approach, a better pull request process would be to * Only open pull requests if it's a bug fix, or the merge window is open and the change is ready to be merged in the developer's opinion. * Developers CC bitcoin-dev list to discuss Bitcoin Core-bound projects * Developers maintain and publish projects via their own git trees * Pull requests should be closed if unmerged after 7 days, unless it is an important bug fix etc. The problem with projects like libconsensus is that they can get unfocused and open ended. Code movement changes in particular are cheap to generate. It is low developer cost for the developer to iterate all the way to the end state, see what that looks like, and see if people like it. That end state is not something you would merge all in one go. I would likely stash that tree, and then start again, seek the most optimal and least disruptive set of refactors, and generate and merge those into bitcoin/bitcoin.git in a time-based, paced manner. Announce the pace ahead of time - "cosmetic stuff that breaks your patches will be merged 1st week of every second month" To underscore, the higher priority must be given to having a source code base and disciplined development process that maximizes the collective developers' ability to maintain The Router that maintains most of our network. Modularity, refactoring, cleaning up grotty code generates a deep seated happiness in many engineers. Field experience however shows refactoring is a never ending process which sometimes gets in the way of More Important Work.