From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 4EBDD25A for ; Fri, 30 Oct 2015 04:28:48 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-ig0-f173.google.com (mail-ig0-f173.google.com [209.85.213.173]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 9EC1212A for ; Fri, 30 Oct 2015 04:28:47 +0000 (UTC) Received: by igbhv6 with SMTP id hv6so3604513igb.0 for ; Thu, 29 Oct 2015 21:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=72FH0ykTMrej2HRR1BUp919wGQ23XrwJdF9+7PWz7Lw=; b=lVS8LTNCteUOrialmITRJKZbnzHW99lJ6ZJNhou0rDl5p+2dlgkBrBBRCld4BaKUdI 86/JMg0tw7WW9cwffVD0PzrpmKt9BuNhzJsFrGyUYSHeh+azc3SOzrKjc/gDO6BTEiVm GwFNv4VaY3FcAfHnVF1b4icE6ULojnfK4ujK+nYB7RTyBRRnDrQ77ysyqjIalcHmpmE6 kK1AavIMsMb4fzGArPHi7V0iPUAegMDMON+wl8Hw3fP04atEr+OFAa5tC13rsK79u25y tBkQ3j193dCXIc4jh732recbwmhju/8g18IbCivwSOdYqvs1JTfLlSlL1bg06+sMXff4 xhrQ== MIME-Version: 1.0 X-Received: by 10.50.43.200 with SMTP id y8mr819646igl.48.1446179327142; Thu, 29 Oct 2015 21:28:47 -0700 (PDT) Received: by 10.107.192.199 with HTTP; Thu, 29 Oct 2015 21:28:47 -0700 (PDT) In-Reply-To: <3CB90C47-293E-4C18-A381-E5203483D68F@gmx.com> References: <5631C363.5060705@neomailbox.net> <201510290803.52734.luke@dashjr.org> <5632DE33.7030600@bitcartel.com> <3CB90C47-293E-4C18-A381-E5203483D68F@gmx.com> Date: Fri, 30 Oct 2015 04:28:47 +0000 Message-ID: From: Gregory Maxwell To: Peter R Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev , telemaco Subject: Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 04:28:48 -0000 On Fri, Oct 30, 2015 at 4:04 AM, Peter R wrote: > Can you give a specific example of how nodes that used different database= technologies might determine different answers to whether a given transact= ion is valid or invalid? I=E2=80=99m not a database expert, but to me it w= ould seem that if all the unspent outputs can be found in the database, and= if the relevant information about each output can be retrieved without cor= ruption, then that=E2=80=99s all that really matters as far as the database= is concerned. If you add to those set of assumptions the handling of write ordering is the same (e.g. multiple updates in an change end up with the same entry surviving) and read/write interleave returning the same results then it wouldn't. But databases sometimes have errors which cause them to fail to return records, or to return stale data. And if those exist consistency must be maintained; and "fixing" the bug can cause a divergence in consensus state that could open users up to theft. Case in point, prior to leveldb's use in Bitcoin Core it had a bug that, under rare conditions, could cause it to consistently return not found on records that were really there (I'm running from memory so I don't recall the specific cause). Leveldb fixed this serious bug in a minor update. But deploying a fix like this in an uncontrolled manner in the bitcoin network would potentially cause a fork in the consensus state; so any such fix would need to be rolled out in an orderly manner. > I=E2=80=99d like a concrete example to help me understand why more than o= ne implementation of something like the UTXO database would be unreasonable= . It's not unreasonable, but great care is required around the specifics. Bitcoin consensus implements a mathematical function that defines the operation of the system and above all else all systems must agree (or else the state can diverge and permit double-spends); if you could prove that a component behaves identically under all inputs to another function then it can be replaced without concern but this is something that cannot be done generally for all software, and proving equivalence even in special cases it is an open area of research. The case where the software itself is identical or nearly so is much easier to gain confidence in the equivalence of a change through testing and review. With that cost in mind one must then consider the other side of the equation-- utxo database is an opaque compressed representation, several of the posts here have been about desirability of blockchain analysis interfaces, and I agree they're sometimes desirable but access to the consensus utxo database is not helpful for that. Similarly, other things suggested are so phenomenally slow that it's unlikely that a node would catch up and stay synced even on powerful hardware. Regardless, in Bitcoin core the storage engine for this is fully internally abstracted and so it is relatively straight forward for someone to drop something else in to experiment with; whatever the motivation. I think people are falling into a trap of thinking "It's a , I know a for that!"; but the application and needs are very specialized here; no less than, say-- the table of pre-computed EC points used for signing in the ECDSA application. It just so happens that on the back of the very bitcoin specific cryptographic consensus algorithim there was a slot where a pre-existing high performance key-value store fit; and so we're using one and saving ourselves some effort. If, in the future, Bitcoin Core adopts a merkelized commitment for the UTXO it would probably need to stop using any off-the-shelf key value store entirely, in order to avoid a 20+ fold write inflation from updating hash tree paths (And Bram Cohen has been working on just such a thing, in fact).