From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 35BAB8EE for ; Sun, 15 Nov 2015 01:02:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8D12390 for ; Sun, 15 Nov 2015 01:02:44 +0000 (UTC) Received: from [192.168.50.29] ([69.50.179.106]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0LqQTl-1aaw1N3P5F-00e3Sr; Sun, 15 Nov 2015 02:02:37 +0100 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) From: Peter R In-Reply-To: Date: Sat, 14 Nov 2015 17:02:33 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <571D9B7F-077D-4B80-B577-1C18FF2ECF31@gmx.com> References: <5631C363.5060705@neomailbox.net> <201510290803.52734.luke@dashjr.org> <5632DE33.7030600@bitcartel.com> <3CB90C47-293E-4C18-A381-E5203483D68F@gmx.com> To: Gregory Maxwell X-Mailer: Apple Mail (2.2104) Sender: Peter_R@gmx.com X-Provags-ID: V03:K0:hJOP4A6k42YNTcRTX5IZ6HaS2pH4pUtL7PvuQOYXJ5bYf41cauH 3cslW3aN40fJ1WoNxrA1C4NqcTr35xZfxvPfLHZxtm3hI7hPYjvh2fprk6TGOXGvFlalQ6V x58hKz0Bjg9Q37n+ZEhVPTxXqwpW044vrZAiUshw6HQYzF669+zvq4Pn+16WhoqVhZdvDvF 0bKxupFj0yLNpGS8ZZyxw== X-UI-Out-Filterresults: notjunk:1;V01:K0:m5oPBZ0c7f0=:4gwraTYQLajA2hJDobBcFt jLc3g2CnW9uBNW11Ie/7+cu9IcVuBF3tIGcNVyd6y7M/1PIPQOKUIerdnIkuuZZx+yPHBH57I 1m6XTJd7Zfnw/0lUigS+XI4CVcrxNlKPXVwGzb7FmkqM8A9NlWDT3p4UpiKGLMvvcZxia7baa pZB7c5lQ5d1vN/hTHysOaHsoh1lBDLCw77Egkjlc/VP5eVyqG9KASN+XyNjNIRgWkziBJfNfE CwZUmD7+8iFZlTFjC71furR7mcGvRciYGSqOeIFO1yh3BKDGZZTnBhuEWuz4D32UaAvd7iQLI 4BfmwpBqzI6qHRk2xyEdx2p691p9fexZPUzzyuM2Eg4C2qxTN1iVdoApBWttCtJ6bJ9Ni5UAy /jk71wWk8lR6ZK9s9c0T5lOyt38ZxVWFm7y0WGprzxF02Pck/9erhlCg3vLiD+evwtLOir49i y9U1eGwTLaM7FuB6hDoO+2OOYZUwbz7UOqd8SW6lDNs6RJla8dMt4im9Y1QzuAESMa4JFULaX EFeEajmweZ6wCrqedduRVCdV2pvQJ12/bdpuGUNNvpWFJoTPpuVxLHsb/kryD8l5LClulzXt5 ouaq0VPUFeUnytN9StdKSR/JtT2leY51q5p3xdMahq2k1X6251MuPe2Vyrysu3sbMBKQQM8zV NLytqbpTxTRydZ0XH5/foTvofbESVY4Ir6ihviNxjnB/ZnHABV9J+X0oxvitLxvwp63QtWWEW h/dXyYLv/Sj6vGR4gIycgtyh3NMz7zEdK+FpfRDa8IZBJwkCz8GztXOuDO3jg5do6auYvXJXh GbaFlMg X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev , telemaco Subject: Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Nov 2015 01:02:46 -0000 Hi Greg, Like you said, the issue with using more than one database technology is = not that one node would prove that Block X is valid while the another = node proves that Block X is NOT valid. Instead, the problem is that one = node might say =E2=80=9Cvalid=E2=80=9D while the other node says =E2=80=9C= I don=E2=80=99t know.=E2=80=9D It is often said that what caused the Level DB fork was that the old = version determined that the triggering block was invalid while the new = version determined the opposite. This interpretation of the fork event = has lead to the =E2=80=9Cbug-for-bug=E2=80=9D-compatibility fear = regarding multiple implementations of the protocol (or multiple database = technologies). In reality, this fear is largely unfounded. =20 The old version ran out of LDB locks and couldn=E2=80=99t execute = properly. If the software was written with the philosophy that tracking = consensus was more important than bug-for-bug compatibility, it would = have returned an exception rather than =E2=80=9Cinvalid.=E2=80=9D At = that point, the software would have checked what decision the rest of = the network came to about the block in question. My node would have = forked to the longest chain, automatically assuming that the = questionable block was valid; your node may have made a different = decision (it=E2=80=99s a question of ideology at that point). =20 A group of us have been exploring this =E2=80=9Cmeta-cognition=E2=80=9D = idea with Bitcoin Unlimited. For example, Bitcoin Unlimited can be = (optionally) made to automatically fork to the longest chain if it = =E2=80=9Cgets stuck=E2=80=9D and can neither prove that a block is valid = nor that the block is invalid. Similarly, Bitcoin Unlimited can be = (optionally) made to automatically fork to a chain that contains a block = marginally bigger than its block size limit=E2=80=94once that block is = buried sufficiently in what has emerged as the longest persistent chain.=20= Thinking from this perspective might go along way towards decentralizing = development, the emergence of multiple competing implementations of the = protocol, and true =E2=80=9Cbottom up=E2=80=9D governance. =20 Best regards, Peter > On Oct 29, 2015, at 9:28 PM, Gregory Maxwell = wrote: >=20 > On Fri, Oct 30, 2015 at 4:04 AM, Peter R wrote: >> Can you give a specific example of how nodes that used different = database technologies might determine different answers to whether a = given transaction is valid or invalid? I=E2=80=99m not a database = expert, but to me it would seem that if all the unspent outputs can be = found in the database, and if the relevant information about each output = can be retrieved without corruption, then that=E2=80=99s all that really = matters as far as the database is concerned. >=20 > If you add to those set of assumptions the handling of write ordering > is the same (e.g. multiple updates in an change end up with the same > entry surviving) and read/write interleave returning the same results > then it wouldn't. >=20 > But databases sometimes have errors which cause them to fail to return > records, or to return stale data. And if those exist consistency must > be maintained; and "fixing" the bug can cause a divergence in > consensus state that could open users up to theft. >=20 > Case in point, prior to leveldb's use in Bitcoin Core it had a bug > that, under rare conditions, could cause it to consistently return not > found on records that were really there (I'm running from memory so I > don't recall the specific cause). Leveldb fixed this serious bug in a > minor update. But deploying a fix like this in an uncontrolled manner > in the bitcoin network would potentially cause a fork in the consensus > state; so any such fix would need to be rolled out in an orderly > manner. >=20 >> I=E2=80=99d like a concrete example to help me understand why more = than one implementation of something like the UTXO database would be = unreasonable. >=20 > It's not unreasonable, but great care is required around the = specifics. >=20 > Bitcoin consensus implements a mathematical function that defines the > operation of the system and above all else all systems must agree (or > else the state can diverge and permit double-spends); if you could > prove that a component behaves identically under all inputs to another > function then it can be replaced without concern but this is something > that cannot be done generally for all software, and proving > equivalence even in special cases it is an open area of research. The > case where the software itself is identical or nearly so is much > easier to gain confidence in the equivalence of a change through > testing and review. >=20 > With that cost in mind one must then consider the other side of the > equation-- utxo database is an opaque compressed representation, > several of the posts here have been about desirability of blockchain > analysis interfaces, and I agree they're sometimes desirable but > access to the consensus utxo database is not helpful for that. > Similarly, other things suggested are so phenomenally slow that it's > unlikely that a node would catch up and stay synced even on powerful > hardware. Regardless, in Bitcoin core the storage engine for this is > fully internally abstracted and so it is relatively straight forward > for someone to drop something else in to experiment with; whatever the > motivation. >=20 > I think people are falling into a trap of thinking "It's a , > I know a for that!"; but the application and needs are > very specialized here; no less than, say-- the table of pre-computed > EC points used for signing in the ECDSA application. It just so > happens that on the back of the very bitcoin specific cryptographic > consensus algorithim there was a slot where a pre-existing high > performance key-value store fit; and so we're using one and saving > ourselves some effort. If, in the future, Bitcoin Core adopts a > merkelized commitment for the UTXO it would probably need to stop > using any off-the-shelf key value store entirely, in order to avoid a > 20+ fold write inflation from updating hash tree paths (And Bram Cohen > has been working on just such a thing, in fact).