[bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

Here is the beginnings of an implementation to replace leveldb with sqlite:
https://github.com/jgarzik/bitcoin/tree/2015_sqlite

It builds, but still needs work before passing tests.

It was noted that leveldb is unmaintained, and this is part of researching
alternatives that are maintained and reliable.

[-- Attachment #2: Type: text/html, Size: 473 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Patrick Strateman]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

Benchmarks?

I cant imagine that's very fast.

On 10/22/2015 02:26 PM, Jeff Garzik via bitcoin-dev wrote:
> Here is the beginnings of an implementation to replace leveldb with
> sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite
>
> It builds, but still needs work before passing tests.
>
> It was noted that leveldb is unmaintained, and this is part of
> researching alternatives that are maintained and reliable.
>
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[-- Attachment #2: Type: text/html, Size: 1606 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Joseph Gleason ⑈]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

I have done a lot of recent work on local key value stores, mostly for a
java electrum server I am working on.

I'd suggest considering LMDB.  One downside is that it is memory mapped so
32-bit systems that need over 2gb of storage are right out.  Other than
that, it is quite fast and seems reliable in my testing.


On Thu, Oct 22, 2015 at 2:37 PM Jeff Garzik via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Here is the beginnings of an implementation to replace leveldb with
> sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite
>
> It builds, but still needs work before passing tests.
>
> It was noted that leveldb is unmaintained, and this is part of researching
> alternatives that are maintained and reliable.
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 1597 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Jonas Schnelli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Here is the beginnings of an implementation to replace leveldb
> with sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite
> 
> It builds, but still needs work before passing tests.

Nice work!

Although not sure if we should focus directly on sqlite4 (could be
optional with a configure flag and a subtree [until stable], sqlite3
supported over depends).

Also i personally would recommend to not implement it as a
replacement, instead, support multiple backends (wrapper header /
different wrapper implementations [leveldb / sqlite3 / sqlite4]). But
– agreed – should not be the focus, but a nice additional flexibility
if it not require much more work.
And – in theory – multiple database back-ends would allow migration.

Before investigating to deep, i think we need a dbwrapper bench tool
that represents our needs and our style how we hit the database.
Gavins recently added bench target / bench environment could be used
for that.

/jonas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJWKdlgAAoJECnUvLZBb1Ps7J8P/2L6215fd0rWGv5uvbLSnQvm
hy1T2AaOfH5HXd2m95icaKYk+ugvQAL80Q/67YwZbPLsT4fErgegw8n75Z6nh/Pc
OJ1EtAvD+Kc/Vm0Wcvt421HXwnm4f+j+8eoEvpG6DdC8gfIG+efhM7DljeNbPFrA
ieNe7XrQ1EZ29lMzpQv0nx3bi6tUHOWuazk82B6vnK49MjH7nrUFipcc18nXbSpM
ZKjQakXmfqIBG8QP9ZUYUlW4aoG0oaOoZnGjQA2LeXBJpIPLpE/WPg0XaXubC+No
232CJNtNyUOmjkb2qbep6vSYgqGJNy1HbCU5y3qooxJhFnKdo63CQsyJKSpL/ssi
0lWraNxjbacxsBn+63In3wEkj02orwm2zTO4I77wCrZmJgpvBFb9bZWTeL8DCYSG
DTkZoKWEK74xvm+dNpEWXK9Lm1ltfyhPdaFeMoRDub4w2uuYlk3KuD8Vdy81HYJb
sak8FbkiWw9xx2OP9+G56Arf9W6mnJ3YHJGrY4SXeRAfuNYwGFHIGt6I1JobuHy4
VWmcHuooz/Q9JLUu3Rr3HsEdXYgCmgmWuzTgWG+Hx92Y6XLWV5pOX+vFRO6J5fSm
aTYPD4GsTM3FROXw5ezlXJ+2y1+ITzmrfm03fEDQvoIyH0TwBqBc5sMBBkma5DDR
0HUthPHWCD+AxbBPbRVa
=CUMX
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Lucas Betschart]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

Facebook has a LevelDB fork which is maintained.
It's called RocksDB and the API seems to be nearly the same as for LevelDB,
thus maybe easy to replace: http://rocksdb.org/
https://github.com/facebook/rocksdb

Although I don't know if we might have some negative effects for our
use-case since RocksDB was optimized for big databases running on multiple
cores.


2015-10-22 23:26 GMT+02:00 Jeff Garzik via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org>:

> Here is the beginnings of an implementation to replace leveldb with
> sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite
>
> It builds, but still needs work before passing tests.
>
> It was noted that leveldb is unmaintained, and this is part of researching
> alternatives that are maintained and reliable.
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 1777 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Sean Lynch]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On Fri, Oct 23, 2015 at 1:23 AM Lucas Betschart via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Facebook has a LevelDB fork which is maintained.
> It's called RocksDB and the API seems to be nearly the same as for
> LevelDB, thus maybe easy to replace: http://rocksdb.org/
> https://github.com/facebook/rocksdb
>
> Although I don't know if we might have some negative effects for our
> use-case since RocksDB was optimized for big databases running on multiple
> cores.
>

While RocksDB is pretty decent, note that it's optimized for flash. Not
sure how well it will work on spinning disks.

[-- Attachment #2: Type: text/html, Size: 1063 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

On Wed, Oct 28, 2015 at 4:28 PM, Sean Lynch <seanl@literati.org> wrote:

> On Fri, Oct 23, 2015 at 1:23 AM Lucas Betschart via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Facebook has a LevelDB fork which is maintained.
>> It's called RocksDB and the API seems to be nearly the same as for
>> LevelDB, thus maybe easy to replace: http://rocksdb.org/
>> https://github.com/facebook/rocksdb
>>
>> Although I don't know if we might have some negative effects for our
>> use-case since RocksDB was optimized for big databases running on multiple
>> cores.
>>
>
> While RocksDB is pretty decent, note that it's optimized for flash. Not
> sure how well it will work on spinning disks.
>

That's OK for our purposes.  We have a huge database which already
incentivized having zero seek time.

[-- Attachment #2: Type: text/html, Size: 1673 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Tom Zander]

On Thursday 22 Oct 2015 17:26:42 Jeff Garzik via bitcoin-dev wrote:
> It was noted that leveldb is unmaintained, and this is part of researching
> alternatives that are maintained and reliable.

Apart from it being unmaintained, any links to what are problems with levelDB?

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Douglas Roark]

On 2015/10/23 03:30, Tom Zander via bitcoin-dev wrote:
> On Thursday 22 Oct 2015 17:26:42 Jeff Garzik via bitcoin-dev wrote:
>> It was noted that leveldb is unmaintained, and this is part of researching
>> alternatives that are maintained and reliable.
> 
> Apart from it being unmaintained, any links to what are problems with levelDB?

While not exactly the most rigorous link,
https://en.wikipedia.org/wiki/LevelDB#Bugs_and_Reliability seems like an
okay place to start. One thing I can attest to is that, when Armory used
LevelDB (0.8 - 0.92, IIRC), quite a few users had DB corruption issues,
particularly on Windows. Even when a switch to LMDB occurred for 0.93,
loads of complaints would come in from users whose LevelDB-based Core
DBs would fail. I know that the guy who moved Armory over to LMDB would
love to have more time in the day so that he could write a Core patch
that does the same. It's a very sore spot for him.

(FWIW, LMDB seems to work quite nicely, at least once you patch up the
source a little bit. The latest version is also compatible with Core's
cross-compiling scheme. I'd love to see it added to Core one day.)

Doug

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Tom Zander]

On Monday 26 Oct 2015 11:06:56 Douglas Roark via bitcoin-dev wrote:
> While not exactly the most rigorous link,
> https://en.wikipedia.org/wiki/LevelDB#Bugs_and_Reliability seems like an
> okay place to start.

Thanks for that link!

Another Google open source product I'll avoid like the plague ;)

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Jonathan Wilkins]

[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]

Benchmarks for various DBs under discussion:
http://symas.com/mdb/microbench/


On Mon, Oct 26, 2015 at 11:06 AM, Douglas Roark via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On 2015/10/23 03:30, Tom Zander via bitcoin-dev wrote:
> > On Thursday 22 Oct 2015 17:26:42 Jeff Garzik via bitcoin-dev wrote:
> >> It was noted that leveldb is unmaintained, and this is part of
> researching
> >> alternatives that are maintained and reliable.
> >
> > Apart from it being unmaintained, any links to what are problems with
> levelDB?
>
> While not exactly the most rigorous link,
> https://en.wikipedia.org/wiki/LevelDB#Bugs_and_Reliability seems like an
> okay place to start. One thing I can attest to is that, when Armory used
> LevelDB (0.8 - 0.92, IIRC), quite a few users had DB corruption issues,
> particularly on Windows. Even when a switch to LMDB occurred for 0.93,
> loads of complaints would come in from users whose LevelDB-based Core
> DBs would fail. I know that the guy who moved Armory over to LMDB would
> love to have more time in the day so that he could write a Core patch
> that does the same. It's a very sore spot for him.
>
> (FWIW, LMDB seems to work quite nicely, at least once you patch up the
> source a little bit. The latest version is also compatible with Core's
> cross-compiling scheme. I'd love to see it added to Core one day.)
>
> Doug
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 2390 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[telemaco]

Why not allow two options:

1/ a default RocksDB/SQLite/LevelDB (whatever is decided)
2/ alternative provide instructions for connection to any other rdbms 
using odbc or jdbc.

Why not allowing async disk writes or incredibly fast database systems 
if someone wants to have a node in a very fast datacenter or connected 
with their existing leveraged dataservers. It is the traditional 
approach to just use the open standard for database connectivity.

Any person or any organization would just need to have one machine with 
their bitcoin node with a rdbms client installed (SAP Sybase client, or 
oracle client, or microsoft). The bitcoin node would just store their 
data using the odbc/jdbc protocol on ANY rdbms installed anywhere in 
their organization (other machine or the same). They would just need to 
issue a "create table" with a very simple table structure and they would 
benefit from async and indexes and using their already licensed, and 
configured system of their choosing, with bitcoin information being 
available to thousands of software packages and available aswell to 
thousands of programmers that work with rdbms and not just "RocksDB" or 
some obscure database system.

Why not "outsource" totally that data management part to the already 
existing with decades of experience database world. People would be able 
to create incredibly easy bitcoin statistics/graphs/analisys with 
existing software packages (hey even excel or libreoffice like) or 
connect bitcoin data to their own sources and if so they chose analyze 
bitcoin data on a datawarehouse or any imaginable approach. Of course 
every transaction would be have to do through the bitcoin node and only 
the data management would be on rdbms side.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Luke Dashjr]

On Thursday, October 29, 2015 6:57:39 AM telemaco via bitcoin-dev wrote:
> Why not allow two options:
> 
> 1/ a default RocksDB/SQLite/LevelDB (whatever is decided)
> 2/ alternative provide instructions for connection to any other rdbms
> using odbc or jdbc.

I predict this would be a disaster. UTXO storage is CONSENSUS-CRITICAL code.
Any divergence in implementation behaviour, including bugs AND bugfixes, may 
cause consensus failure. For this to have a reasonable *hope* of working, we 
need to choose one storage engine, and *will* need to maintain consensus-
compatibility of it ourselves (since nobody else cares).

Fixing LevelDB frankly seems like an easier task than switching to anything 
SQL-based, which would require a *lot* more *difficult-to-get-consensus-
compatible* code that we are all (or at least mostly) very unfamiliar with.

Research is fine, but let's be realistic about deployment.

Luke

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Simon Liu]

Storage of UTXO data looks like an implementation detail and thus one
would have thought that the choice of database would not increase the
odds of consensus protocol failure.

Btcd, a full node implementation written in Go, already provides a
database interface which supports different backends:

https://github.com/btcsuite/btcd/tree/master/database

Given that UTXO storage is considered critical, it seems reasonable to
let a node operator decide for themselves if they want data stored in
LevelDB (which is not fully ACID compliant) or a database like Sqlite,
Oracle, DB2 etc.

If the storage requirements for UTXO data are fairly simple, consisting
mainly of puts and gets, there is a decent argument that using a
dedicated key-value store provides superior performance over a
traditional SQL database.

However, from a practical perspective, given that nodes operate on a
range of different hardware and even a little Raspberry Pi can run a
full node and keep up with the network, why not let those users with the
resources to operate big iron databases do so?  It would be a good
feature to have.


On 10/29/2015 01:03 AM, Luke Dashjr via bitcoin-dev wrote:
> I predict this would be a disaster. UTXO storage is CONSENSUS-CRITICAL code.
> Any divergence in implementation behaviour, including bugs AND bugfixes, may 
> cause consensus failure. For this to have a reasonable *hope* of working, we 
> need to choose one storage engine, and *will* need to maintain consensus-
> compatibility of it ourselves (since nobody else cares).
> 
> Fixing LevelDB frankly seems like an easier task than switching to anything 
> SQL-based, which would require a *lot* more *difficult-to-get-consensus-
> compatible* code that we are all (or at least mostly) very unfamiliar with.
> 
> Research is fine, but let's be realistic about deployment.
> 
> Luke
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Gregory Maxwell]

On Fri, Oct 30, 2015 at 3:04 AM, Simon Liu via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Given that UTXO storage is considered critical, it seems reasonable to

This sounds like a misunderstanding of what consensus criticial means.
It does not mean that it must be right (though obviously that is
preferable) but that it must be _consistent_, between all nodes.

> full node and keep up with the network, why not let those users with the
> resources to operate big iron databases do so?  It would be a good
> feature to have.

Because it provides no value, the data is opaque and propritarily
encoded with a compression function which we may change from version
to version, and because many of these alternatives are enormously
slow; enough that they present problems with falling behind the
network even on high performance hardware.

Moreover, additional functional which will not be sufficiently used
will not adequately maintained and result in increased maintains costs
and more bugs.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

> On Oct 29, 2015, at 8:35 PM, Gregory Maxwell via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> On Fri, Oct 30, 2015 at 3:04 AM, Simon Liu via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> Given that UTXO storage is considered critical, it seems reasonable to
> 
> This sounds like a misunderstanding of what consensus criticial means.
> It does not mean that it must be right (though obviously that is
> preferable) but that it must be _consistent_, between all nodes.

Can you give a specific example of how nodes that used different database technologies might determine different answers to whether a given transaction is valid or invalid?  I’m not a database expert, but to me it would seem that if all the unspent outputs can be found in the database, and if the relevant information about each output can be retrieved without corruption, then that’s all that really matters as far as the database is concerned.

Let’s use an unspent pay-to-pubkey-hash output as an example: Alice spends this to Bob (she signs it properly), the TX propagates across the network and…then what?  Do some nodes disagree on whether or not the TX is valid?  What exactly would they disagree on?  Are you suggesting that a database bug would cause some nodes to think the output was actually already spent, while others can correctly see that it’s unspent?  Or maybe some nodes think the output doesn’t exist while others do?  Or are you suggesting that the details about this output might be retrieved with errors from certain databases but correctly from others?  

I’d like a concrete example to help me understand why more than one implementation of something like the UTXO database would be unreasonable.    

Peter

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Gregory Maxwell]

On Fri, Oct 30, 2015 at 4:04 AM, Peter R <peter_r@gmx.com> wrote:
> Can you give a specific example of how nodes that used different database technologies might determine different answers to whether a given transaction is valid or invalid?  I’m not a database expert, but to me it would seem that if all the unspent outputs can be found in the database, and if the relevant information about each output can be retrieved without corruption, then that’s all that really matters as far as the database is concerned.

If you add to those set of assumptions the handling of write ordering
is the same (e.g. multiple updates in an change end up with the same
entry surviving) and read/write interleave returning the same results
then it wouldn't.

But databases sometimes have errors which cause them to fail to return
records, or to return stale data. And if those exist consistency must
be maintained; and "fixing" the bug can cause a divergence in
consensus state that could open users up to theft.

Case in point, prior to leveldb's use in Bitcoin Core it had a bug
that, under rare conditions, could cause it to consistently return not
found on records that were really there (I'm running from memory so I
don't recall the specific cause).  Leveldb fixed this serious bug in a
minor update.  But deploying a fix like this in an uncontrolled manner
in the bitcoin network would potentially cause a fork in the consensus
state; so any such fix would need to be rolled out in an orderly
manner.

> I’d like a concrete example to help me understand why more than one implementation of something like the UTXO database would be unreasonable.

It's not unreasonable, but great care is required around the specifics.

Bitcoin consensus implements a mathematical function that defines the
operation of the system and above all else all systems must agree (or
else the state can diverge and permit double-spends);  if you could
prove that a component behaves identically under all inputs to another
function then it can be replaced without concern but this is something
that cannot be done generally for all software, and proving
equivalence even in special cases it is an open area of research.  The
case where the software itself is identical or nearly so is much
easier to gain confidence in the equivalence of a change through
testing and review.

With that cost in mind one must then consider the other side of the
equation-- utxo database is an opaque compressed representation,
several of the posts here have been about desirability of blockchain
analysis interfaces, and I agree they're sometimes desirable but
access to the consensus utxo database is not helpful for that.
Similarly, other things suggested are so phenomenally slow that it's
unlikely that a node would catch up and stay synced even on powerful
hardware.  Regardless, in Bitcoin core the storage engine for this is
fully internally abstracted and so it is relatively straight forward
for someone to drop something else in to experiment with; whatever the
motivation.

I think people are falling into a trap of thinking "It's a <database>,
I know a <black box> for that!"; but the application and needs are
very specialized here; no less than, say-- the table of pre-computed
EC points used for signing in the ECDSA application. It just so
happens that on the back of the very bitcoin specific cryptographic
consensus algorithim there was a slot where a pre-existing high
performance key-value store fit; and so we're using one and saving
ourselves some effort.  If, in the future, Bitcoin Core adopts a
merkelized commitment for the UTXO it would probably need to stop
using any off-the-shelf key value store entirely, in order to avoid a
20+ fold write inflation from updating hash tree paths (And Bram Cohen
has been working on just such a thing, in fact).

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

Hi Greg,

Like you said, the issue with using more than one database technology is not that one node would prove that Block X is valid while the another node proves that Block X is NOT valid.  Instead, the problem is that one node might say “valid” while the other node says “I don’t know.”

It is often said that what caused the Level DB fork was that the old version determined that the triggering block was invalid while the new version determined the opposite.  This interpretation of the fork event has lead to the “bug-for-bug”-compatibility fear regarding multiple implementations of the protocol (or multiple database technologies).  In reality, this fear is largely unfounded.  

The old version ran out of LDB locks and couldn’t execute properly.  If the software was written with the philosophy that tracking consensus was more important than bug-for-bug compatibility, it would have returned an exception rather than “invalid.”  At that point, the software would have checked what decision the rest of the network came to about the block in question.   My node would have forked to the longest chain, automatically assuming that the questionable block was valid; your node may have made a different decision (it’s a question of ideology at that point).      

A group of us have been exploring this “meta-cognition” idea with Bitcoin Unlimited.  For example, Bitcoin Unlimited can be (optionally) made to automatically fork to the longest chain if it “gets stuck” and can neither prove that a block is valid nor that the block is invalid.  Similarly, Bitcoin Unlimited can be (optionally) made to automatically fork to a chain that contains a block marginally bigger than its block size limit—once that block is buried sufficiently in what has emerged as the longest persistent chain. 

Thinking from this perspective might go along way towards decentralizing development, the emergence of multiple competing implementations of the protocol, and true “bottom up” governance.  

Best regards,
Peter



> On Oct 29, 2015, at 9:28 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> 
> On Fri, Oct 30, 2015 at 4:04 AM, Peter R <peter_r@gmx.com> wrote:
>> Can you give a specific example of how nodes that used different database technologies might determine different answers to whether a given transaction is valid or invalid?  I’m not a database expert, but to me it would seem that if all the unspent outputs can be found in the database, and if the relevant information about each output can be retrieved without corruption, then that’s all that really matters as far as the database is concerned.
> 
> If you add to those set of assumptions the handling of write ordering
> is the same (e.g. multiple updates in an change end up with the same
> entry surviving) and read/write interleave returning the same results
> then it wouldn't.
> 
> But databases sometimes have errors which cause them to fail to return
> records, or to return stale data. And if those exist consistency must
> be maintained; and "fixing" the bug can cause a divergence in
> consensus state that could open users up to theft.
> 
> Case in point, prior to leveldb's use in Bitcoin Core it had a bug
> that, under rare conditions, could cause it to consistently return not
> found on records that were really there (I'm running from memory so I
> don't recall the specific cause).  Leveldb fixed this serious bug in a
> minor update.  But deploying a fix like this in an uncontrolled manner
> in the bitcoin network would potentially cause a fork in the consensus
> state; so any such fix would need to be rolled out in an orderly
> manner.
> 
>> I’d like a concrete example to help me understand why more than one implementation of something like the UTXO database would be unreasonable.
> 
> It's not unreasonable, but great care is required around the specifics.
> 
> Bitcoin consensus implements a mathematical function that defines the
> operation of the system and above all else all systems must agree (or
> else the state can diverge and permit double-spends);  if you could
> prove that a component behaves identically under all inputs to another
> function then it can be replaced without concern but this is something
> that cannot be done generally for all software, and proving
> equivalence even in special cases it is an open area of research.  The
> case where the software itself is identical or nearly so is much
> easier to gain confidence in the equivalence of a change through
> testing and review.
> 
> With that cost in mind one must then consider the other side of the
> equation-- utxo database is an opaque compressed representation,
> several of the posts here have been about desirability of blockchain
> analysis interfaces, and I agree they're sometimes desirable but
> access to the consensus utxo database is not helpful for that.
> Similarly, other things suggested are so phenomenally slow that it's
> unlikely that a node would catch up and stay synced even on powerful
> hardware.  Regardless, in Bitcoin core the storage engine for this is
> fully internally abstracted and so it is relatively straight forward
> for someone to drop something else in to experiment with; whatever the
> motivation.
> 
> I think people are falling into a trap of thinking "It's a <database>,
> I know a <black box> for that!"; but the application and needs are
> very specialized here; no less than, say-- the table of pre-computed
> EC points used for signing in the ECDSA application. It just so
> happens that on the back of the very bitcoin specific cryptographic
> consensus algorithim there was a slot where a pre-existing high
> performance key-value store fit; and so we're using one and saving
> ourselves some effort.  If, in the future, Bitcoin Core adopts a
> merkelized commitment for the UTXO it would probably need to stop
> using any off-the-shelf key value store entirely, in order to avoid a
> 20+ fold write inflation from updating hash tree paths (And Bram Cohen
> has been working on just such a thing, in fact).

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Gregory Maxwell]

On Sun, Nov 15, 2015 at 1:02 AM, Peter R <peter_r@gmx.com> wrote:
> Hi Greg,
>
> Like you said, the issue with using more than one database technology is not that one node would prove that Block X is valid while the another node proves that Block X is NOT valid.  Instead, the problem is that one node might say “valid” while the other node says “I don’t know.”

Sometimes errors are such that you can catch them (if you're super
vigilant and know an error is even possible in that case)-- and
indeed, in that case you can get a "I don't know, something is
wrong.", other times errors are undetectable.

> In reality, this fear is largely unfounded.

I cited an issue in leveldb (before we used it) where it would
silently fail to return records.

> If the software was written with the philosophy that tracking consensus was more important than bug-for-bug compatibility, it would have returned an exception rather than “invalid.”

That invariant requires the software to be completely free of errors
that would violate it.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

> On Nov 14, 2015, at 5:08 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> 
> On Sun, Nov 15, 2015 at 1:02 AM, Peter R <peter_r@gmx.com> wrote:
>> Hi Greg,
>> 
>> Like you said, the issue with using more than one database technology is not that one node would prove that Block X is valid while the another node proves that Block X is NOT valid.  Instead, the problem is that one node might say “valid” while the other node says “I don’t know.”
> 
> Sometimes errors are such that you can catch them (if you're super
> vigilant and know an error is even possible in that case)-- and
> indeed, in that case you can get a "I don't know, something is
> wrong.", other times errors are undetectable.

Agreed.  There are two cases to consider:

Type 1.  One implementation says “yes” or “no,” while the other says “I don’t know”, and

Type 2.  One implementation says “yes” and the other says “no,” because of a bug.  

My previous email described how Type 1 consensus failures can be safely dealt with.  These include many kinds of database exceptions (e.g., the LevelDB fork at block #225,430), or consensus mismatches regarding the max size of a block.  

Type 2 consensus failures are more severe but also less likely (I’m not aware of a Type 2 consensus failure besides the 92 million bitcoin bug from August 2010).  If Core was to accept a rogue TX that created another 92 million bitcoins, I think it would be a good thing if the other implementations forked away from it (we don’t want bug-for-bug compatibility here).   

This once again reveals the benefits of multiple competing implementations.  

Sincerely,
Peter 

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Gregory Maxwell]

On Sun, Nov 15, 2015 at 1:45 AM, Peter R <peter_r@gmx.com> wrote:
> My previous email described how Type 1 consensus failures can be safely dealt with.  These include many kinds of database exceptions (e.g., the LevelDB fork at block #225,430), or consensus mismatches regarding the max size of a block.

The size of a block is not a type 1 failure. There is a clear, known,
unambiguous, and easily measured rule in the system that a block is
not permitted to have a size over a threshold. A block that violates
this threshold is invalid.   The only way I can see to classify that
as a type 1 failure is to call the violation of any known system rule
a type 1 failure.  "Spends a coin that was already spent" "provides a
signature that doesn't verify with the pubkey" "creates bitcoins out
of thin air" -- these are not logically different than "if size>x
return false".

> Type 2 consensus failures are more severe but also less likely (I’m not aware of a Type 2 consensus failure besides the 92 million bitcoin bug from August 2010).  If Core was to accept a rogue TX that created another 92 million bitcoins, I think it would be a good thing if the other implementations forked away from it (we don’t want bug-for-bug compatibility here).

The only consensus consistency error we've ever that I would have
classified as potentially type 1 had has been the BDB locks issue.
Every other one, including the most recently fixed one (eliminated by
BIP66) was a type 2, by your description. They are _much_ more common;
because if the author understood the possible cases well enough to
create an "I don't know" case, they could have gone one step further
and fully defined it in a consistent way. The fact that an
inconsistency was possible was due to a lack of complete knowledge.

Even in the BDB locks case, I don't know that the type 1 distinction
completely applies, a lower layer piece of software threw an error
that higher layer software didn't know was possible, and so it
interpreted "I don't know" as "no"-- it's not that it chose to treat I
don't know as no, its that it wasn't authored with the possibility of
I don't know in mind, and that exceptions were used to handle general
failures (which should have been treated as no). Once you step back to
the boundary of the calling code (much less the whole application) the
"I don't know" doesn't exist anymore; there.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

> On Nov 14, 2015, at 6:10 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> 
> On Sun, Nov 15, 2015 at 1:45 AM, Peter R <peter_r@gmx.com> wrote:
>> My previous email described how Type 1 consensus failures can be safely dealt with.  These include many kinds of database exceptions (e.g., the LevelDB fork at block #225,430), or consensus mismatches regarding the max size of a block.
> 
> The size of a block is not a type 1 failure. There is a clear, known,
> unambiguous, and easily measured rule in the system that a block is
> not permitted to have a size over a threshold. A block that violates
> this threshold is invalid.  

You are looking at the problem from a “top down” governance perspective assuming you know what code is actually being run and what rules the market wants.  The reality is that you can only make estimates about these two things.  As Bitcoin evolves away from the current development monoculture, rules such as the max block size may no longer be perfectly clear.  However, we can prove that the following results will continue to hold:

1. A node with a block size limit greater than the hash-power weighted median will always follow the longest chain.

2. An excessive (e.g., greater than 1 MB) block will be accepted into the longest chain if it is smaller than the hash-power weighted median block size limit.

Already today, different nodes have different block size limits.  There are even some miners today that will attempt to build upon blocks larger than 1 MB if anyone dares to create one.  At some point, the majority of the hash power will support something larger than 1 MB.  When that first bigger block comes and gets buried in the longest chain, hold-out nodes (e.g., MP says he will never increase his limit) will have to make a choice: fight consensus or track consensus!  I know that I would want my node to give up on its block size limit and track consensus.  You may decide to make a different choice.  

You said that "a block that violates this [block size limit] threshold is invalid.”  I agree.  If the nodes and miners rejected the block as invalid then it would not persist as the longest chain. If the nodes and miners accepted the block and continued to build on top of it, then that chain would be Bitcoin (whether you personally agree of not).  

Bitcoin is ultimately a creature of the market, governed by the code people freely choose to run. Consensus is then an emergent property, objectively represented by the longest persistent chain.  Proof-of-work both enforces and defines the rules of the network.  


>  The only way I can see to classify that
> as a type 1 failure is to call the violation of any known system rule
> a type 1 failure.  "Spends a coin that was already spent" "provides a
> signature that doesn't verify with the pubkey" "creates bitcoins out
> of thin air" -- these are not logically different than "if size>x
> return false”.

I think you’re being intentionally obtuse here: accepting a block composed entirely of valid transactions that is 1.1 MB is entirely different than accepting a TX that creates a ten thousand bitcoins out of thin air.  The market would love the former but abhor the later.  I believe you can recognize the difference.  


>> Type 2 consensus failures are more severe but also less likely (I’m not aware of a Type 2 consensus failure besides the 92 million bitcoin bug from August 2010).  If Core was to accept a rogue TX that created another 92 million bitcoins, I think it would be a good thing if the other implementations forked away from it (we don’t want bug-for-bug compatibility here).
> 
> The only consensus consistency error we've ever that I would have
> classified as potentially type 1 had has been the BDB locks issue.

Thank you for conceding on that point. 

> Every other one, including the most recently fixed one (eliminated by
> BIP66) was a type 2, by your description. They are _much_ more common;
> because if the author understood the possible cases well enough to
> create an "I don't know" case, they could have gone one step further
> and fully defined it in a consistent way. The fact that an
> inconsistency was possible was due to a lack of complete knowledge.
> 
> Even in the BDB locks case, I don't know that the type 1 distinction
> completely applies, a lower layer piece of software threw an error
> that higher layer software didn't know was possible, and so it
> interpreted "I don't know" as "no"-- it's not that it chose to treat I
> don't know as no, its that it wasn't authored with the possibility of
> I don't know in mind, and that exceptions were used to handle general
> failures (which should have been treated as no). Once you step back to
> the boundary of the calling code (much less the whole application) the
> "I don't know" doesn't exist anymore; there.

Please don’t take my comments and observations as criticisms.  I think the Core Dev team has done excellent work!  What I am saying instead is that as we move forward—as development becomes decentralized and multiple protocol implementations emerge—development philosophies will change. Tracking consensus and the will of the market will be most important.  Personally, I hope to see design philosophies that support “bottom up” governance instead of the current “top down” model.  

Best regards,
Peter

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Gregory Maxwell]

On Sun, Nov 15, 2015 at 2:58 AM, Peter R <peter_r@gmx.com> wrote:
> I think you’re being intentionally obtuse here: accepting a block composed entirely of valid transactions that is 1.1 MB is entirely different than accepting a TX that creates a ten thousand bitcoins out of thin air.  The market would love the former but abhor the later.  I believe you can recognize the difference.

It is not technically distinct--today; politically-- perhaps, but--
sorry, no element of your prior message indicated that you were
interested in discussing politics rather than technology; on a mailing
list much more strongly scoped for the latter; I hope you can excuse
me for missing your intention prior to your most recent post.

That said, I believe you are privileging your own political
preferences in seeing the one rule of the bitcoin system as
categorically distinct even politically. No law of nature leaves the
other criteria I specified less politically negotiable, and we can see
concrete examples all around us -- the notion that funds can be
confiscated via external authority (spending without the owners
signature) is a more or less universal property of other modern
systems of money, that economic controls out to exist to regulate the
supply of money for the good of an economy is another widely deployed
political perspective. You, yourself, recently published a work on the
stable self regulation of block sizes based on mining incentives that
took as its starting premise a bitcoin that was forever inflationary.
Certainly things differ in degrees, but this is not the mailing list
to debate the details of political inertia.

> Thank you for conceding on that point.

You're welcome, but I would have preferred that you instead of your
thanks you would have responded in kind and acknowledged my correction
that other consensus inconsistencies discovered in implementations
thus far (none, that I'm aware of) could be classified as "maybe"; and
in doing so retained a semblance of a connection to a the technical
purposes of this mailing list.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

Hi Greg,

>> Thank you for conceding on that point.
> 
> You're welcome, but I would have preferred that you instead of your
> thanks you would have responded in kind and acknowledged my correction
> that other consensus inconsistencies discovered in implementations
> thus far (none, that I'm aware of) could be classified as "maybe"; and
> in doing so retained a semblance of a connection to a the technical
> purposes of this mailing list.

I apologize for that, Greg.  I have not read enough on the issues you brought up to comment intelligibly.  I should have conceded that you could very well be correct that those were Type 2 consensus failures.  

>> I think you’re being intentionally obtuse here: accepting a block composed entirely of valid transactions that is 1.1 MB is entirely different than accepting a TX that creates a ten thousand bitcoins out of thin air.  The market would love the former but abhor the later.  I believe you can recognize the difference.
> 
> It is not technically distinct--today; politically-- perhaps, but--
> sorry, no element of your prior message indicated that you were
> interested in discussing politics rather than technology; on a mailing
> list much more strongly scoped for the latter; I hope you can excuse
> me for missing your intention prior to your most recent post.

The difference between a 1.1 MB block full of valid transactions and an invalid TX that creates 10,000 BTC out of thin air is *not* a matter of “politics.”  If people could freely award themselves coins, then Bitcoin would not be money.  It’s like saying that “technically” there’s no difference between picking up a penny from the sidewalk and holding up a bank teller at gunpoint.  Ask the average person: there is more than a “political” difference between creating coins out of thin air and increasing the block size limit. 
 
> That said, I believe you are privileging your own political
> preferences in seeing the one rule of the bitcoin system as
> categorically distinct even politically.

What rules does Bitcoin obey?  What is Bitcoin’s nature?  This brings us to the age-old debate between Rationalism versus Empiricism.

Rationalism holds that some propositions are known to be true by intuition alone and that others are knowable by being deduced from intuited propositions. The Rationalist may hold the view that Bitcoin has a 21-million coin limit or a 1 MB block size limit, based on deductive reasoning from the rules enforced by the Bitcoin Core source code. Such a Rationalists might believe that the code represents some immutable truth and then his understanding of Bitcoin follows from axiomatic deductions from that premise.

The Empiricist rejects the Rationalist’s intuition and deduction, believing instead that knowledge is necessarily a posteriori, dependent upon observation and sense experience. The Empiricist questions the notion that Bitcoin has a 21-million coin limit, instead observing that its money supply grew by 50 BTC per block for the first 210,000 and then 25 BTC per block ever since. The Empiricist rejects the idea that Bitcoin has any sort of block size limit, having observed previous empirical limits collapse in the face of increased demand.

I am not convinced that Bitcoin even *has* a block size limit, let alone that it can enforce one against the invisible hand of the market.  

> No law of nature leaves the
> other criteria I specified less politically negotiable, and we can see
> concrete examples all around us -- the notion that funds can be
> confiscated via external authority (spending without the owners
> signature) is a more or less universal property of other modern
> systems of money, that economic controls out to exist to regulate the
> supply of money for the good of an economy is another widely deployed
> political perspective. You, yourself, recently published a work on the
> stable self regulation of block sizes based on mining incentives that
> took as its starting premise a bitcoin that was forever inflationary.
> Certainly things differ in degrees, but this is not the mailing list
> to debate the details of political inertia.

You were the one who just brought up politics, Greg.  Not I. 

Best regards,
Peter

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Jorge Timón]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

On Nov 15, 2015 5:10 AM, "Peter R via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> What rules does Bitcoin obey?

Thanks to the worl of many people, part of the consensus rules are finally
encapsulated in the libbitcoinconsensus library. I'm currently writing a
document to complete the encapsulation of the specification of the
consensus rules.

> I am not convinced that Bitcoin even *has* a block size limit, let alone
that it can enforce one against the invisible hand of the market.

You keep insisting that some consensus rules are not consensus rules while
others "are clearly a very different thing". What technical difference is
there between the rule that impedes me from creating transactions bigger
than X and the rules that prevent me frm creatin new coins (not as a miner,
as a regular user in a transaction with more coins in the outputs than in
the inputs)? What about property enforcement? If the invisible hand of the
market is what decides consensus rules instead of their (still incomple)
specification (aka libconsensus), then the market could decide to stop
enforcing ownership.
Will you still think that Bitcoin is a useful system when/if you
empirically observe the invisible hand of the market taking coins out of
your pocket?

You also keep assuming that somehow it is a universal law that users must
eventually converge under the most-work chain. People follow the most-work
VALID chain, but if they consciously decide to implement different rules
(different definitions of "valid block") then their chains can diverge, and
once they do they won't converge again (unless/until one group decides to
implement the rules of the other exactly again), just like when the
implementation of the rules diverge in a unintentional consensus fork. But
in this case they could decide to never implement the same rules.
See bip99 and specially the "schism hardforks" section for more details.

> You were the one who just brought up politics, Greg.  Not I.

Please, read the thread again. I think it is pretty clear that you did.
Nothing wrong with that, just move it to the discussion ml.

[-- Attachment #2: Type: text/html, Size: 2397 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Jorge Timón]

Going back on topic, I believe libconsensus shouldn't depend on any
particular database because assuming it will continue to be stateless
(the current libbitcoinconsensus is stateless) end therefore has no
storage. I know some people disagree in various degrees.
At the same time, the parts of the consensus rules verification that
depends on storage has not been encapsulated out to
libbitcoinconsensus yet, and I agree that changing the database is
unnecessarily risky at this point.
Even when the consensus rules are encapsulated, that doesn't mean that
Bitcoin Core should be DB agnostic or that we can guarantee that it
will follow the longest valid chain with databases that have not been
tested.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

> On Nov 15, 2015, at 3:28 AM, Jorge Timón <jtimon@jtimon.cc> wrote:
> 
> Going back on topic, I believe libconsensus shouldn't depend on any
> particular database because assuming it will continue to be stateless
> (the current libbitcoinconsensus is stateless) end therefore has no
> storage.

Agreed.  

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

[-- Attachment #1: Type: text/plain, Size: 3838 bytes --]

Hello Jorge:
> > What rules does Bitcoin obey? 
> 
> Thanks to the worl of many people, part of the consensus rules are finally encapsulated in the libbitcoinconsensus library. I'm currently writing a document to complete the encapsulation of the specification of the consensus rules.
> 
I applaud your work on the consensus library.  I think it an important step to encouraging multiple competing implementations of the protocol.
> > I am not convinced that Bitcoin even *has* a block size limit, let alone that it can enforce one against the invisible hand of the market.
> 
> You keep insisting that some consensus rules are not consensus rules while others "are clearly a very different thing". What technical difference is there between the rule that impedes me from creating transactions bigger than X and the rules that prevent me frm creatin new coins (not as a miner, as a regular user in a transaction with more coins in the outputs than in the inputs)?
> 
What technical difference is there between a cat and a dog? They both have four legs and a furry coat. 

I think you’re using the term “technical difference” to mean something very specific.  Perhaps you could clarify exactly how you are defining that term because to me it is crystal clear that creating coins out of thin air is very different than accepting a block 1.1 MB in size and full of valid TXs.  There are many technical differences between the two. For example, technically the first allows coins to be created randomly while the second doesn’t.  
> What about property enforcement? If the invisible hand of the market is what decides consensus rules instead of their (still incomple) specification (aka libconsensus), then the market could decide to stop enforcing ownership.
> 
Correct.  Bitcoin is an experiment and could still fail (e.g., the network could allow people to move coins without valid signatures).  For Bitcoin to be viable, the network of miners and node operators being net-econo-rational actually is probably a core axiom that must be accepted.  
> You also keep assuming that somehow it is a universal law that users must eventually converge under the most-work chain. People follow the most-work VALID chain, but if they consciously decide to implement different rules (different definitions of "valid block") then their chains can diverge, and once they do they won't converge again (unless/until one group decides to implement the rules of the other exactly again)
> 
It is fact that two competing forks can persist for at least a short amount of time—we saw this a few years ago with the LevelDB bug and again this summer with the SPV mining incident.  In both cases, there was tremendous pressure to converge back to a single chain.

Could two chains persist indefinitely?  I don’t know.  No one knows.  My gut feeling is that since users would have coins on both sides of the fork, there would be a fork arbitrage event (a “forkbitrage”) where speculators would sell the coins on the side they predict to lose in exchange for additional coins on the side they expect to win.  This could actually be facilitated by exchanges once the fork event is credible and before the fork actually occurs, or even in a futures market somehow.  I suspect that the forkbitrage would create an unstable equilibrium where coins on one side quickly devalue.  Miners would then abandon that side in favour of the other, killing the fork because difficulty would be too high to find new blocks.  Anyways, I think even *this* would be highly unlikely.  I suspect nodes and miners would get inline with consensus as soon as the fork event was credible.  

Cryptocurrency is a new area of interdisciplinary research.  We are all learning together and no one knows how things will unfold.  

Best regards,
Peter


[-- Attachment #2: Type: text/html, Size: 4698 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Tamas Blummer]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

Isolating storage from the rest of consensus code is technically desirable, but implementations using different storage will be unlikely bug-for-bug compatible,
hence able to split the network.

Such split was disastrous on the network level if partitions were of comparable magnitude - as was the case in the March 2013 fork between versions of Bitcoin Core.

This means high level implementation diversity was great, provided we would get there without blowing up the network on the way from monoculture to true decentralization of code.

Libconsensus is immensely valuable to get diversity, as it makes alternate implementations bug-for-bug compatible for a big part of the consensus code.

Tamas Blummer

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Tom Harding]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

On Nov 17, 2015 5:54 AM, "Tamas Blummer via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> Isolating storage from the rest of consensus code is technically
desirable, but implementations using different storage will be unlikely
bug-for-bug compatible,
> hence able to split the network.

The problem with unknown bugs is you don't know how serious they are.  A
serious bug could itself be devastating.

[-- Attachment #2: Type: text/html, Size: 562 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[telemaco]

[-- Attachment #1: Type: text/plain, Size: 3521 bytes --]

Shouldn't a odbc jdbc jconnect or equivalent be totally transparent for the consensus code? I mean, the client would write or store the data communicating to the driver provided by the vendor. Using the schema bitcoin suggests adapted to many different vendors (one table schema for Oracle, other for mysql, etc with their slight syntax particularities), installed in the machine with the node and from that communication to the driver  the storage would be totally controlled by the third party rdbms. 
Regarding bugs or risk of fork, does not have actual client any defense against someone forking core and slightly changing the actual database used maybe wrongly and creating a fork by themselves? 
Does the client have any way to verify that what is stored is correct? Maybe inserting a column with a hash of what is stored in each row and another column with a incremental row by row hash composed by the hash of each row and the previous column one., so any tampering in a previous row can be verified up to where is not consistent.
I just imagine what would be for people to be able to access easily (with the thousands of software packages already bought and licensed by ALL companies in the world that already use open standard connectivity or equivalents)., the bitcoin blockchain. 
SUBSCRIPTION: for a couple decades replication servers have allowed a publish/subscription model using replication agents. If I am a guy working on a lever in the warehouse with my pda I do not need on my pda all the company info or maybe all the blockchain. If a company., that has already licensed a rdbms package with dozens of related software packages needs one guy to suscribe to something on the bitcoin blockchain, he can either use one of the purchased methods in their company and access the company database that holds blockchain data or hire a rare bitcoin developer that will create a interfaz bitcoin for a specific need up to the millions of needs out there. 
PUBLISHING Maybe even to have a publishing daemon that would allow those companies and their software packages to write things in the bitcoin blockchain provided of couse that they fund the agent with a small bitcoin amount to send transactions and they comply with the database constraint of being the owners of the private key. The publishing agent would check for changes every X minutes on that specific address  in the db and if funded it would publish "send" the transaction through the bitcoin client. People would be able to publish info on the decentralized ledger from 90% of enterprise software packages.,paying ofc  and with the small delay of the publishing agent checking for changes. In fact the db would allow publishing info while the publishing agent could just take its time publishing at its own rate like a slow write cache.
In any case shouldn't even actual consensus be shielded from a malfunctioning or Ill forked database from core client

El 17 de noviembre de 2015 16:24:42 CET, Tom Harding <tomh@thinlink.com> escribió:
>On Nov 17, 2015 5:54 AM, "Tamas Blummer via bitcoin-dev" <
>bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> Isolating storage from the rest of consensus code is technically
>desirable, but implementations using different storage will be unlikely
>bug-for-bug compatible,
>> hence able to split the network.
>
>The problem with unknown bugs is you don't know how serious they are. 
>A
>serious bug could itself be devastating.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 3929 bytes --]

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Jorge Timón]

On Tue, Nov 17, 2015 at 11:17 PM, telemaco via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Shouldn't a odbc jdbc jconnect or equivalent be totally transparent for the
> consensus code?

Yes, but we're only testing levelDB and we couldn't assure that it
won't produce unintentional consensus forks with other databases
behind the whatever db-agnostic interface.
I believe Bitcoin Core should officially support only one database at
a time. And if that is to change in the future, I don't think it
should be before a storage-agnostic libconsensus is encapsulated (and
after that there will still be risks and costs in officially
supporting several several databases simultaneously).
As has been said, these kind of experiments are welcomed outside of
bitcoin/master though.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Rusty Russell]

Peter R via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> writes:
> You are looking at the problem from a “top down” governance
> perspective assuming you know what code is actually being run and what
> rules the market wants.

We have strayed far from both the Subject line and from making progress
on bitcoin development.  Please redirect to bitcoin-discuss.

I have set the moderation bits on the three contributors from here down
(CC'd): your next post will go to moderation.

Thanks,
Rusty.

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Luke Dashjr]

On Sunday, November 15, 2015 1:02:33 AM Peter R via bitcoin-dev wrote:
> A group of us have been exploring this “meta-cognition” idea with Bitcoin
> Unlimited.  For example, Bitcoin Unlimited can be (optionally) made to
> automatically fork to the longest chain if it “gets stuck” and can neither
> prove that a block is valid nor that the block is invalid.

This situation isn't something that can be ignored and simply moved past. If 
you can't determine the validity of a block, you also cannot process its 
results correctly. Taking for example the BDB/LevelDB issue, the result was 
that BDB failed to accept further changes to the UTXO set. Unless the UTXO set 
could be updated correctly, there is no way to even attempt to validate the 
next block or any new transactions.

Luke

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Peter R]

> On Sunday, November 15, 2015 1:02:33 AM Peter R via bitcoin-dev wrote:
>> A group of us have been exploring this “meta-cognition” idea with Bitcoin
>> Unlimited.  For example, Bitcoin Unlimited can be (optionally) made to
>> automatically fork to the longest chain if it “gets stuck” and can neither
>> prove that a block is valid nor that the block is invalid.
> 
> This situation isn't something that can be ignored and simply moved past. If 
> you can't determine the validity of a block, you also cannot process its 
> results correctly. Taking for example the BDB/LevelDB issue, the result was 
> that BDB failed to accept further changes to the UTXO set. Unless the UTXO set 
> could be updated correctly, there is no way to even attempt to validate the 
> next block or any new transactions.

Great point, Luke! 

Indeed, whether the program can or cannot continue after a Type 1 consensus mismatch depends on the specifics of the situation and exactly how the code was written.  But I agree: there are cases where the program *can’t* continue.  In those cases it would halt.  This would require manual intervention to fix but avoids the problem of potential double-spends during the fork event.  This would be preferable to knowingly causing a fork.  

Peter

Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db

[Gregory Maxwell]

On Thu, Oct 29, 2015 at 6:57 AM, telemaco via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Why not "outsource" totally that data management part to the already
> existing with decades of experience database world. People would be able to
> create incredibly easy bitcoin statistics/graphs/analisys with existing
> software packages (hey even excel or libreoffice like) or connect bitcoin
> data to their own sources and if so they chose analyze bitcoin data on a
> datawarehouse or any imaginable approach. Of course every transaction would
> be have to do through the bitcoin node and only the data management would be
> on rdbms side.

The word "database" is likely confusing people here.  This is not a
database in an ordinary sense.

The bitcoin core consensus engine requires a highly optimized ultra
compact data structure to perform the lookups for coin existence. The
data stored is highly compressed and very specialized, it would not be
useful to other applications.  Right now, on boring laptop hardware,
during network synchronization updates to this database run at over
10,000 records per second, while the system is also busy doing the
other validation chores of a node. This is backended by a high
performance transactional key value store.  The need for performance
here is essential to even keeping up with the network, it's not about
enabling any kind of fancy querying (bitcoin core does not offer fancy
querying), it's about the base load that every node must handle to
usably sync up and keep up with the Bitcoin network.

The backend can be swapped out for something else that provides the
same properties, but doing so does not give you any of the
inspection/analytics that you're looking for.  Systems that do that
exist, and they require databases taking hundreds of gigabytes of
storage and take days to weeks to import the network data.  They're
great for what they're for, but they're not suitable for consensus use
in the system for space efficiency, performance, and consensus
consistency reasons.