[Bitcoin-development] Faster databases than LevelDB

[Bitcoin-development] Faster databases than LevelDB

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

LevelDB is fast - very fast if you give it enough CPU time and disk seeks.
But it's not the last word in performance.

HyperLevelDB is a forked LevelDB with some changes, mostly, finer grained
locking and changes to how compaction works:

http://hyperdex.org/performance/leveldb/

However, it comes with a caveat - one of the changes they made is to take
away write throttling if compaction falls behind, the app itself is
expected to do that.

Sophia is a competitor to LevelDB. The website claims that in benchmarks it
completely smokes LevelDB. I have not explored how it does this or tried to
replicate their benchmarks myself:

http://sphia.org/index.html
http://sphia.org/benchmarks.html

It's written in C and BSD licensed.

As an example of the kind of speedup they claim to be capable of, they say
LevelDB could do 167,476 random reads per second on their SSD based
machine. Sophia could do 438,084 reads/sec. Random reads are of course the
most interesting for us because that's what UTXO lookups involve.

They also compare against HyperLevelDB, where the differences are much less
pronounced and actually HyperLevelDB appears to be able to do random writes
faster than Sophia.

[-- Attachment #2: Type: text/html, Size: 1623 bytes --]

Re: [Bitcoin-development] Faster databases than LevelDB

[Jorge Timón]

Only slightly related to this...
What's the reason why BerkleyDB is maintained for the wallet?
I think it would be a good thing to get rid of the libdb4.8++-dev
dependency that makes bitcoind harder to compile on debian and ubuntu.
Unless, of course, there's a reason I am missing...


On 9/17/13, Mike Hearn <mike@plan99.net> wrote:
> LevelDB is fast - very fast if you give it enough CPU time and disk seeks.
> But it's not the last word in performance.
>
> HyperLevelDB is a forked LevelDB with some changes, mostly, finer grained
> locking and changes to how compaction works:
>
> http://hyperdex.org/performance/leveldb/
>
> However, it comes with a caveat - one of the changes they made is to take
> away write throttling if compaction falls behind, the app itself is
> expected to do that.
>
> Sophia is a competitor to LevelDB. The website claims that in benchmarks it
> completely smokes LevelDB. I have not explored how it does this or tried to
> replicate their benchmarks myself:
>
> http://sphia.org/index.html
> http://sphia.org/benchmarks.html
>
> It's written in C and BSD licensed.
>
> As an example of the kind of speedup they claim to be capable of, they say
> LevelDB could do 167,476 random reads per second on their SSD based
> machine. Sophia could do 438,084 reads/sec. Random reads are of course the
> most interesting for us because that's what UTXO lookups involve.
>
> They also compare against HyperLevelDB, where the differences are much less
> pronounced and actually HyperLevelDB appears to be able to do random writes
> faster than Sophia.
>


-- 
Jorge Timón

http://freico.in/

Re: [Bitcoin-development] Faster databases than LevelDB

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1919 bytes --]

Nobody has written code to use a better format, migrate old wallets, etc.


On Tue, Sep 17, 2013 at 1:41 PM, Jorge Timón <jtimon@monetize.io> wrote:

> Only slightly related to this...
> What's the reason why BerkleyDB is maintained for the wallet?
> I think it would be a good thing to get rid of the libdb4.8++-dev
> dependency that makes bitcoind harder to compile on debian and ubuntu.
> Unless, of course, there's a reason I am missing...
>
>
> On 9/17/13, Mike Hearn <mike@plan99.net> wrote:
> > LevelDB is fast - very fast if you give it enough CPU time and disk
> seeks.
> > But it's not the last word in performance.
> >
> > HyperLevelDB is a forked LevelDB with some changes, mostly, finer grained
> > locking and changes to how compaction works:
> >
> > http://hyperdex.org/performance/leveldb/
> >
> > However, it comes with a caveat - one of the changes they made is to take
> > away write throttling if compaction falls behind, the app itself is
> > expected to do that.
> >
> > Sophia is a competitor to LevelDB. The website claims that in benchmarks
> it
> > completely smokes LevelDB. I have not explored how it does this or tried
> to
> > replicate their benchmarks myself:
> >
> > http://sphia.org/index.html
> > http://sphia.org/benchmarks.html
> >
> > It's written in C and BSD licensed.
> >
> > As an example of the kind of speedup they claim to be capable of, they
> say
> > LevelDB could do 167,476 random reads per second on their SSD based
> > machine. Sophia could do 438,084 reads/sec. Random reads are of course
> the
> > most interesting for us because that's what UTXO lookups involve.
> >
> > They also compare against HyperLevelDB, where the differences are much
> less
> > pronounced and actually HyperLevelDB appears to be able to do random
> writes
> > faster than Sophia.
> >
>
>
> --
> Jorge Timón
>
> http://freico.in/
>

[-- Attachment #2: Type: text/html, Size: 2779 bytes --]

Re: [Bitcoin-development] Faster databases than LevelDB

[Gregory Maxwell]

On Tue, Sep 17, 2013 at 4:00 AM, Mike Hearn <mike@plan99.net> wrote:
> LevelDB is fast - very fast if you give it enough CPU time and disk seeks.
> But it's not the last word in performance.

I'd looked at the hyperleveldb, but their performance graphs made it
seem like it would be slower for the actual database sizes we're using
today.

Is there a competitor that specializes in being more robust to corruption? :(

Re: [Bitcoin-development] Faster databases than LevelDB

[Jorge Timón]

On 9/17/13, Mike Hearn <mike@plan99.net> wrote:
> Nobody has written code to use a better format, migrate old wallets, etc.

ACK, thanks.

Re: [Bitcoin-development] Faster databases than LevelDB

[Mark Friedenbach]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Also somewhat related, I have been looking for some time now to
abstract out the UTXO and block databases so that a variety of
key/value stores could be used as a backend, configured by a command
line parameter. In particular, it would be interesting for some server
applications to support HyperDex, which is basically a distributed,
fault-tolerant version of LevelDB:

http://hyperdex.org/

By the same mechanism you could just as easily support a Sophia backend.

Mark


On 9/17/13 4:00 AM, Mike Hearn wrote:
> LevelDB is fast - very fast if you give it enough CPU time and
> disk seeks. But it's not the last word in performance.
> 
> HyperLevelDB is a forked LevelDB with some changes, mostly, finer 
> grained locking and changes to how compaction works:
> 
> http://hyperdex.org/performance/leveldb/
> 
> However, it comes with a caveat - one of the changes they made is
> to take away write throttling if compaction falls behind, the app
> itself is expected to do that.
> 
> Sophia is a competitor to LevelDB. The website claims that in
> benchmarks it completely smokes LevelDB. I have not explored how it
> does this or tried to replicate their benchmarks myself:
> 
> http://sphia.org/index.html http://sphia.org/benchmarks.html
> 
> It's written in C and BSD licensed.
> 
> As an example of the kind of speedup they claim to be capable of,
> they say LevelDB could do 167,476 random reads per second on their
> SSD based machine. Sophia could do 438,084 reads/sec. Random reads
> are of course the most interesting for us because that's what UTXO
> lookups involve.
> 
> They also compare against HyperLevelDB, where the differences are
> much less pronounced and actually HyperLevelDB appears to be able
> to do random writes faster than Sophia.
> 
> 
> 
> 
> ------------------------------------------------------------------------------
>
> 
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
> 1,500+ hours of tutorials including VisualStudio 2012, Windows 8,
> SharePoint 2013, SQL 2012, MVC 4, more. BEST VALUE: New
> Multi-Library Power Pack includes Mobile, Cloud, Java, and UX
> Design. Lowest price ever! Ends 9/20/13. 
> http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
>
> 
> 
> 
> _______________________________________________ Bitcoin-development
> mailing list Bitcoin-development@lists.sourceforge.net 
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSOIymAAoJEAdzVfsmodw4H48QALC+ae4wRLEg3lrg9sgayfOn
ukLM079PXgEbARFPt6WxkLnNGYzEbb7IzT0uvaKH4VIW/rrORy9VqNPmliF+834h
XygUwfAzU04K/oLyCsdWZcOugj2P8aufNeA6whLS5IijDLtHb3Ueu4ORNcfLBGqp
KKfqPj0QHseusiLJ9f3IW+LrdM1vAoT1jryTngpQy2i+qFFDM6CN3THCq4adJvjr
AnYlfLoJSZ0/obz/krwLv6vP1BbwxXzv5CfD0Q2bdoEV/EgWDP3Bd5tUzUCjj53/
qMmhaACoVlarohh64s3JNSDSkHDFSbHFt65ZgNQbNY1wmSeyilQcd8FGWOF/WRzW
Z/pl2IdhoCm3t86xSggRGivj/EVeBJlD36i7ohpDbVWFPsf6B4e5M6xSdso/2WBp
fr55TwehCaGE+UHa0gITkE/si1txvY4gti0bLNvwFDEcZ3qsXRsz4CyLlZLMBbPX
4aRNGyqv2yJ2AivkEyNOUugo1Q8RKEKZWfWWDecI53DHdebzKX1zu9GLJwlGJqGw
Qzm7Tdb7S8J/D6IIHf4Xq2LDhQ2fnPylmGSmtuVFEMxeDhmdbNqKSr3kqlWQf3T8
Oa8bm6kUQFJ+11jLEkVEGZJC4e42+faQBxR+CsqvVsTEezDCP1dE7D3QV8ry9YBc
DwXt3299Q03B5LoxpWTq
=KseH
-----END PGP SIGNATURE-----