[Bitcoin-development] Scalability issues

[Bitcoin-development] Scalability issues

[grarpamp]

Given a testbed: Pentium 4 1.8GHz single core, 2GB ram, FreeBSD 8,
disk is geli aes-128 + zfs sha-256, bitcoin 0.6.3, Tor proxy,
An estimate is made that by the end of the year bitcoin will
completely overrun the capabilities of this reasonable class of
machines.
It already takes a month to build a new blockchain, let alone keep up
with new incoming blocks.
Yes, it also has workstation duties, yet even if those were removed,
it would probably choke by mid 2013.

It would appear bitcoin has some *serious* scalability hurdles coming
down the road.
Most certainly if the user expects to independantly build, manage, and
trust their own blockchain.

Re: [Bitcoin-development] Scalability issues

[Raphael NICOLLE]

Hello,

Even though I'm not a dev, I can't agree more, and would like to know if 
they are expected steps being taken, some fixes coming, or whatever?

Thank you all for your hard work.
Raphael

On 07/23/2012 12:37 AM, grarpamp wrote:
> Given a testbed: Pentium 4 1.8GHz single core, 2GB ram, FreeBSD 8,
> disk is geli aes-128 + zfs sha-256, bitcoin 0.6.3, Tor proxy,
> An estimate is made that by the end of the year bitcoin will
> completely overrun the capabilities of this reasonable class of
> machines.
> It already takes a month to build a new blockchain, let alone keep up
> with new incoming blocks.
> Yes, it also has workstation duties, yet even if those were removed,
> it would probably choke by mid 2013.
>
> It would appear bitcoin has some *serious* scalability hurdles coming
> down the road.
> Most certainly if the user expects to independantly build, manage, and
> trust their own blockchain.
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Scalability issues

[Gregory Maxwell]

On Sun, Jul 22, 2012 at 6:37 PM, grarpamp <grarpamp@gmail.com> wrote:
> It already takes a month to build a new blockchain, let alone keep up
> with new incoming blocks.

Please fix your software stack. Something is wrong with your system
and I doubt it has much to do with bitcoin. A full sync here takes
something like an hour.

There are certainly lots of scalability things going on, but there is
no cause for concern for regular hardware being unable to _keep up_
without a hardforking change to the protocol first.

Re: [Bitcoin-development] Scalability issues

[Michael Grønager]

I would guess that you are running the blockchain download through the tor-proxy - that would give you the times you mention. Further, encrypting your disk (aes stuff) will not help you much either, and encrypting a the storage of a public blockchain seems to me a bit odd ?

I get a full blockchain from scratch in 45 minutes on my laptop, but, I still agree with Gregory that scalability improvements are needed, but the problem is far from critical in the sense outlined here.

/M 


On 23/07/2012, at 09:35, Gregory Maxwell wrote:

> On Sun, Jul 22, 2012 at 6:37 PM, grarpamp <grarpamp@gmail.com> wrote:
>> It already takes a month to build a new blockchain, let alone keep up
>> with new incoming blocks.
> 
> Please fix your software stack. Something is wrong with your system
> and I doubt it has much to do with bitcoin. A full sync here takes
> something like an hour.
> 
> There are certainly lots of scalability things going on, but there is
> no cause for concern for regular hardware being unable to _keep up_
> without a hardforking change to the protocol first.
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Scalability issues

[grarpamp]

> Please fix your software stack. Something is wrong
> with your system

Nothing wrong, it's all default install. I documented the platform
for anyone who wants to confirm it.

> A full sync here takes something like an hour.

And what, similarly, is your platform?
It takes 5 seconds... on my Cray.

> I would guess that you are running the blockchain download
> through the tor-proxy

Use of Tor was stated. Tor is fast enough. I can copy the entire
3GiB of the .bitcoin dir in 7 days... off a slow hidden service.
And 0.5 days via exit.

> encrypting your disk (aes stuff) will not help you much either

Encryption is a perfectly reasonable thing to expect users of
bitcoin to be interested in doing. In fact, those not encrypting
their disks should probably rethink that plan.

> encrypting a the storage of a public blockchain seems to me a bit odd ?

Well, without detachdb, it's somehow tied to the wallet, whether
while processing or offline. And the wallet and debug.log are
not relocatable from the data. And encrypting everything is perfectly
reasonable anyways. As is storing your valuable data on filesystems
that verify the integrity of their data on disk, such as ZFS/BTRFS.

These days, crypto, Tor, and ZFS are common and non-arguments.

> I get a full blockchain from scratch in 45 minutes on my laptop

Again, timings with no CPU/OS/disk specs are useless infos.

Re: [Bitcoin-development] Scalability issues

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Michael,

On 23/07/2012 10:00, Michael Grønager wrote:
> I get a full blockchain from scratch in 45 minutes on my laptop,
> /M
> 
Hang on a sec, in 45 minutes you can download the entire chain from
the genesis block?

I have been doing extensive testing in this area and would love to
know what is special about your setup (I have never had the entire
chain in under 12 hours, infact it is normally closerto 24.) I have an
extensive setup of test machines, everything from e4300 to phenom2x6
to i5's.

as an example on an amd e-450 with 4gb ram, and approx 3gb/s internet
connection it took 2 hours to sync the last 5 days.

Maybe i am missing something important...

Any additional information that you could provide to help me with
testing would be really appreciated.

cheers,

steve

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQDXO4AAoJEFvEB9dQFvtQxdcH/ieqQkyDCg8mKeOa6CqsWaS6
fhoeny3Ke2b/CsvhYmsThCvntN9volIqR2CTn5tkHiVwG9OmlxyHZcNpN0ZTHhK5
lsfLap/Y0QpiysXpV4Bu7Z4Hwp9jnhOP74TshT305r2pX6EGXPQ0CrlHqlIry/X/
vNcunUclliou+KjL7EHcY50GH5wDpqJAjlNyF97Lj9YiPrAC9vahGwWdxkbCYtG+
KUuWGBKMMdHuMAgcQh7nI9q0WT3k/gzRQtuC2kf+v0wvQhaGlTVkku4uanhpuw4p
99blRF3/SfWimGuQgsm6wT3Y7dk+z8MFHLb6XGPxmgV9+gF+TWNczfU3GRzfcXw=
=CQkI
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Scalability issues

[grarpamp]

> You're seriously suggesting that I'm using a system
> which is 720x (one month vs one hour) faster than your
> P4 1.8GHz?

Don't know what you're using since you've not stated it.

> I find this doubtful, especially since bitcoin's sync is effectively
> single threaded.

Extra cores help with disk, crypto, net, etc...

> month

I've spent about two weeks crunching about the last month's
worth of new blocks.

> Your results are not expected and are not believed to be
> representative.

The config is reproducible, and not believed to be uncommon.

> try to isolate it

Mostly disk and crypto.
Shall everyone instead run in bitrot and no-privacy mode?
What do we do when we've got 10k trans a day coming in?
50k? 100k, 1M? When the chain gets 1M, 50M, 500M, 1B long?

Forget my swamped box, these numbers are coming to others.

> try to sync from a local node into tmpfs

I'd bet some people using 'tmpfs' probably have it unknowingly
[fall]backed to swap instead of core.

Bitcoin already takes up 3GiB of disk, how many have that much
free RAM? How many have 30GiB, 300GiB?

> If you're building against BDB later than the recommended 4.8
> be aware that there have been performance regressions with later
> versions

Yes, I left out that bit of platform so here are the remaining
bits... db4830, boost149, vm.kmem_size=650000000


I'm not bashing anything or anybody, just detailing a stock config
that is underwater. Anybody wishing to verify can get the hardware
from their junk pile and the software from freebsd.org. I'll certainly
be looking at both it and different setups too. If anyone's using
say Linux/BSD, BTRFS/ZFS, crypto, on i386/amd64, they could
chime in with their times too.

Disk is the cheapest, easiest thing for Joe to get. Think about
indexing and checkpointing into said disk. I don't know what
bitcoin's doing, but if it's verifying every transaction back to
the root, that would seem a bit ridiculous.

Joe probably won't be happy buying TiB's for bitcoind, so after that's
filled (assuming there's CPU to do it), the trust model has to change.
These scales are coming...

Re: [Bitcoin-development] Scalability issues

[Aidan Thornton]

On Sun, Jul 22, 2012 at 11:37 PM, grarpamp <grarpamp@gmail.com> wrote:
> Given a testbed: Pentium 4 1.8GHz single core, 2GB ram, FreeBSD 8,
> disk is geli aes-128 + zfs sha-256, bitcoin 0.6.3, Tor proxy,
> An estimate is made that by the end of the year bitcoin will
> completely overrun the capabilities of this reasonable class of
> machines.

That's actually slower-than-Atom class hardware you're running on
there, with full disk encryption on what's a very CPU and IO-heavy
workload. From the benchmarks I remember, your CPU is literally slower
than the first generation of single-core netbooks - the ones reviewers
recommended against using full disk encryption on because they just
didn't have enough CPU power to manage it. It's not surprising that
Bitcoin isn't usable on that setup.

Re: [Bitcoin-development] Scalability issues

[Michael Grønager]

Hi Steve,

45-90 minutes - note that its numbers from March/April, so a bit longer today, but far, far away from the 12 hours. 

I am using libcoin and the bitcoind build based on this. Libcoin is based on the Satoshi client, but refactured to use an async concurrency model. I also did a minor tweeks to the db parameters. It has earlier been tested up against Satoshi bitcoin where on some OS'es it performs similarly (at least on some linuxes) and on some faster (e.g. mac). 

What is your CPU load during a block download ? (both initially/up to the point where verification sets in and after). The initial download is typically disk I/O bound, the verification stage CPU bound, though I lean to believe that even there it is disk I/O bound (at least on my system ~50% CPU load). What should be better in libcoin is the concurrency model. The Satoshi client uses a pure reentrant mutexes model, that is not generally believed to motivate the best coding practice nor performance, you might end up without the concurrency you initially strived for *). As mentioned earlier libcoin uses a pure async concurrency model (and so does libbitcoin btw). 

I would like to stress again that these numbers will depend largely on the system running the test - I would call my laptop a bit over the average today (MB Pro, 2.66Ghz i7 dual core, 8GBRAM, 512GB SSD). But again 12 hours - I only reach such numbers on some of my VPS'es (linode 1024) that are known for notoriously slow disk I/O. (here I have a few % CPU load during the verification indicating indeed that the disk i/o is the culprit).

Cheers,

Michael


*) I like this Dave Butenhof quote: "The biggest of all the big problems with recursive mutexes is that they encourage you to completely lose track of your locking scheme and scope. This is deadly. Evil. It's the "thread eater". You hold locks for the absolutely shortest possible time. Period. Always. If you're calling something with a lock held simply because you don't know it's held, or because you don't know whether the callee needs the mutex, then you're holding it too long. You're aiming a shotgun at your application and pulling the trigger. You presumably started using threads to get concurrency; but you've just PREVENTED concurrency."




On 23/07/2012, at 17:54, steve wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi Michael,
> 
> On 23/07/2012 10:00, Michael Grønager wrote:
>> I get a full blockchain from scratch in 45 minutes on my laptop,
>> /M
>> 
> Hang on a sec, in 45 minutes you can download the entire chain from
> the genesis block?
> 
> I have been doing extensive testing in this area and would love to
> know what is special about your setup (I have never had the entire
> chain in under 12 hours, infact it is normally closerto 24.) I have an
> extensive setup of test machines, everything from e4300 to phenom2x6
> to i5's.
> 
> as an example on an amd e-450 with 4gb ram, and approx 3gb/s internet
> connection it took 2 hours to sync the last 5 days.
> 
> Maybe i am missing something important...
> 
> Any additional information that you could provide to help me with
> testing would be really appreciated.
> 
> cheers,
> 
> steve
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQEcBAEBAgAGBQJQDXO4AAoJEFvEB9dQFvtQxdcH/ieqQkyDCg8mKeOa6CqsWaS6
> fhoeny3Ke2b/CsvhYmsThCvntN9volIqR2CTn5tkHiVwG9OmlxyHZcNpN0ZTHhK5
> lsfLap/Y0QpiysXpV4Bu7Z4Hwp9jnhOP74TshT305r2pX6EGXPQ0CrlHqlIry/X/
> vNcunUclliou+KjL7EHcY50GH5wDpqJAjlNyF97Lj9YiPrAC9vahGwWdxkbCYtG+
> KUuWGBKMMdHuMAgcQh7nI9q0WT3k/gzRQtuC2kf+v0wvQhaGlTVkku4uanhpuw4p
> 99blRF3/SfWimGuQgsm6wT3Y7dk+z8MFHLb6XGPxmgV9+gF+TWNczfU3GRzfcXw=
> =CQkI
> -----END PGP SIGNATURE-----
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Scalability issues

[Mike Hearn]

> The Satoshi client uses a pure reentrant mutexes model

As you presumably already know, the reference client doesn't attempt
to parallelise most operations at all. Chain download is entirely
single threaded.

Re: [Bitcoin-development] Scalability issues

[steve]

Hi Michael,

from what I have noticed, bitcoin blockchain download/verfication all
happens in 1 thread.  (so multicores doesnt really help)

That said, I have never tried on an ssd.

What I do have is 6 SATA 6gbs configed as RAID0 Drives.
32gb of ram. ubuntu 64 (yeah I know), this runs upto 16 VM's
(I have 4 of these)

However I have not tried to download the blockchain on the master os,
just in virtulisation.  However, the dedicated machines that I have been
using for benchmarking the VM's against is a q6600 8gb ram sata2 hdd -
Win 7 (seems faster than slackware...) to me it has always felt like
network bandwidth was the issue.  I might instrument the bitcoin-qt exe
to only pick low ping nodes (has someone already done this?)

I guess it is time to start some benchmarking (like the gpu comparison page)

hte verification for the 5 past 5 days was negliglable. I am off on a
flight to australia tomorrrow, so I will set some breakpoints and do
some timings in a debugger.

This will all happen on an e-450 (wonderful machine!)

Thanks very much for your response. it would seem that I am 'doing it
wrong' :/

cheers mate,

steve

(this message isnt signed because I have forgotten my password.)

On 24/07/2012 09:25, Michael Grønager wrote:
> Hi Steve,
> 
> 45-90 minutes - note that its numbers from March/April, so a bit
> longer today, but far, far away from the 12 hours.
> 
> I am using libcoin and the bitcoind build based on this. Libcoin is
> based on the Satoshi client, but refactured to use an async
> concurrency model. I also did a minor tweeks to the db parameters. It
> has earlier been tested up against Satoshi bitcoin where on some
> OS'es it performs similarly (at least on some linuxes) and on some
> faster (e.g. mac).
> 
> What is your CPU load during a block download ? (both initially/up to
> the point where verification sets in and after). The initial download
> is typically disk I/O bound, the verification stage CPU bound, though
> I lean to believe that even there it is disk I/O bound (at least on
> my system ~50% CPU load). What should be better in libcoin is the
> concurrency model. The Satoshi client uses a pure reentrant mutexes
> model, that is not generally believed to motivate the best coding
> practice nor performance, you might end up without the concurrency
> you initially strived for *). As mentioned earlier libcoin uses a
> pure async concurrency model (and so does libbitcoin btw).
> 
> I would like to stress again that these numbers will depend largely
> on the system running the test - I would call my laptop a bit over
> the average today (MB Pro, 2.66Ghz i7 dual core, 8GBRAM, 512GB SSD).
> But again 12 hours - I only reach such numbers on some of my VPS'es
> (linode 1024) that are known for notoriously slow disk I/O. (here I
> have a few % CPU load during the verification indicating indeed that
> the disk i/o is the culprit).
> 
> Cheers,
> 
> Michael
> 
> 
> *) I like this Dave Butenhof quote: "The biggest of all the big
> problems with recursive mutexes is that they encourage you to
> completely lose track of your locking scheme and scope. This is
> deadly. Evil. It's the "thread eater". You hold locks for the
> absolutely shortest possible time. Period. Always. If you're calling
> something with a lock held simply because you don't know it's held,
> or because you don't know whether the callee needs the mutex, then
> you're holding it too long. You're aiming a shotgun at your
> application and pulling the trigger. You presumably started using
> threads to get concurrency; but you've just PREVENTED concurrency."
> 
> 
> 
> 
> On 23/07/2012, at 17:54, steve wrote:
> 
> Hi Michael,
> 
> On 23/07/2012 10:00, Michael Grønager wrote:
>>>> I get a full blockchain from scratch in 45 minutes on my
>>>> laptop, /M
>>>> 
> Hang on a sec, in 45 minutes you can download the entire chain from 
> the genesis block?
> 
> I have been doing extensive testing in this area and would love to 
> know what is special about your setup (I have never had the entire 
> chain in under 12 hours, infact it is normally closerto 24.) I have
> an extensive setup of test machines, everything from e4300 to
> phenom2x6 to i5's.
> 
> as an example on an amd e-450 with 4gb ram, and approx 3gb/s
> internet connection it took 2 hours to sync the last 5 days.
> 
> Maybe i am missing something important...
> 
> Any additional information that you could provide to help me with 
> testing would be really appreciated.
> 
> cheers,
> 
> steve
> 
>> 
>> ------------------------------------------------------------------------------
>>
>> 
Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and 
>> threat landscape has changed and how IT managers can respond.
>> Discussions will include endpoint security, mobile security and the
>> latest in malware threats.
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ 
>> _______________________________________________ Bitcoin-development
>> mailing list Bitcoin-development@lists.sourceforge.net 
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 
> 
> 
> 
> 

Re: [Bitcoin-development] Scalability issues

[Michael Grønager]

Hi Steve,

I see dramatic differences in performance on virtual machines vs running directly on the iron. I am not an expert in virtual machines, but it seems to me that they are weak when it comes to disk i/o. And berkeley DB, as used by bitcoin is a sucker for disk i/o. In top I easily hit >1/#processors in top wa, meaning that the cpu doing the blockchain download is just waiting for the disk all the time.

I would like to do a test keeping database log files in memory. It should not matter for durability of the wallet, as it flushes at each write anyway. As for the blockindex, it will remain consistent, but might be lagging some blocks behind at startup, which shouldn't really matter (except that the same block could end up appearing twice in the block00X files, inelegant, but not really a problem).

Otherwise the system you describe (raid0 over 6 disks) should perform like crazy wrt disk i/o, at least on par with SSD. It is your virtualization I am worried about.

Have a safe trip to down under!

/M

On 24/07/2012, at 21:56, steve wrote:

> Hi Michael,
> 
> from what I have noticed, bitcoin blockchain download/verfication all
> happens in 1 thread.  (so multicores doesnt really help)
> 
> That said, I have never tried on an ssd.
> 
> What I do have is 6 SATA 6gbs configed as RAID0 Drives.
> 32gb of ram. ubuntu 64 (yeah I know), this runs upto 16 VM's
> (I have 4 of these)
> 
> However I have not tried to download the blockchain on the master os,
> just in virtulisation.  However, the dedicated machines that I have been
> using for benchmarking the VM's against is a q6600 8gb ram sata2 hdd -
> Win 7 (seems faster than slackware...) to me it has always felt like
> network bandwidth was the issue.  I might instrument the bitcoin-qt exe
> to only pick low ping nodes (has someone already done this?)
> 
> I guess it is time to start some benchmarking (like the gpu comparison page)
> 
> hte verification for the 5 past 5 days was negliglable. I am off on a
> flight to australia tomorrrow, so I will set some breakpoints and do
> some timings in a debugger.
> 
> This will all happen on an e-450 (wonderful machine!)
> 
> Thanks very much for your response. it would seem that I am 'doing it
> wrong' :/
> 
> cheers mate,
> 
> steve
> 
> (this message isnt signed because I have forgotten my password.)
> 
> On 24/07/2012 09:25, Michael Grønager wrote:
>> Hi Steve,
>> 
>> 45-90 minutes - note that its numbers from March/April, so a bit
>> longer today, but far, far away from the 12 hours.
>> 
>> I am using libcoin and the bitcoind build based on this. Libcoin is
>> based on the Satoshi client, but refactured to use an async
>> concurrency model. I also did a minor tweeks to the db parameters. It
>> has earlier been tested up against Satoshi bitcoin where on some
>> OS'es it performs similarly (at least on some linuxes) and on some
>> faster (e.g. mac).
>> 
>> What is your CPU load during a block download ? (both initially/up to
>> the point where verification sets in and after). The initial download
>> is typically disk I/O bound, the verification stage CPU bound, though
>> I lean to believe that even there it is disk I/O bound (at least on
>> my system ~50% CPU load). What should be better in libcoin is the
>> concurrency model. The Satoshi client uses a pure reentrant mutexes
>> model, that is not generally believed to motivate the best coding
>> practice nor performance, you might end up without the concurrency
>> you initially strived for *). As mentioned earlier libcoin uses a
>> pure async concurrency model (and so does libbitcoin btw).
>> 
>> I would like to stress again that these numbers will depend largely
>> on the system running the test - I would call my laptop a bit over
>> the average today (MB Pro, 2.66Ghz i7 dual core, 8GBRAM, 512GB SSD).
>> But again 12 hours - I only reach such numbers on some of my VPS'es
>> (linode 1024) that are known for notoriously slow disk I/O. (here I
>> have a few % CPU load during the verification indicating indeed that
>> the disk i/o is the culprit).
>> 
>> Cheers,
>> 
>> Michael
>> 
>> 
>> *) I like this Dave Butenhof quote: "The biggest of all the big
>> problems with recursive mutexes is that they encourage you to
>> completely lose track of your locking scheme and scope. This is
>> deadly. Evil. It's the "thread eater". You hold locks for the
>> absolutely shortest possible time. Period. Always. If you're calling
>> something with a lock held simply because you don't know it's held,
>> or because you don't know whether the callee needs the mutex, then
>> you're holding it too long. You're aiming a shotgun at your
>> application and pulling the trigger. You presumably started using
>> threads to get concurrency; but you've just PREVENTED concurrency."
>> 
>> 
>> 
>> 
>> On 23/07/2012, at 17:54, steve wrote:
>> 
>> Hi Michael,
>> 
>> On 23/07/2012 10:00, Michael Grønager wrote:
>>>>> I get a full blockchain from scratch in 45 minutes on my
>>>>> laptop, /M
>>>>> 
>> Hang on a sec, in 45 minutes you can download the entire chain from 
>> the genesis block?
>> 
>> I have been doing extensive testing in this area and would love to 
>> know what is special about your setup (I have never had the entire 
>> chain in under 12 hours, infact it is normally closerto 24.) I have
>> an extensive setup of test machines, everything from e4300 to
>> phenom2x6 to i5's.
>> 
>> as an example on an amd e-450 with 4gb ram, and approx 3gb/s
>> internet connection it took 2 hours to sync the last 5 days.
>> 
>> Maybe i am missing something important...
>> 
>> Any additional information that you could provide to help me with 
>> testing would be really appreciated.
>> 
>> cheers,
>> 
>> steve
>> 
>>> 
>>> ------------------------------------------------------------------------------
>>> 
>>> 
> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and 
>>> threat landscape has changed and how IT managers can respond.
>>> Discussions will include endpoint security, mobile security and the
>>> latest in malware threats.
>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ 
>>> _______________________________________________ Bitcoin-development
>>> mailing list Bitcoin-development@lists.sourceforge.net 
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>> 
>> 
>> 
>> 
>> 
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Scalability issues

[steve]

On 25/07/2012 10:45, Michael Grønager wrote:
> Hi Steve,
> 
> I see dramatic differences in performance on virtual machines vs 
> running directly on the iron. I am not an expert in virtual
> machines,

They can be, it depends on how they are set up.  For reference, these
VM's used to test network stacks and file format bugs.  They do this
via debug tracing, trace into, not over.  They then dump this data in
files and it can keep up with a core2 duo laptop for file io. however
I moved to using an in ram database (4gb chunks, that then get dumped
over the network port to a db on a seperate machine)

I am not sure I have the skills to instrament this into bitcoind

> 
> I would like to do a test keeping database log files in memory. It 
> should not matter for durability of the wallet, as it flushes at
> each write anyway. As for the blockindex, it will remain
> consistent, but might be lagging some blocks behind at startup,
> which shouldn't really matter (except that the same block could end
> up appearing twice in the block00X files, inelegant, but not really
> a problem).
> 
> Otherwise the system you describe (raid0 over 6 disks) should
> perform like crazy wrt disk i/o, at least on par with SSD. It is
> your virtualization I am worried about.

I will test that when I get back :)

> 
> Have a safe trip to down under!

will do, thanks for the response. :)

> 
> /M
> 
> On 24/07/2012, at 21:56, steve wrote:
> 
>> Hi Michael,
>> 
>> from what I have noticed, bitcoin blockchain
>> download/verfication all happens in 1 thread.  (so multicores
>> doesnt really help)
>> 
>> That said, I have never tried on an ssd.
>> 
>> What I do have is 6 SATA 6gbs configed as RAID0 Drives. 32gb of 
>> ram. ubuntu 64 (yeah I know), this runs upto 16 VM's (I have 4
>> of these)
>> 
>> However I have not tried to download the blockchain on the
>> master os, just in virtulisation.  However, the dedicated
>> machines that I have been using for benchmarking the VM's against
>> is a q6600 8gb ram sata2 hdd - Win 7 (seems faster than
>> slackware...) to me it has always felt like network bandwidth was
>> the issue.  I might instrument the bitcoin-qt exe to only pick
>> low ping nodes (has someone already done this?)
>> 
>> I guess it is time to start some benchmarking (like the gpu 
>> comparison page)
>> 
>> hte verification for the 5 past 5 days was negliglable. I am off
>> on a flight to australia tomorrrow, so I will set some
>> breakpoints and do some timings in a debugger.
>> 
>> This will all happen on an e-450 (wonderful machine!)
>> 
>> Thanks very much for your response. it would seem that I am
>> 'doing it wrong' :/
>> 
>> cheers mate,
>> 
>> steve
>> 
>> (this message isnt signed because I have forgotten my password.)
>> 
>> On 24/07/2012 09:25, Michael Grønager wrote:
>>> Hi Steve,
>>> 
>>> 45-90 minutes - note that its numbers from March/April, so a
>>> bit longer today, but far, far away from the 12 hours.
>>> 
>>> I am using libcoin and the bitcoind build based on this.
>>> Libcoin is based on the Satoshi client, but refactured to use
>>> an async concurrency model. I also did a minor tweeks to the
>>> db parameters. It has earlier been tested up against Satoshi
>>> bitcoin where on some OS'es it performs similarly (at least on
>>> some linuxes) and on some faster (e.g. mac).
>>> 
>>> What is your CPU load during a block download ? (both 
>>> initially/up to the point where verification sets in and
>>> after). The initial download is typically disk I/O bound, the 
>>> verification stage CPU bound, though I lean to believe that
>>> even there it is disk I/O bound (at least on my system ~50% CPU
>>> load). What should be better in libcoin is the concurrency
>>> model. The Satoshi client uses a pure reentrant mutexes model,
>>> that is not generally believed to motivate the best coding
>>> practice nor performance, you might end up without the
>>> concurrency you initially strived for *). As mentioned earlier
>>> libcoin uses a pure async concurrency model (and so does
>>> libbitcoin btw).
>>> 
>>> I would like to stress again that these numbers will depend 
>>> largely on the system running the test - I would call my laptop
>>> a bit over the average today (MB Pro, 2.66Ghz i7 dual core,
>>> 8GBRAM, 512GB SSD). But again 12 hours - I only reach such
>>> numbers on some of my VPS'es (linode 1024) that are known for
>>> notoriously slow disk I/O. (here I have a few % CPU load during
>>> the verification indicating indeed that the disk i/o is the 
>>> culprit).
>>> 
>>> Cheers,
>>> 
>>> Michael
>>> 
>>> 
>>> *) I like this Dave Butenhof quote: "The biggest of all the big
>>>  problems with recursive mutexes is that they encourage you to
>>>  completely lose track of your locking scheme and scope. This
>>> is deadly. Evil. It's the "thread eater". You hold locks for
>>> the absolutely shortest possible time. Period. Always. If
>>> you're calling something with a lock held simply because you
>>> don't know it's held, or because you don't know whether the
>>> callee needs the mutex, then you're holding it too long. You're
>>> aiming a shotgun at your application and pulling the trigger.
>>> You presumably started using threads to get concurrency; but
>>> you've just PREVENTED concurrency."
>>> 
>>> 
>>> 
>>> 
>>> On 23/07/2012, at 17:54, steve wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 23/07/2012 10:00, Michael Grønager wrote:
>>>>>> I get a full blockchain from scratch in 45 minutes on my
>>>>>>  laptop, /M
>>>>>> 
>>> Hang on a sec, in 45 minutes you can download the entire chain 
>>> from the genesis block?
>>> 
>>> I have been doing extensive testing in this area and would
>>> love to know what is special about your setup (I have never had
>>> the entire chain in under 12 hours, infact it is normally
>>> closerto 24.) I have an extensive setup of test machines,
>>> everything from e4300 to phenom2x6 to i5's.
>>> 
>>> as an example on an amd e-450 with 4gb ram, and approx 3gb/s 
>>> internet connection it took 2 hours to sync the last 5 days.
>>> 
>>> Maybe i am missing something important...
>>> 
>>> Any additional information that you could provide to help me
>>> with testing would be really appreciated.
>>> 
>>> cheers,
>>> 
>>> steve
>>> 
>>>> 
>>>> ------------------------------------------------------------------------------
>>>>
>>>>
>>
>>>>
>>>> 
Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's
>>>> security and threat landscape has changed and how IT managers
>>>> can respond. Discussions will include endpoint security,
>>>> mobile security and the latest in malware threats. 
>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ 
>>>> _______________________________________________ 
>>>> Bitcoin-development mailing list 
>>>> Bitcoin-development@lists.sourceforge.net 
>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>>>
>>>> 
------------------------------------------------------------------------------
>> Live Security Virtual Conference Exclusive live event will cover 
>> all the ways today's security and threat landscape has changed
>> and how IT managers can respond. Discussions will include
>> endpoint security, mobile security and the latest in malware
>> threats. 
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ 
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net 
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 

Re: [Bitcoin-development] Scalability issues

[grarpamp]

Update: this class of machine just became useless for bitcoin.
When blk0002.dat was created to store more blocks, all forward
progress processing blocks turned into losing ground by 20 or so
a day. Guessing both datfiles were being accessed at once resulting
in disk based overload. I've not seen any other mentions of crypto
in this thread so I'm not sure how well new hardware would perform.
Going shopping I guess.

Re: [Bitcoin-development] Scalability issues

[Gregory Maxwell]

On Fri, Jul 27, 2012 at 12:20 AM, grarpamp <grarpamp@gmail.com> wrote:
> Update: this class of machine just became useless for bitcoin.
> When blk0002.dat was created to store more blocks, all forward
> progress processing blocks turned into losing ground by 20 or so
> a day. Guessing both datfiles were being accessed at once resulting
> in disk based overload. I've not seen any other mentions of crypto
> in this thread so I'm not sure how well new hardware would perform.
> Going shopping I guess.

I now have an 1.8 ghz p3 celeron (128k cache) which should be
substantially slower than your machine, running vintage 2.6.20 linux.
Unfortunately I forgot to turn on timestamp logging so I don't know
how long it took to sync the chain, but it was less than two days as
that was the span between when I checked on it. It's staying current
just fine.

Again, I encourage you to investigate your software configuration.

Re: [Bitcoin-development] Scalability issues

[grarpamp]

> I now have an 1.8 ghz p3 celeron (128k cache) which should be
> substantially slower than your machine, running vintage 2.6.20 linux.
> Unfortunately I forgot to turn on timestamp logging so I don't know
> how long it took to sync the chain, but it was less than two days as
> that was the span between when I checked on it. It's staying current

Well, are you running bitcoin on, say, an FS with sha256 integrity
trees for all bits and AES-128-XTS/CBC disk encryption?
If not, we're not comparing the same apples, let alone the same OS.

> Again, I encourage you to investigate your software configuration.

Someone suggested I investigate turning off the above features.
Since I'd find their loss undesirable [1], and there's not much to be
tuned there anyways, I've given up and am investigating what more
GHz and cores will do.

[1] Keeping data both intact and private is a good thing. Does your
checkbook deserve any less?

Re: [Bitcoin-development] Scalability issues

[Luke-Jr]

On Friday, July 27, 2012 5:59:20 AM grarpamp wrote:
> > I now have an 1.8 ghz p3 celeron (128k cache) which should be
> > substantially slower than your machine, running vintage 2.6.20 linux.
> > Unfortunately I forgot to turn on timestamp logging so I don't know
> > how long it took to sync the chain, but it was less than two days as
> > that was the span between when I checked on it. It's staying current
> 
> Well, are you running bitcoin on, say, an FS with sha256 integrity
> trees for all bits and AES-128-XTS/CBC disk encryption?

Trying to run state-of-the-art encryption on EVERYTHING on an ancient computer 
is fairly ill-advised. I encourage you to continue with the plan to go 
shopping.

> Someone suggested I investigate turning off the above features.
> Since I'd find their loss undesirable [1], and there's not much to be
> tuned there anyways, I've given up and am investigating what more
> GHz and cores will do.
> 
> [1] Keeping data both intact and private is a good thing. Does your
> checkbook deserve any less?

Sounds reasonable...
but why do you also need to encrypt 2+ GB of public record?

Luke

Re: [Bitcoin-development] Scalability issues

[grarpamp]

> shopping.

Good thing I can still spend, even with an incomplete blockchain :)

> but why do you also need to encrypt 2+ GB of public record?

1) I'm not seeing an option to split the wallet, debug log and other
privates pathwise from the blockchain.
2) Because encrypt everything is reasonable standard practice.
https://en.wikiquote.org/wiki/Cardinal_Richelieu [ref: disputed quote]

BTW, logs for this box say at least 9 days were spent attempting to
crunch the most recent 3100 blocks before it was overrun with new
ones and retired. (There's no formal start timestamp, just some entries...)

Re: [Bitcoin-development] Scalability issues

[Gregory Maxwell]

On Fri, Jul 27, 2012 at 1:59 AM, grarpamp <grarpamp@gmail.com> wrote:
>> I now have an 1.8 ghz p3 celeron (128k cache) which should be
>> substantially slower than your machine, running vintage 2.6.20 linux.
>> Unfortunately I forgot to turn on timestamp logging so I don't know
>> how long it took to sync the chain, but it was less than two days as
>> that was the span between when I checked on it. It's staying current
>
> Well, are you running bitcoin on, say, an FS with sha256 integrity
> trees for all bits and AES-128-XTS/CBC disk encryption?
> If not, we're not comparing the same apples, let alone the same OS.

The file system is using twofish-cbc-essiv:sha256, apparently.  (I
went and dug up a mothballed machine of mine because of your post).

And I agree, encrypting everything is a good practice— I once got a
disk back from RMA where only the first sectors were zeroed and the
rest had someone elses data, since then I've encrypted everything
because you can't wipe a dead drive.

I'd love to know precisely what Bitcoin is doing thats making your
machine so unhappy... but your configuration is uncommon for bitcoin
nodes in many distinct ways so it's not clear where to start.

Re: [Bitcoin-development] Scalability issues

[Andreas Petersson]

I propose a pragmatic solution: Try running the Multibit client. i am 
not sure if the linux/java based installer would work,so maybe you have 
to build it from source.
I tried it out is really fast compared to bitcoin-qt. after install it 
took me 15 seconds to get updated and running. Importing a private 
key/rescanning the blockchain was done in under 30 minutes.
It requires Java 6, i think there is a distro even for freebsd.
of course, you cannot do things as solomining with it since it uses SPV.

Re: [Bitcoin-development] Scalability issues

[Pieter Wuille]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

And now to the list...

On Jul 27, 2012 6:21 AM, "grarpamp" <grarpamp@gmail.com> wrote:
>
> Update: this class of machine just became useless for bitcoin.
> When blk0002.dat was created to store more blocks, all forward
> progress processing blocks turned into losing ground by 20 or so
> a day. Guessing both datfiles were being accessed at once resulting
> in disk based overload. I've not seen any other mentions of crypto
> in this thread so I'm not sure how well new hardware would perform.
> Going shopping I guess.

I doubt the encryption is the problem. I have a much more recent machine
(i7 with 8 GiB RAM), and my blockchain is on a (userspace!) encrypted
filesystem. However, I do not notice any measurable slowdown from doing so.

You are however using a filesystem (ZFS) that is uses its own filesystem
caching implementation to reach some performance, and is known to be very
memory-hungry at that. Furthermore, I believe it is known to have
performance issues on 32-bit architectures. The bdb backend Bitcoin uses
does many I/O operations, and writes them synchronously to disk, killing
whatever caching your filesystem provides. For those who run a large
database on ZFS, I believe it is advised to put ZFS's intent log on a
separate SSD-backed device, to get fast synchronous writes.

That said, some improvememts may be coming. Mike has been working on
changing the backend from bdb to leveldb, which may (or may not) result in
a very different performance profile on your machine. Also, I've been
working on switching the bitcoin block verifier to use a different style
database layout ("ultraprune"), which is smaller, faster, and will support
pruning. A recent test on my own machine synced the blockchain up to the
latest checkpoint in 7 minutes (6 minutes when stored in RAM instead of
disk).

-- 
Pieter

[-- Attachment #2: Type: text/html, Size: 2034 bytes --]

Re: [Bitcoin-development] Scalability issues

[grarpamp]

> You are however using a filesystem (ZFS)

I'm aware of the memory and i386 issues, and going shopping.

> The bdb backend Bitcoin uses
> does many I/O operations, and writes them synchronously to disk, killing
> whatever caching your filesystem provides.

Sync... yes, depending on the rate/sec and size of them, that would be an
issue. "Enterprise" systems with UPS, good disk, etc assume writes are
committed upon return, eliminating need for software apps to do sync. So
I need to figure out how to turn that off?

Is sync on for everything, or just the wallet (where it could be argued as ok)?

> For those who run a large
> database on ZFS, I believe it is advised to put ZFS's intent log on a
> separate SSD-backed device, to get fast synchronous writes.

Guessing bitcoin's writes are small? So a RAM dev intent would be cheaper
and faster than SSD for that.

> on switching the bitcoin block verifier to use a different style database
> layout ("ultraprune"), which is smaller, faster, and will support pruning.

I'll try to search that. If it's anything like "delete old blocks/tx/coins that
have both been validated in the past and fully spent in the future since we
no longer need to validate further back beyond them [1]", that would be
interesting.

[1] Unless you're a historian or some usage other than casual transactions.

Re: [Bitcoin-development] Scalability issues

[grarpamp]

> I'd love to know precisely what Bitcoin is doing thats making your
> machine so unhappy... but your configuration is uncommon for bitcoin
> nodes in many distinct ways so it's not clear where to start.

That's why I posted the details of the machine so interested people
could duplicate it if they are working in these areas. Once I'm on new
hardware and playing around I, also, may find some discoveries.

I have no doubt that 'common' is windows seven on core2duo
without ecc and no crypto or hashing for the disk :)