From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1StlEQ-0001ht-BP for bitcoin-development@lists.sourceforge.net; Tue, 24 Jul 2012 19:57:22 +0000 Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of mistfpga.net designates 208.91.199.217 as permitted sender) client-ip=208.91.199.217; envelope-from=steve@mistfpga.net; helo=us2.outbound.mailhostbox.com; Received: from us2.outbound.mailhostbox.com ([208.91.199.217]) by sog-mx-1.v43.ch3.sourceforge.com with esmtp (Exim 4.76) id 1StlEP-0007Ne-A9 for bitcoin-development@lists.sourceforge.net; Tue, 24 Jul 2012 19:57:22 +0000 Received: from [10.10.10.55] (5ad2e75a.bb.sky.com [90.210.231.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: steve@mistfpga.net) by us2.outbound.mailhostbox.com (Postfix) with ESMTPSA id D8A1B6990EC; Tue, 24 Jul 2012 15:57:13 -0400 (EDT) Message-ID: <500EFE00.7060200@mistfpga.net> Date: Tue, 24 Jul 2012 20:56:48 +0100 From: steve User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Michael_Gr=F8nager?= References: <500D73B9.5030806@mistfpga.net> In-Reply-To: X-Enigmail-Version: 1.4.3 Content-Type: text/plain; charset=ISO-8859-1 X-CTCH-Spam: Unknown X-CTCH-VOD: Unknown X-CTCH-RefID: str=0001.0A020209.500EFE1B.011A, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-Scanned-By: MIMEDefang 2.72 on 172.16.214.9 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.4 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid 0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid X-Headers-End: 1StlEP-0007Ne-A9 Cc: bitcoin-development@lists.sourceforge.net Subject: Re: [Bitcoin-development] Scalability issues X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2012 19:57:22 -0000 Hi Michael, from what I have noticed, bitcoin blockchain download/verfication all happens in 1 thread. (so multicores doesnt really help) That said, I have never tried on an ssd. What I do have is 6 SATA 6gbs configed as RAID0 Drives. 32gb of ram. ubuntu 64 (yeah I know), this runs upto 16 VM's (I have 4 of these) However I have not tried to download the blockchain on the master os, just in virtulisation. However, the dedicated machines that I have been using for benchmarking the VM's against is a q6600 8gb ram sata2 hdd - Win 7 (seems faster than slackware...) to me it has always felt like network bandwidth was the issue. I might instrument the bitcoin-qt exe to only pick low ping nodes (has someone already done this?) I guess it is time to start some benchmarking (like the gpu comparison pa= ge) hte verification for the 5 past 5 days was negliglable. I am off on a flight to australia tomorrrow, so I will set some breakpoints and do some timings in a debugger. This will all happen on an e-450 (wonderful machine!) Thanks very much for your response. it would seem that I am 'doing it wrong' :/ cheers mate, steve (this message isnt signed because I have forgotten my password.) On 24/07/2012 09:25, Michael Gr=F8nager wrote: > Hi Steve, >=20 > 45-90 minutes - note that its numbers from March/April, so a bit > longer today, but far, far away from the 12 hours. >=20 > I am using libcoin and the bitcoind build based on this. Libcoin is > based on the Satoshi client, but refactured to use an async > concurrency model. I also did a minor tweeks to the db parameters. It > has earlier been tested up against Satoshi bitcoin where on some > OS'es it performs similarly (at least on some linuxes) and on some > faster (e.g. mac). >=20 > What is your CPU load during a block download ? (both initially/up to > the point where verification sets in and after). The initial download > is typically disk I/O bound, the verification stage CPU bound, though > I lean to believe that even there it is disk I/O bound (at least on > my system ~50% CPU load). What should be better in libcoin is the > concurrency model. The Satoshi client uses a pure reentrant mutexes > model, that is not generally believed to motivate the best coding > practice nor performance, you might end up without the concurrency > you initially strived for *). As mentioned earlier libcoin uses a > pure async concurrency model (and so does libbitcoin btw). >=20 > I would like to stress again that these numbers will depend largely > on the system running the test - I would call my laptop a bit over > the average today (MB Pro, 2.66Ghz i7 dual core, 8GBRAM, 512GB SSD). > But again 12 hours - I only reach such numbers on some of my VPS'es > (linode 1024) that are known for notoriously slow disk I/O. (here I > have a few % CPU load during the verification indicating indeed that > the disk i/o is the culprit). >=20 > Cheers, >=20 > Michael >=20 >=20 > *) I like this Dave Butenhof quote: "The biggest of all the big > problems with recursive mutexes is that they encourage you to > completely lose track of your locking scheme and scope. This is > deadly. Evil. It's the "thread eater". You hold locks for the > absolutely shortest possible time. Period. Always. If you're calling > something with a lock held simply because you don't know it's held, > or because you don't know whether the callee needs the mutex, then > you're holding it too long. You're aiming a shotgun at your > application and pulling the trigger. You presumably started using > threads to get concurrency; but you've just PREVENTED concurrency." >=20 >=20 >=20 >=20 > On 23/07/2012, at 17:54, steve wrote: >=20 > Hi Michael, >=20 > On 23/07/2012 10:00, Michael Gr=F8nager wrote: >>>> I get a full blockchain from scratch in 45 minutes on my >>>> laptop, /M >>>>=20 > Hang on a sec, in 45 minutes you can download the entire chain from=20 > the genesis block? >=20 > I have been doing extensive testing in this area and would love to=20 > know what is special about your setup (I have never had the entire=20 > chain in under 12 hours, infact it is normally closerto 24.) I have > an extensive setup of test machines, everything from e4300 to > phenom2x6 to i5's. >=20 > as an example on an amd e-450 with 4gb ram, and approx 3gb/s > internet connection it took 2 hours to sync the last 5 days. >=20 > Maybe i am missing something important... >=20 > Any additional information that you could provide to help me with=20 > testing would be really appreciated. >=20 > cheers, >=20 > steve >=20 >>=20 >> ----------------------------------------------------------------------= -------- >> >>=20 Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and=20 >> threat landscape has changed and how IT managers can respond. >> Discussions will include endpoint security, mobile security and the >> latest in malware threats. >> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/=20 >> _______________________________________________ Bitcoin-development >> mailing list Bitcoin-development@lists.sourceforge.net=20 >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >=20 >=20 >=20 >=20 >=20