From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 411FFB90 for ; Fri, 7 Apr 2017 16:02:37 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4EB4E151 for ; Fri, 7 Apr 2017 16:02:36 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 97F8720AC1; Fri, 7 Apr 2017 12:02:35 -0400 (EDT) Received: from web3 ([10.202.2.213]) by compute2.internal (MEProxy); Fri, 07 Apr 2017 12:02:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=BnVY9p lJBVd+tQwwdJHcIIzjgRHS78W4ClxqmRyiz54=; b=agoTBvsjlu5wP4k3dVA2om TNnMTp2bzbEORsgapedPlJ7ydnWHPFigXAaYZgf0UuPSPamUURGelgof6689PIFC wOizS45TafsiSqTEnSzK5gSp7uPU6CfmDLst+MgvLJiqNzmkAK7Hj19y4m08cqsi lbFDp8uq4DP0GagC+PvKZ/q8rGpXkVJdxeH6NAXA7nWxq1pIP1n1pZIbiQSdAwMp sacnv4aeezX793N+22OfroBcfAC01o90cXWxu3yLbeMLVpd7OubNeFnHFwRm5yAZ EcVhzR1mjMtRgW89dmxsNG2oC/+p8ZK5sAp5FeCUCX9Qi1l4cKYT4wElWiIUfsyg == X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 7642B9EC4C; Fri, 7 Apr 2017 12:02:35 -0400 (EDT) Message-Id: <1491580955.73830.937628696.4EEA507C@webmail.messagingengine.com> From: Tomas To: Greg Sanders , Bitcoin Protocol Discussion MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: multipart/alternative; boundary="_----------=_1491580955738300" X-Mailer: MessagingEngine.com Webmail Interface - ajax-7c174d5d In-Reply-To: Date: Fri, 07 Apr 2017 18:02:35 +0200 References: <1491516747.3791700.936828232.69F82904@webmail.messagingengine.com> <1491554876.1963053.937226528.7010832E@webmail.messagingengine.com> X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,URIBL_RHS_DOB autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Fri, 07 Apr 2017 16:04:16 +0000 Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Apr 2017 16:02:37 -0000 This is a multi-part message in MIME format. --_----------=_1491580955738300 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" Thank you, The benches are running in Google Cloud Engine; currently on 8 vCPU 32gb, but I tend to switch hardware regularly. Roughly, the results are better for Bitcrust with high end hardware and the difference for total block validations is mostly diminished at 2 vCPU, 7,5 gb. Note that the spend-tree optimization primarily aims to improve peak load order validation; when a block with pre-synced transactions comes in, but this is tricky to accurately bench with Core using this simple method of comparison by logs. I will upgrade to, and show the results against 0.14 in the next weeks. Best, Tomas On Fri, Apr 7, 2017, at 16:14, Greg Sanders wrote: > Interesting work. > > I was wondering if you could tellank us what specs for the machine > being used as preliminary benchmark is here: > https://bitcrust.org/results ? > > I'd be interested to also see comparisons with 0.14 which has some > improvements for script validation with more cores. > > On Fri, Apr 7, 2017 at 4:47 AM, Tomas via bitcoin-dev dev@lists.linuxfoundation.org> wrote: >> Thank you Marcos, >> >> Though written in Rust, bitcrust-db is definitely usable as >> pluggable >> module as its interface will be roughly some queries, add_tx and >> add_block with blobs and flags. (Bitcrust internally uses a >> deserialize-only model, keeping references to the blobs with the >> parsed >> data). >> >> However, from Core's side I believe network and storage are >> currently >> rather tightly coupled, which will make this far from trivial. >> >> Regardless, I am also hoping (with funding & a team) to build a >> Bitcrust >> networking component as well to bring a strong competitor to the >> market. >> >> best, >> Tomas >> >> >> >> >> On Fri, Apr 7, 2017, at 09:55, Marcos mayorga wrote: >> > Hi Tomas, >> > >> > I've read it and think it is an excellent work, I'd like to see it >> > integrated into bitcoin-core as a 'kernel module'. >> > >> > I see there are a lot of proof of concepts out there, IMO >> > every one >> > deserve a room in the bitcoin client as a selectable feature, to >> > make the >> > software more flexible and less dictatorial, an user could easily >> > select >> > which features she wants to run. >> > >> > Best regards, >> > Marcos >> > >> > > I have been working on a bitcoin implementation that uses a >> > > different >> > > approach to indexing for verifying the order of transactions. >> > > Instead of >> > > using an index of unspent outputs, double spends are verified by >> > > using a >> > > spend-tree where spends are scanned against spent outputs >> > > instead of >> > > unspent outputs. >> > > >> > > This allows for much better concurrency, as not only blocks, but >> > > also >> > > individual inputs can be verified fully in parallel. >> > > >> > > I explain the approach at https://bitcrust.org, source code is >> > > available >> > > at https://github.com/tomasvdw/bitcrust >> > > >> > > I am sharing this not only to ask for your feedback, but also to >> > > call >> > > for a clear separation of protocol and implementations: As this >> > > solution, reversing the costs of outputs and inputs, seems to >> > > have >> > > excellent performance characteristics (as shown in the test >> > > results), >> > > updates to the protocol addressing the UTXO growth, might not be >> > > worth >> > > considering *protocol improvements* and it might be best to >> > > address >> > > these concerns as implementation details. >> > > >> > > Kind regards, >> > > Tomas van der Wansem >> > > tomas@bitcrust.org >> > > Bitcrust >> > > _______________________________________________ >> > > bitcoin-dev mailing list >> > > bitcoin-dev@lists.linuxfoundation.org >> > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> > > >> > >> > >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev --_----------=_1491580955738300 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset="utf-8"
Thank you,

The benches are running in Google Cloud Engine; currently on 8 vCPU 32gb, but I tend to switch hardware regularly.

Roughly, the results are better for Bitcrust with high end hardware and the difference for total block validations is mostly diminished at 2 vCPU, 7,5 gb.

Note that the spend-tree optimization primarily aims to improve peak load order validation; when a block with pre-synced transactions comes in, but this is tricky to accurately bench with Core using this simple method of comparison by logs.

I will upgrade to, and show the results against 0.14 in the next weeks.

Best,
Tomas


On Fri, Apr 7, 2017, at 16:14, Greg Sanders wrote:
Interesting work.

I was wondering if you could tellank  us what specs for the machine being used as preliminary benchmark is here: https://bitcrust.org/results ?

I'd be interested to also see comparisons with 0.14 which has some improvements for script validation with more cores.

On Fri, Apr 7, 2017 at 4:47 AM, Tomas via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
Thank you Marcos,

Though written in Rust, bitcrust-db is definitely usable as pluggable
module as its interface will be roughly some queries, add_tx and
add_block with blobs and flags. (Bitcrust internally uses a
deserialize-only model, keeping references to the blobs with the parsed
data).

However, from Core's side I believe network and storage are currently
rather tightly coupled, which will make this far from trivial.

Regardless, I am also hoping (with funding & a team) to build a Bitcrust
networking component as well to bring a strong competitor to the market.

best,
Tomas




On Fri, Apr 7, 2017, at 09:55, Marcos mayorga wrote:
> Hi Tomas,
>
> I've read it and think it is an excellent work, I'd like to see it
> integrated into bitcoin-core as a 'kernel module'.
>
> I see there are a lot of proof of concepts out there, IMO every one
> deserve a room in the bitcoin client as a selectable feature, to make the
> software more flexible and less dictatorial, an user could easily select
> which features she wants to run.
>
> Best regards,
> Marcos
>
> > I have been working on a bitcoin implementation that uses a different
> > approach to indexing for verifying the order of transactions. Instead of
> > using an index of unspent outputs, double spends are verified by using a
> > spend-tree where spends are scanned against spent outputs instead of
> > unspent outputs.
> >
> > This allows for much better concurrency, as not only blocks, but also
> > individual inputs can be verified fully in parallel.
> >
> > I explain the approach at https://bitcrust.org, source code is available
> >
> > I am sharing this not only to ask for your feedback, but also to call
> > for a clear separation of protocol and implementations: As this
> > solution, reversing the costs of outputs and inputs, seems to have
> > excellent performance characteristics (as shown in the test results),
> > updates to the protocol addressing the UTXO growth, might not be worth
> > considering *protocol improvements* and it might be best to address
> > these concerns as implementation details.
> >
> > Kind regards,
> > Tomas van der Wansem
> > Bitcrust
> > _______________________________________________
> > bitcoin-dev mailing list
> >
>
>
_______________________________________________
bitcoin-dev mailing list

--_----------=_1491580955738300--