From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1Uz1vR-00086U-19 for bitcoin-development@lists.sourceforge.net; Tue, 16 Jul 2013 09:52:05 +0000 Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.219.44 as permitted sender) client-ip=209.85.219.44; envelope-from=mh.in.england@gmail.com; helo=mail-oa0-f44.google.com; Received: from mail-oa0-f44.google.com ([209.85.219.44]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1Uz1vP-0008Qg-9e for bitcoin-development@lists.sourceforge.net; Tue, 16 Jul 2013 09:52:04 +0000 Received: by mail-oa0-f44.google.com with SMTP id l10so558249oag.3 for ; Tue, 16 Jul 2013 02:51:58 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.60.79.131 with SMTP id j3mr797568oex.96.1373968317898; Tue, 16 Jul 2013 02:51:57 -0700 (PDT) Sender: mh.in.england@gmail.com Received: by 10.76.23.36 with HTTP; Tue, 16 Jul 2013 02:51:57 -0700 (PDT) In-Reply-To: <2F20A509-13A9-4C84-86D7-A15C21BACD53@include7.ch> References: <3E7894A0-06F3-453D-87F8-975A244EBACF@include7.ch> <2BDA0943-22BB-4405-9AF0-86FB41FD04A6@include7.ch> <2F20A509-13A9-4C84-86D7-A15C21BACD53@include7.ch> Date: Tue, 16 Jul 2013 11:51:57 +0200 X-Google-Sender-Auth: BI6VhVigBv-yq5UtCbHeOlFVcds Message-ID: From: Mike Hearn To: Jonas Schnelli , Bitcoin Dev Content-Type: multipart/alternative; boundary=047d7b67812685670d04e19ded42 X-Spam-Score: -0.5 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (mh.in.england[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1Uz1vP-0008Qg-9e Subject: Re: [Bitcoin-development] Introducing BitcoinKit.framework X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jul 2013 09:52:05 -0000 --047d7b67812685670d04e19ded42 Content-Type: text/plain; charset=UTF-8 Let's re-add the list as this is a topic of general interest. On Tue, Jul 16, 2013 at 11:36 AM, Jonas Schnelli wrote: > What do you think about extending the BitcoinKit.framework (based on > bitcoind) so that the kit could handle the very basic SPV concept > (getHeaders aka fast catchup, wallet-birthday, bloom filters)? > Maybe it would be possible to only port some of the bitcoinj work to c++ > (and use it for extending BitcoinKit or bitcoind)? > Maybe then it could also be a starting point for someone who likes to > create a SPV mode for bitcoind? > Making bitcoind/Bitcoin-Qt support SPV mode was the original plan some years ago, Satoshi even sent me some code he wrote that did the first parts, but it was incomplete. At the time, I decided to do a separate implementation for a few different reasons. One is that my understanding of his code wasn't so good back then and I lacked confidence to change it. Especially as there were no unit tests back then (and still aren't any for most of it), making invasive changes to the core validation code was and is highly risky. A separate code base seemed to reduce the risk a lot. Another reason is that Satoshi encouraged me to write a simple re-implementation that people could learn from. And I wanted a documented, object oriented API that people could use to build a variety of apps. Yet another reason was bitcoind is security critical code that scrapes complex data structures from untrusted sources on the internet, and it's written in an unmanaged language. Ordinarily this would be a recipe for disaster as a single overflow or memory management error could lead to hacking and theft on a massive scale. It's like taking a chainsaw and using it to carve an ice sculpture. Satoshi, incredibly, pulled it off, mostly by using advanced C++ features that made his code hard to read for many people and by being very, very careful. I was not convinced I could do such a good job and was worried about accidentally introducing vulnerabilities. A final reason is that it was clear that the bitcoind codebase would need serious changes for mobiles, beyond that required for ordinary SPV support. For example, Satoshi's code assumes it has access to block headers via a std::map and that assumption is made in a lot of places. On Android phones, you can't fit all the block headers in RAM. bitcoinj uses a circular ring buffer of the last N thousand headers for this reason. It's quite different to how bitcoind works. All that said, it was a ton of work and it's still unclear that it was the right call. Anyway, your situation is a little different. Firstly you don't care about mobiles, your app is intended for desktops. So the changes required are less invasive. Also, there are more unit tests and more people with a good understanding of the code these days, so perhaps the risk of introducing bugs is lower. And these days we have some nice APIs for building apps so that need is already met. If you wanted to implement SPV mode in bitcoind, Gavin or I could send you Satoshi's old patch although of course it is no longer usable. It would indicate the basic cut lines though. --047d7b67812685670d04e19ded42 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Let's re-add the list as this is a topic of general in= terest.

On Tue, Ju= l 16, 2013 at 11:36 AM, Jonas Schnelli <jonas.schnelli@include7.c= h> wrote:
What do you think about extending the Bitco= inKit.framework (based on bitcoind) so that the kit could handle the very b= asic SPV concept (getHeaders aka fast catchup, wallet-birthday, bloom filte= rs)?
Maybe it would be possible to only port some of the bitcoinj wor= k to c++ (and use it for extending BitcoinKit or bitcoind)?
Maybe= then it could also be a starting point for someone who likes to create a S= PV mode for bitcoind?

Making bitcoind/Bitcoin-Qt support S= PV mode was the original plan some years ago, Satoshi even sent me some cod= e he wrote that did the first parts, but it was incomplete.

At the time, I decided to do a separate implementation for a few= different reasons. One is that my understanding of his code wasn't so = good back then and I lacked confidence to change it. Especially as there we= re no unit tests back then (and still aren't any for most of it), makin= g invasive changes to the core validation code was and is highly risky. A s= eparate code base seemed to reduce the risk a lot.

Another reason is that Satoshi encouraged me to write a= simple re-implementation that people could learn from. And I wanted a docu= mented, object oriented API that people could use to build a variety of app= s.

Yet another reason was bitcoind is security critical co= de that scrapes complex data structures from untrusted sources on the inter= net, and it's written in an unmanaged language. Ordinarily this would b= e a recipe for disaster as a single overflow or memory management error cou= ld lead to hacking and theft on a massive scale. It's like taking a cha= insaw and using it to carve an ice sculpture. Satoshi, incredibly, pulled i= t off, mostly by using advanced C++ features that made his code hard to rea= d for many people and by being very, very careful. I was not convinced I co= uld do such a good job and was worried about accidentally introducing vulne= rabilities.

A final reason is that it was clear that the bitcoind c= odebase would need serious changes for mobiles, beyond that required for or= dinary SPV support. For example, Satoshi's code assumes it has access t= o block headers via a std::map and that assumption is made in a lot of plac= es. On Android phones, you can't fit all the block headers in RAM. bitc= oinj uses a circular ring buffer of the last N thousand headers for this re= ason. It's quite different to how bitcoind works.

All that said, it was a ton of work and it's still = unclear that it was the right call.

Anyway, your s= ituation is a little different. Firstly you don't care about mobiles, y= our app is intended for desktops. So the changes required are less invasive= . Also, there are more unit tests and more people with a good understanding= of the code these days, so perhaps the risk of introducing bugs is lower. = And these days we have some nice APIs for building apps so that need is alr= eady met.

If you wanted to implement SPV mode in bitcoind, Gavin = or I could send you Satoshi's old patch although of course it is no lon= ger usable. It would indicate the basic cut lines though.=C2=A0
=
--047d7b67812685670d04e19ded42--