[Bitcoin-development] Introducing BitcoinKit.framework

[Bitcoin-development] Introducing BitcoinKit.framework

[Wendell]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

Hi devs,

Just wanted to cross-post this here since it seems very relevant.

We're launching BitcoinKit.framework, a Cocoa framework that allows developers to write Bitcoin wallet apps for Mac OS X. BitcoinKit uses bitcoind, and serves a small and tidy API for developer use. Support for other Bitcoin implementations (libcbitcoin, etc) is soon to follow.

BitcoinKit's first application is as the backbone of a new Mac wallet app called Hive, which will be released soon at www.grabhive.com.

Grab the source here:
https://github.com/grabhive/BitcoinKit

Support is available via GitHub issues and this Bitcointalk thread:
https://bitcointalk.org/index.php?topic=256583.msg2733523

A sample GUI app is also included:
http://imgur.com/FzqA00X

Cheers everyone!

-Wendell

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1962 bytes --]

That's great! I'm all for more wallets, especially user friendly UIs.

However being based on bitcoind means it will take a very long time to
synchronize for new users. We know a lot of users drop out. The best fix
for this is SPV mode. Do you have any plans in this direction?

So far, the only SPV mode implementation I know about is bitcoinj. I am
experimenting with trans-piling bitcoinj to C++ to make it usable from
Objective-C++ exactly with your use case in mind.



On Mon, Jul 15, 2013 at 12:07 PM, Wendell <w@grabhive.com> wrote:

> Hi devs,
>
> Just wanted to cross-post this here since it seems very relevant.
>
> We're launching BitcoinKit.framework, a Cocoa framework that allows
> developers to write Bitcoin wallet apps for Mac OS X. BitcoinKit uses
> bitcoind, and serves a small and tidy API for developer use. Support for
> other Bitcoin implementations (libcbitcoin, etc) is soon to follow.
>
> BitcoinKit's first application is as the backbone of a new Mac wallet app
> called Hive, which will be released soon at www.grabhive.com.
>
> Grab the source here:
> https://github.com/grabhive/BitcoinKit
>
> Support is available via GitHub issues and this Bitcointalk thread:
> https://bitcointalk.org/index.php?topic=256583.msg2733523
>
> A sample GUI app is also included:
> http://imgur.com/FzqA00X
>
> Cheers everyone!
>
> -Wendell
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 3021 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Wendell]

Hi Mike,

You are absolutely right about the synchronize time, it's one of our main frustration points right now and we clearly won't deliver the kind of user experience we want, without fixing this. Actually we were thinking of extending Jeff Garzik's picocoin as time permits, but the plan is far from concrete at the moment.

What you say about trans-piling bitcoinj is _really_ appealing. We discounted Java simply because an OS X JVM is no longer guaranteed, but otherwise bitcoinj is ideal for our purposes. How can we assist or otherwise accelerate such an effort?

-w

On Jul 15, 2013, at 3:19 PM, Mike Hearn wrote:

> That's great! I'm all for more wallets, especially user friendly UIs.
> 
> However being based on bitcoind means it will take a very long time to synchronize for new users. We know a lot of users drop out. The best fix for this is SPV mode. Do you have any plans in this direction?
> 
> So far, the only SPV mode implementation I know about is bitcoinj. I am experimenting with trans-piling bitcoinj to C++ to make it usable from Objective-C++ exactly with your use case in mind.

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 4469 bytes --]

Oracle provide an OSX JVM and will do so for the forseeable future, it's
also open source, so the community could carry on if they stopped. The
primary problem with the Oracle JVM is lack of retina support for Swing,
but if you'd write a Cocoa UI yourself then it doesn't matter of course as
Java won't handle any GUI stuff. Retina support for JavaFX2 (the
current-gen gui toolkit) is available in Java 8 so it's definitely being
actively developed, it's not abandoned or anything.

So the question then becomes, which is better:

a) Take bitcoinj completely out of the Java world via native compilation or
transpilation to C++
b) Embed the JVM and link the two worlds together?

(b) is much less ambitious, especially if you're OK with writing a bit of
Java code to keep the interface thin. Basically the Java side calls into
your app when interesting user-visible things happen, like new transactions
appearing, then your GUI can call into the java side to send money. There
are auto-translators that make the glue work easy, like
https://code.google.com/p/javacpp/. You probably wouldn't want to expose
the entire bitcoinj API that way because it's very large, but the code
needed to bring up a wallet app is very small. I knocked one up this
weekend in about one evenings worth of coding, completed with nice
animations. The interfaces you'd need are basically some Objective-C++
methods that receive information from the Bitcoin side, like the balance
having changed, a list of transactions, etc, and then a callback into the
Java side to send money. If you look at the javacpp site you can see
example code for making calls both ways.

If I were in your shoes, I'd go for (b) because it is the most well trodden
path and will let you achieve the best user visible results quickly. The
JVM can be bundled with your app and stripped down if you're worried about
download size.

If it's unclear how the code would look, let me know and I'll try and knock
up a really simple prototype.

There's also (a). I'm investigating transpilation for a few reasons, one of
which is to do with a private project. I'm working with the author of j2c:
https://code.google.com/a/eclipselabs.org/p/j2c/. It's a rather
sophisticated transpiler that converts Java to clean, readable C++11 that
looks much like code a human would write. It's complete enough to transpile
the entire standard Java class library, including all the GUI toolkits and
other things - so, pretty amazing piece of code. However it's incomplete
because where the Java code calls native methods (that would be provided by
the JVM) it just spits out stubs you're expected to fill out yourself, for
starting threads and so on. As there's no JVM it's just like using a C++
library that is missing a "portability layer".

I'm working on this myself and don't really need much help at the moment,
I'm just making steady progress towards getting something up and running. I
can let you know once I reach some interesting milestones. The point of
this is that whilst you don't need access to most of the API to write a
wallet app, I'd like to make every kind of app easy from C++, not just GUI
wallets. Then the compile-to-C++ approach is much more appealing, even
though it's also more work.





On Mon, Jul 15, 2013 at 4:39 PM, Wendell <w@grabhive.com> wrote:

> Hi Mike,
>
> You are absolutely right about the synchronize time, it's one of our main
> frustration points right now and we clearly won't deliver the kind of user
> experience we want, without fixing this. Actually we were thinking of
> extending Jeff Garzik's picocoin as time permits, but the plan is far from
> concrete at the moment.
>
> What you say about trans-piling bitcoinj is _really_ appealing. We
> discounted Java simply because an OS X JVM is no longer guaranteed, but
> otherwise bitcoinj is ideal for our purposes. How can we assist or
> otherwise accelerate such an effort?
>
> -w
>
> On Jul 15, 2013, at 3:19 PM, Mike Hearn wrote:
>
> > That's great! I'm all for more wallets, especially user friendly UIs.
> >
> > However being based on bitcoind means it will take a very long time to
> synchronize for new users. We know a lot of users drop out. The best fix
> for this is SPV mode. Do you have any plans in this direction?
> >
> > So far, the only SPV mode implementation I know about is bitcoinj. I am
> experimenting with trans-piling bitcoinj to C++ to make it usable from
> Objective-C++ exactly with your use case in mind.
>
>

[-- Attachment #2: Type: text/html, Size: 5411 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 7649 bytes --]

You can cut down the JVM to be a few megabytes if you're aggressive about
it. But for a desktop app I'm not sure it's really necessary these days. A
few megabytes used to make a noticeable difference to success rates but
bandwidth improved a lot since then.

Portability to android is a given, it's already Java based. IOS is a non
starter until apple is convinced to allow wallet apps into the App store,
language is not the issue there.

There is no point manually rewriting bitcoinj to c++ when j2c does such a
great job already. You would want to at last start from what it generates
even if you fork from there.
On 15 Jul 2013 20:19, "Jonas Schnelli" <jonas.schnelli@include7.ch> wrote:

> The embedded JVM is a way. But the binary will be huge.
> And how about the portability (like iOS and Android)?
>
> If i would have "unlimited resources" and like to make a "perfect native
> client" i would see two solutions:
> a) add SPV features to bitcoind and go on with BitcoinKit.framework (maybe
> first SPV features are only available through "API's" and not for bitcoind
> as runnable binary)
> b) rewrite bitcoinj in c++ (*auto-port the unit-tests* and try to rewrite
> line by line to c++)
>
> also a mix of a) and b) is possible. Like extending bitcoind with the
> SPVBlockstore, bloom filter, etc. from bitcoinj (rewritten in c++). The
> wallet birthday must also be added somehow.
>
> both a) and b) (or the mix) is a lot of work and might take much longer as
> Mike's JVM idea.
> But it then might end up in a stable, portable and extendable pice of code.
>
> </jonas>
>
>
>
> Oracle provide an OSX JVM and will do so for the forseeable future, it's
> also open source, so the community could carry on if they stopped. The
> primary problem with the Oracle JVM is lack of retina support for Swing,
> but if you'd write a Cocoa UI yourself then it doesn't matter of course as
> Java won't handle any GUI stuff. Retina support for JavaFX2 (the
> current-gen gui toolkit) is available in Java 8 so it's definitely being
> actively developed, it's not abandoned or anything.
>
> So the question then becomes, which is better:
>
> a) Take bitcoinj completely out of the Java world via native compilation
> or transpilation to C++
> b) Embed the JVM and link the two worlds together?
>
> (b) is much less ambitious, especially if you're OK with writing a bit of
> Java code to keep the interface thin. Basically the Java side calls into
> your app when interesting user-visible things happen, like new transactions
> appearing, then your GUI can call into the java side to send money. There
> are auto-translators that make the glue work easy, like
> https://code.google.com/p/javacpp/. You probably wouldn't want to expose
> the entire bitcoinj API that way because it's very large, but the code
> needed to bring up a wallet app is very small. I knocked one up this
> weekend in about one evenings worth of coding, completed with nice
> animations. The interfaces you'd need are basically some Objective-C++
> methods that receive information from the Bitcoin side, like the balance
> having changed, a list of transactions, etc, and then a callback into the
> Java side to send money. If you look at the javacpp site you can see
> example code for making calls both ways.
>
> If I were in your shoes, I'd go for (b) because it is the most well
> trodden path and will let you achieve the best user visible results
> quickly. The JVM can be bundled with your app and stripped down if you're
> worried about download size.
>
> If it's unclear how the code would look, let me know and I'll try and
> knock up a really simple prototype.
>
> There's also (a). I'm investigating transpilation for a few reasons, one
> of which is to do with a private project. I'm working with the author of
> j2c: https://code.google.com/a/eclipselabs.org/p/j2c/. It's a rather
> sophisticated transpiler that converts Java to clean, readable C++11 that
> looks much like code a human would write. It's complete enough to transpile
> the entire standard Java class library, including all the GUI toolkits and
> other things - so, pretty amazing piece of code. However it's incomplete
> because where the Java code calls native methods (that would be provided by
> the JVM) it just spits out stubs you're expected to fill out yourself, for
> starting threads and so on. As there's no JVM it's just like using a C++
> library that is missing a "portability layer".
>
> I'm working on this myself and don't really need much help at the moment,
> I'm just making steady progress towards getting something up and running. I
> can let you know once I reach some interesting milestones. The point of
> this is that whilst you don't need access to most of the API to write a
> wallet app, I'd like to make every kind of app easy from C++, not just GUI
> wallets. Then the compile-to-C++ approach is much more appealing, even
> though it's also more work.
>
>
>
>
>
> On Mon, Jul 15, 2013 at 4:39 PM, Wendell <w@grabhive.com> wrote:
>
>> Hi Mike,
>>
>> You are absolutely right about the synchronize time, it's one of our main
>> frustration points right now and we clearly won't deliver the kind of user
>> experience we want, without fixing this. Actually we were thinking of
>> extending Jeff Garzik's picocoin as time permits, but the plan is far from
>> concrete at the moment.
>>
>> What you say about trans-piling bitcoinj is _really_ appealing. We
>> discounted Java simply because an OS X JVM is no longer guaranteed, but
>> otherwise bitcoinj is ideal for our purposes. How can we assist or
>> otherwise accelerate such an effort?
>>
>> -w
>>
>> On Jul 15, 2013, at 3:19 PM, Mike Hearn wrote:
>>
>> > That's great! I'm all for more wallets, especially user friendly UIs.
>> >
>> > However being based on bitcoind means it will take a very long time to
>> synchronize for new users. We know a lot of users drop out. The best fix
>> for this is SPV mode. Do you have any plans in this direction?
>> >
>> > So far, the only SPV mode implementation I know about is bitcoinj. I am
>> experimenting with trans-piling bitcoinj to C++ to make it usable from
>> Objective-C++ exactly with your use case in mind.
>>
>>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk_______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
> °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
> include7 AG
> Jonas Schnelli
> Mattengasse 27
> CH-8005 Zürich
> Switzerland
> Office: +41 44 500 16 70
>
> Mail: jonas.schnelli@include7.ch
> Web: www.include7.ch
> V-Card: www.include7.ch/js.vcf
> °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
>
> ACHTUNG
> Bitte senden sie uns keine sensitiven Daten in unverschlüsselten E-Mails.
> Verwenden Sie hierzu folgenden Link:
> https://include7.ch/contact/secureform
>
> °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
>
>

[-- Attachment #2: Type: text/html, Size: 10145 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 3024 bytes --]

> Clear. Your right. C++ would give us more flexibility (other platforms)
> and also android compatibility (through NDK).
>

I'm a bit confused I'm afraid. bitcoinj already runs SPV wallets on Android
on top of Dalvik. In fact that's what it's designed for. The NDK is not
necessary to work with Bitcoin at any point.


> That's a great idea.
> Let me look into the quality of j2c's output.
>

There's an example of what it looks like here:

https://code.google.com/a/eclipselabs.org/p/j2c/wiki/Examples

If you're serious about playing with j2c let me know. It's an amazing piece
of work BUT it was written for fun, and as such isn't really documented at
all. It took me a little while to figure out how to make it work properly.
I'm now fixing bugs in it and making various improvements along with
filling out the native stubs (a.k.a. portability layer). If you want to
catch up to where I'm at, I can send you some notes because otherwise you
might waste a lot of time on blind alleys.

The main things be aware of so far are:

   - Lots of explicit null pointer checks are generated. The reason is that
   the output is meant to be entirely portable, so Jacek doesn't want to rely
   on platform specific stuff like signals or SEH. Simplest solution is just
   to disable npc() generation entirely because normal C++ libraries just
   segfault if a null pointer gets in the wrong place, they don't throw
   exceptions. Losing the Java behaviour would not be a downgrade for people
   used to C++.

   - Array accesses don't seem to be properly bounds-checked. That's a part
   of the Java security model - bitcoinj is written on the assumption that
   buffer and heap overflows aren't possible because they're caught by the
   runtime. If those checks go missing then it'd likely become possible to
   hack your program by exploiting buffer overflows. So that needs to be fixed.

   - Generated code doesn't use the STL of course, it can't because the
   Java library has more features than the STL. However as the way j2c works
   is you transpile your code alongside a copy of the (open source) Java class
   library, you can go in and modify the generated code for java::lang::String
   or java::util::List and so on to add helper methods for converting to
   various other forms. On Linux you'd have implicit c'tors to go back and
   forth between std::string, on MacOS X you'd have conversions for NSString,
   you could add code for QStrings or raw C strings too. Once the code has
   been generated you can extend or patch it to make the API more convenient.

   - Obviously, the resulting code requires the Boehm GC because there are
   no explicit delete calls anywhere. This is a safety feature though, it
   avoids use-after-free and double-free bugs that can create security holes.

   - The code generator doesn't do dependency tracing, so you end up with
   generated code that isn't used anywhere. It's up to the linker to do a dead
   code elimination pass. Otherwise the resulting binaries can be huge.

[-- Attachment #2: Type: text/html, Size: 3924 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

Let's re-add the list as this is a topic of general interest.

On Tue, Jul 16, 2013 at 11:36 AM, Jonas Schnelli <jonas.schnelli@include7.ch
> wrote:

> What do you think about extending the BitcoinKit.framework (based on
> bitcoind) so that the kit could handle the very basic SPV concept
> (getHeaders aka fast catchup, wallet-birthday, bloom filters)?
> Maybe it would be possible to only port some of the bitcoinj work to c++
> (and use it for extending BitcoinKit or bitcoind)?
> Maybe then it could also be a starting point for someone who likes to
> create a SPV mode for bitcoind?
>

Making bitcoind/Bitcoin-Qt support SPV mode was the original plan some
years ago, Satoshi even sent me some code he wrote that did the first
parts, but it was incomplete.

At the time, I decided to do a separate implementation for a few different
reasons. One is that my understanding of his code wasn't so good back then
and I lacked confidence to change it. Especially as there were no unit
tests back then (and still aren't any for most of it), making invasive
changes to the core validation code was and is highly risky. A separate
code base seemed to reduce the risk a lot.

Another reason is that Satoshi encouraged me to write a simple
re-implementation that people could learn from. And I wanted a documented,
object oriented API that people could use to build a variety of apps.

Yet another reason was bitcoind is security critical code that scrapes
complex data structures from untrusted sources on the internet, and it's
written in an unmanaged language. Ordinarily this would be a recipe for
disaster as a single overflow or memory management error could lead to
hacking and theft on a massive scale. It's like taking a chainsaw and using
it to carve an ice sculpture. Satoshi, incredibly, pulled it off, mostly by
using advanced C++ features that made his code hard to read for many people
and by being very, very careful. I was not convinced I could do such a good
job and was worried about accidentally introducing vulnerabilities.

A final reason is that it was clear that the bitcoind codebase would need
serious changes for mobiles, beyond that required for ordinary SPV support.
For example, Satoshi's code assumes it has access to block headers via a
std::map and that assumption is made in a lot of places. On Android phones,
you can't fit all the block headers in RAM. bitcoinj uses a circular ring
buffer of the last N thousand headers for this reason. It's quite different
to how bitcoind works.

All that said, it was a ton of work and it's still unclear that it was the
right call.

Anyway, your situation is a little different. Firstly you don't care about
mobiles, your app is intended for desktops. So the changes required are
less invasive. Also, there are more unit tests and more people with a good
understanding of the code these days, so perhaps the risk of introducing
bugs is lower. And these days we have some nice APIs for building apps so
that need is already met.

If you wanted to implement SPV mode in bitcoind, Gavin or I could send you
Satoshi's old patch although of course it is no longer usable. It would
indicate the basic cut lines though.

[-- Attachment #2: Type: text/html, Size: 3826 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Wendell]

I for one would be interested in taking a look at that.

In San Jose I was asking around about the possibility of hiring someone to complete such a patch. Pieter Wuille introduced me to Eric Lombrozo, who expressed interest, but has since gotten quite busy. So we haven't arrived at a detailed estimate of what it would involve.

Maybe it would be better to start a completely new thread for this topic, but I would very much be interested in an open dissection of what adding SPV support to bitcoind would take. I am willing to fund or (ideally) co-fund this endeavor, if I can ever get my head around it. I'm super interested in all of these possibilities (including micro-stripped-VMs and transpilation), but would simply like to encourage the proliferation of _options_ whenever possible.

-wendell

grabhive.com | twitter.com/grabhive

On Jul 16, 2013, at 11:51 AM, Mike Hearn wrote:

> If you wanted to implement SPV mode in bitcoind, Gavin or I could send you Satoshi's old patch although of course it is no longer usable. It would indicate the basic cut lines though. 

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 12892 bytes --]

I think that's a great approach. Here is the patch Satoshi sent me back in
2010. All the code has changed since but it can be a source of inspiration.

From Satoshi:

*The simplified payment verification in the paper imagined you would
receive transactions directly, as with sending to IP address which nobody
uses, or a node would index all transactions by public key and you could
download them like downloading mail from a mail server.

Instead, I think client-only nodes should receive full blocks so they can
scan them for their own transactions.  They don't need to store them or
index them.  For the initial download, they only need to download headers,
since there couldn't be any payments before the first time the program was
run (a header download command was added in 0.3.18).  From then on, they
download full blocks (but only store the headers).

Code for client-only mode is mostly implemented.  There's a feature branch
on github with it, also I'm attaching the patch to this message.

Here's some more about it:

"Here's my client-mode implementation so far.  Client-only mode only
records block headers and doesn't use the tx index.  It can't generate, but
it can still send and receive transactions.  It's not fully finished for
use by end-users, but it doesn't matter because it's a complete no-op if
fClient is not enabled.  At this point it's mainly documentation showing
the cut-lines for client-only re-implementers.

With fClient=true, I've only tested the header-only initial download.

A little background.  CBlockIndex contains all the information of the block
header, so to operate with headers only, I just maintain the CBlockIndex
structure as usual.  The nFile/nBlockPos are null, since the full block is
not recorded on disk.

The code to gracefully switch between client-mode on/off without deleting
blk*.dat in between is not implemented yet.  It would mostly be a matter of
having non-client LoadBlockIndex ignore block index entries with null block
pos.  That would make it re-download those as full blocks.  Switching back
to client-mode is no problem, it doesn't mind if the full blocks are there.

If the initial block download becomes too long, we'll want client mode as
an option so new users can get running quickly.  With graceful switch-off
of client mode, they can later turn off client mode and have it download
the full blocks if they want to start generating.  They should rather just
use a getwork miner to join a pool instead.

Client-only re-implementations would not need to implement EvalScript at
all, or at most just implement the five ops used by the standard
transaction templates."
*




diff -u old\db.cpp new\db.cpp
--- old\db.cpp  Sat Dec 18 18:35:59 2010
+++ new\db.cpp  Sun Dec 19 20:53:59 2010
@@ -464,29 +464,32 @@
     ReadBestInvalidWork(bnBestInvalidWork);

     // Verify blocks in the best chain
-    CBlockIndex* pindexFork = NULL;
-    for (CBlockIndex* pindex = pindexBest; pindex && pindex->pprev; pindex
= pindex->pprev)
+    if (!fClient)
     {
-        if (pindex->nHeight < nBestHeight-2500 &&
!mapArgs.count("-checkblocks"))
-            break;
-        CBlock block;
-        if (!block.ReadFromDisk(pindex))
-            return error("LoadBlockIndex() : block.ReadFromDisk failed");
-        if (!block.CheckBlock())
+        CBlockIndex* pindexFork = NULL;
+        for (CBlockIndex* pindex = pindexBest; pindex && pindex->pprev;
pindex = pindex->pprev)
         {
-            printf("LoadBlockIndex() : *** found bad block at %d,
hash=%s\n", pindex->nHeight, pindex->GetBlockHash().ToString().c_str());
-            pindexFork = pindex->pprev;
+            if (pindex->nHeight < nBestHeight-2500 &&
!mapArgs.count("-checkblocks"))
+                break;
+            CBlock block;
+            if (!block.ReadFromDisk(pindex))
+                return error("LoadBlockIndex() : block.ReadFromDisk
failed");
+            if (!block.CheckBlock())
+            {
+                printf("LoadBlockIndex() : *** found bad block at %d,
hash=%s\n", pindex->nHeight, pindex->GetBlockHash().ToString().c_str());
+                pindexFork = pindex->pprev;
+            }
+        }
+        if (pindexFork)
+        {
+            // Reorg back to the fork
+            printf("LoadBlockIndex() : *** moving best chain pointer back
to block %d\n", pindexFork->nHeight);
+            CBlock block;
+            if (!block.ReadFromDisk(pindexFork))
+                return error("LoadBlockIndex() : block.ReadFromDisk
failed");
+            CTxDB txdb;
+            block.SetBestChain(txdb, pindexFork);
         }
-    }
-    if (pindexFork)
-    {
-        // Reorg back to the fork
-        printf("LoadBlockIndex() : *** moving best chain pointer back to
block %d\n", pindexFork->nHeight);
-        CBlock block;
-        if (!block.ReadFromDisk(pindexFork))
-            return error("LoadBlockIndex() : block.ReadFromDisk failed");
-        CTxDB txdb;
-        block.SetBestChain(txdb, pindexFork);
     }

     return true;
diff -u old\main.cpp new\main.cpp
--- old\main.cpp        Sat Dec 18 18:35:59 2010
+++ new\main.cpp        Sun Dec 19 20:53:59 2010
@@ -637,6 +637,9 @@
     if (!IsStandard())
         return error("AcceptToMemoryPool() : nonstandard transaction
type");

+    if (fClient)
+        return true;
+
     // Do we already have it?
     uint256 hash = GetHash();
     CRITICAL_BLOCK(cs_mapTransactions)
@@ -1308,23 +1311,26 @@
     if (!CheckBlock())
         return false;

-    //// issue here: it doesn't know the version
-    unsigned int nTxPos = pindex->nBlockPos + ::GetSerializeSize(CBlock(),
SER_DISK) - 1 + GetSizeOfCompactSize(vtx.size());
-
-    map<uint256, CTxIndex> mapUnused;
-    int64 nFees = 0;
-    foreach(CTransaction& tx, vtx)
+    if (!fClient)
     {
-        CDiskTxPos posThisTx(pindex->nFile, pindex->nBlockPos, nTxPos);
-        nTxPos += ::GetSerializeSize(tx, SER_DISK);
+        //// issue here: it doesn't know the version
+        unsigned int nTxPos = pindex->nBlockPos +
::GetSerializeSize(CBlock(), SER_DISK) - 1 + GetSizeOfCompactSize(vtx.size(
));
+
+        map<uint256, CTxIndex> mapUnused;
+        int64 nFees = 0;
+        foreach(CTransaction& tx, vtx)
+        {
+            CDiskTxPos posThisTx(pindex->nFile, pindex->nBlockPos, nTxPos);
+            nTxPos += ::GetSerializeSize(tx, SER_DISK);

-        if (!tx.ConnectInputs(txdb, mapUnused, posThisTx, pindex, nFees,
true, false))
+            if (!tx.ConnectInputs(txdb, mapUnused, posThisTx, pindex,
nFees, true, false))
+                return false;
+        }
+
+        if (vtx[0].GetValueOut() > GetBlockValue(pindex->nHeight, nFees))
             return false;
     }

-    if (vtx[0].GetValueOut() > GetBlockValue(pindex->nHeight, nFees))
-        return false;
-
     // Update block index on disk without changing it in memory.
     // The memory index structure will be changed after the db commits.
     if (pindex->pprev)
@@ -1378,7 +1384,7 @@
     foreach(CBlockIndex* pindex, vDisconnect)
     {
         CBlock block;
-        if (!block.ReadFromDisk(pindex))
+        if (!block.ReadFromDisk(pindex, !fClient))
             return error("Reorganize() : ReadFromDisk for disconnect
failed");
         if (!block.DisconnectBlock(txdb, pindex))
             return error("Reorganize() : DisconnectBlock failed");
@@ -1395,7 +1401,7 @@
     {
         CBlockIndex* pindex = vConnect[i];
         CBlock block;
-        if (!block.ReadFromDisk(pindex))
+        if (!block.ReadFromDisk(pindex, !fClient))
             return error("Reorganize() : ReadFromDisk for connect failed");
         if (!block.ConnectBlock(txdb, pindex))
         {
@@ -1526,7 +1532,7 @@

     txdb.Close();

-    if (pindexNew == pindexBest)
+    if (!fClient && pindexNew == pindexBest)
     {
         // Notify UI to display prev block's coinbase if it was ours
         static uint256 hashPrevBestCoinBase;
@@ -1547,10 +1553,6 @@
     // These are checks that are independent of context
     // that can be verified before saving an orphan block.

-    // Size limits
-    if (vtx.empty() || vtx.size() > MAX_BLOCK_SIZE ||
::GetSerializeSize(*this, SER_NETWORK) > MAX_BLOCK_SIZE)
-        return error("CheckBlock() : size limits failed");
-
     // Check proof of work matches claimed amount
     if (!CheckProofOfWork(GetHash(), nBits))
         return error("CheckBlock() : proof of work failed");
@@ -1559,6 +1561,13 @@
     if (GetBlockTime() > GetAdjustedTime() + 2 * 60 * 60)
         return error("CheckBlock() : block timestamp too far in the
future");

+    if (fClient && vtx.empty())
+        return true;
+
+    // Size limits
+    if (vtx.empty() || vtx.size() > MAX_BLOCK_SIZE ||
::GetSerializeSize(*this, SER_NETWORK) > MAX_BLOCK_SIZE)
+        return error("CheckBlock() : size limits failed");
+
     // First transaction must be coinbase, the rest must not be
     if (vtx.empty() || !vtx[0].IsCoinBase())
         return error("CheckBlock() : first tx is not coinbase");
@@ -1623,13 +1632,14 @@
         return error("AcceptBlock() : out of disk space");
     unsigned int nFile = -1;
     unsigned int nBlockPos = 0;
-    if (!WriteToDisk(nFile, nBlockPos))
-        return error("AcceptBlock() : WriteToDisk failed");
+    if (!fClient)
+        if (!WriteToDisk(nFile, nBlockPos))
+            return error("AcceptBlock() : WriteToDisk failed");
     if (!AddToBlockIndex(nFile, nBlockPos))
         return error("AcceptBlock() : AddToBlockIndex failed");

     // Relay inventory, but don't relay old inventory during initial block
download
-    if (hashBestChain == hash)
+    if (!fClient && hashBestChain == hash)
         CRITICAL_BLOCK(cs_vNodes)
             foreach(CNode* pnode, vNodes)
                 if (nBestHeight > (pnode->nStartingHeight != -1 ?
pnode->nStartingHeight - 2000 : 55000))
@@ -2405,6 +2415,8 @@
         {
             if (fShutdown)
                 return true;
+            if (fClient && inv.type == MSG_TX)
+                continue;
             pfrom->AddInventoryKnown(inv);

             bool fAlreadyHave = AlreadyHave(txdb, inv);
@@ -2441,6 +2453,9 @@

             if (inv.type == MSG_BLOCK)
             {
+                if (fClient)
+                    return true;
+
                 // Send block from disk
                 map<uint256, CBlockIndex*>::iterator mi =
mapBlockIndex.find(inv.hash);
                 if (mi != mapBlockIndex.end())
@@ -2486,6 +2501,8 @@

     else if (strCommand == "getblocks")
     {
+        if (fClient)
+            return true;
         CBlockLocator locator;
         uint256 hashStop;
         vRecv >> locator >> hashStop;
@@ -2556,6 +2573,8 @@

     else if (strCommand == "tx")
     {
+        if (fClient)
+            return true;
         vector<uint256> vWorkQueue;
         CDataStream vMsg(vRecv);
         CTransaction tx;
@@ -2620,6 +2639,33 @@

         if (ProcessBlock(pfrom, &block))
             mapAlreadyAskedFor.erase(inv);
+    }
+
+
+    else if (strCommand == "headers")
+    {
+        if (!fClient)
+            return true;
+        vector<CBlock> vHeaders;
+        vRecv >> vHeaders;
+
+        uint256 hashBestBefore = hashBestChain;
+        foreach(CBlock& block, vHeaders)
+        {
+            block.vtx.clear();
+
+            printf("received header %s\n", block.GetHash().ToString().
substr(0,20).c_str());
+
+            CInv inv(MSG_BLOCK, block.GetHash());
+            pfrom->AddInventoryKnown(inv);
+
+            if (ProcessBlock(pfrom, &block))
+                mapAlreadyAskedFor.erase(inv);
+        }
+
+        // Request next batch
+        if (hashBestChain != hashBestBefore)
+            pfrom->PushGetBlocks(pindexBest, uint256(0));
     }


On Tue, Jul 16, 2013 at 12:17 PM, Wendell <w@grabhive.com> wrote:

> I for one would be interested in taking a look at that.
>
> In San Jose I was asking around about the possibility of hiring someone to
> complete such a patch. Pieter Wuille introduced me to Eric Lombrozo, who
> expressed interest, but has since gotten quite busy. So we haven't arrived
> at a detailed estimate of what it would involve.
>
> Maybe it would be better to start a completely new thread for this topic,
> but I would very much be interested in an open dissection of what adding
> SPV support to bitcoind would take. I am willing to fund or (ideally)
> co-fund this endeavor, if I can ever get my head around it. I'm super
> interested in all of these possibilities (including micro-stripped-VMs and
> transpilation), but would simply like to encourage the proliferation of
> _options_ whenever possible.
>
> -wendell
>
> grabhive.com | twitter.com/grabhive
>
> On Jul 16, 2013, at 11:51 AM, Mike Hearn wrote:
>
> > If you wanted to implement SPV mode in bitcoind, Gavin or I could send
> you Satoshi's old patch although of course it is no longer usable. It would
> indicate the basic cut lines though.
>
>

[-- Attachment #2: Type: text/html, Size: 46356 bytes --]

[Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Wendell]

Hello everyone,

In the previous thread, I expressed interest in seeing an SPV bitcoind, further stating that I would fund such work. Mike Hearn followed up with some of Satoshi's old code for this, which is now quite broken. The offer and interest on my side still stand, as more diversity in SPV options seems like the right way to go.

Time-permitting, I would really appreciate feedback from knowledgable parties about the possible approaches to an SPV bitcoind. We at Hive ideally want to see something that could one be merge into master, rather than a fork.

-wendell

grabhive.com | twitter.com/grabhive

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 4796 bytes --]

You'd want to create and get merged patches in the following order:

1) Be able to store just block headers in the blkXXXX.dat files instead of
full block contents. At this point you are still *downloading* full blocks,
but they are not being stored. The contents are still sent to the wallet
for extracting relevant transactions though (see SyncWithWallets).  You
also need to disable listening and addr announcements to the P2P network at
this point. You need to be able to re-org and do all the usual things
without storing block contents. You also need to short-circuit the leveldbs
so they aren't created or used. All that needs to be unit tested. You need
to also rewrite the mempool logic so it throws out irrelevant transactions.
The RPC interface needs to adjust itself so you can't try to start mining,
query the utxo set, etc.

At this point you have an SPV node, albeit one that still downloads the
entire block chain. However total disk storage used will be much lower.
Getting this written and reviewed is a big chunk of work but is the hardest
part. Once it's done you can breath easy.

2) Next step, use getheaders to catch up with the chain until the
min(wallet birthdays) is reached. You can see in Satoshi's patch where he
adds support for receiving "headers" messages. Because key times are
recorded as dates and you don't know the dates of blocks in advance, you
need to download headers until you see one that goes past the key birthday
minus some slack period, then throw out the headers you downloaded and
switch to downloading full blocks again from that point onwards.

3) Next step, implement client side support for Bloom filtering. Switch
from downloading full blocks to filteredblocks, verify the Merkle branches
then apply them to the wallet. Watch out for accidental re-orderings of
transactions here from block order (e.g. if you accidentally insert them
into a std::map or other unordered collection it can lead to bugs). Come up
with some way to decide on a FP rate. Probably you want a fairly high FP
rate for desktop wallets.

4) Next step (optional), implement monitoring of broadcast propagation for
transactions that are received. SPV clients cannot verify unconfirmed
transactions so you can either just give up entirely and accept any old
garbage, or assume a non-MITMd internet connection and use network
propagation as a rough proxy for "likely to be valid and mined upon".

4) Optimize!

How much you need to optimize really depends on a lot of things. I found
that to be competitive with Electrum/blockchain.info I had to do a ton of
optimizations including very aggressive checkpointing so new users don't
have to download more than a month or twos worth of headers, as downloading
all the headers was becoming a bottleneck. You'd need to download about
16mb+ of data at the moment to grab all the headers and on a weakass mobile
phone with a weak Dalvik VM and 3G internet this was way too much. I also
had to spend some time profiling to ensure we weren't accidentally
thrashing the UI due to too-fast updates, we weren't bottlenecking on
updating last seen block data in the wallet, we weren't accidentally
de/reserializing messages redundantly etc.

After about 3-4 evenings of non-stop profiling and optimising I ended up
with a relatively flat profile whilst doing initial catchup and chain sync.
On a desktop I bet you can get away with much less optimisation because
your CPUs, network and disk tend to be much stronger.



On Tue, Jul 16, 2013 at 4:16 PM, Wendell <w@grabhive.com> wrote:

> Hello everyone,
>
> In the previous thread, I expressed interest in seeing an SPV bitcoind,
> further stating that I would fund such work. Mike Hearn followed up with
> some of Satoshi's old code for this, which is now quite broken. The offer
> and interest on my side still stand, as more diversity in SPV options seems
> like the right way to go.
>
> Time-permitting, I would really appreciate feedback from knowledgable
> parties about the possible approaches to an SPV bitcoind. We at Hive
> ideally want to see something that could one be merge into master, rather
> than a fork.
>
> -wendell
>
> grabhive.com | twitter.com/grabhive
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 5851 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 6951 bytes --]

On Tue, Jul 16, 2013 at 04:16:23PM +0200, Wendell wrote:
> Hello everyone,
> 
> In the previous thread, I expressed interest in seeing an SPV bitcoind, further stating that I would fund such work. Mike Hearn followed up with some of Satoshi's old code for this, which is now quite broken. The offer and interest on my side still stand, as more diversity in SPV options seems like the right way to go.
> 
> Time-permitting, I would really appreciate feedback from knowledgable parties about the possible approaches to an SPV bitcoind. We at Hive ideally want to see something that could one be merge into master, rather than a fork.

Keep in mind that SPV mode is newer than many realize: bloom filters are
a 0.8 feature, itself released only last Febuary. As John Dillon posted
earlier this week in "Protecting Bitcoin against network-wide DoS
attack" the Bitcoin codebase will have to implement much better anti-DoS
attack defences soon, and in a decentralized system there aren't any
options other than requiring peers to either do work (useful or not) or
sacrifice something of value. SPV peers can't do useful work, leaving
only sacrifice - to what extent and how much is unknown. In addition SPV
nodes have serious privacy issues because their peers know that any
transaction sent to them by the SPV node is guaranteed to be from the
node rather than relayed; bloom filters are only really helpful with
payment protocols that don't exist yet and don't apply to merchants.
Then you have MITM problems, vulnerability to fake blocks etc.

It'll be awhile before we know how serious these issues are in practice,
and we're likely to find new issues we didn't think of too. In any case
Bitcoin is far better off if we make it easy to run a full node,
donating whatever resources you can. Fortunately there's a whole
continuum between SPV and full nodes.

The way you do this is by maintaining partial UTXO sets. The trick is
that if you have verified every block in some range i to j, every time
you see a txout created by a transaction, and not subsequently spent,
you can be sure that at height j the txout existed. If height j is the
current block, you can be sure the txout exists provided that the chain
itself is valid. Any transaction that only spends txouts in this partial
set is a transaction you can fully verify and safely relay; for other
transactions you just don't know and have to wait until you see them in
a block.

So what's useful about that? Basically it means your node starts with
the same security level, and usefulness to the network, as a SPV node.
But over time you keep downloading blocks as they are created, and with
whatever bandwidth you have left (out of some user-configurable
allocation) you download additional blocks going further and further
back in time. Gradually your UTXO set becomes more complete, and over
time you can verify a higher and higher % of all valid transactions.
Eventually your node becomes a full node, but in the meantime it was
still useful for the user, and still contributed to the network by
relaying blocks and an increasingly large subset of all transactions.
(optionally you can store a subset of the chain history too for other
nodes to bootstrap from) You've also got better security because you
*are* validating blocks, starting off incompletely, and increasingly
completely until your finally validating fully. Privacy is improved, for
both you and others, by mixing your transactions with others and adding
to the overall anonymity set.

In the future we'll have miners commit a hash of the UTXO set, and that
gives us even more options to, for instance, have relayed transactions
include proof that their inputs were valid, allowing all nodes to relay
them safely.


As for specifics, you need to maintain a UTXO set, and in addition a set
of spent txouts (the STXO set) for which you haven't seen the
transaction that created the txout. As download newer blocks you update
the UTXO set; as you download older blocks you update the UTXO set and
STXO set.

Nodes now advertise this new variable to their peers:

nOldestBlock - The oldest block that we've validated. (and all
subsequent blocks)

We'll also want the ability to advertise what sub-ranges of the
blockchain data we have on hand:

listArchivedBlockRanges - lists of (begin, end pairs)

Nodes should drop all but the largest n pairs, say 5 or something. The
index -1 is reserved to indicate the last block to make it easy to
advertise that you have every block starting at some height to the most
recent. (reserving -n with n as the last block might be a better choice
to show intent, but still allow for specific proofs when we get node
identities)

We probably want to define a NODE_PARTIAL service bit or something; I'll
have to re-read Pieter Wuille's proposal and think about it. Nodes
should NOT advertize NODE_NETWORK unless they have the full chain and
have verified it.

Nodes with partial peers should only relay transactions to those peers
if the transactions spend inputs the peers know about - remember how
even an SPV node has that information if it's not spending unconfirmed
inputs it didn't create. Nodes will have to update their peers
periodically as nOldestBlock changes. That said it may also be
worthwhile to simply relay all transactions in some cases too - a
reasonable way to approach this might be to set a bloom filter for tx's
that you *definitely* want, and if you are interested in everything,
just set the filter to all 1's. If someone comes up with a reasonable
micropayment or proof-of-work system even relaying txs that you haven't
validated is fine - the proof-of-work and prioritization will prevent
DoS attacks just fine.

Remember that if you're running a partial node, it can get new blocks
from any partial node, and it can retrieve historic blockchain data from
any partial node that has archived the sequence of blocks you need next.
On a large scale this is similar to how in BitTorrent you can serve data
to your peers the moment you get it - a significant scalability
improvement for the network as a whole. Even if a large % of the network
was partial nodes running for just a few hours a day the whole system
would work fine due to how partial nodes can serve each other the data
they need.

On startup you can act as a SPV node temporarily, grabbing asking for
filtered blocks matching your wallet, and then go back and get the full
blocks, or just download the full blocks right away. That's a tradeoff
on how long the node has been off.

Anyway, it's a bit more code compared to pure-SPV, but it results in a
much more scalable Bitcoin, and if you can spare the modest bandwidth
requirements to keep up with the blockchain it'll result in much better
robustness against DoS attacks for you and Bitcoin in general.

-- 
'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 8519 bytes --]

Partial UTXO sets is a neat idea. Unfortunately my intuition is that many
SPV wallets only remain open for <1 minute at a time because the user wants
to see they received money, or to send it. It'd be neat to get some
telemetry from the Android wallet for this - I will ask Andreas to let
users opt in to usage statistics.

So for anti-DoS I think smart prioritisation heuristics are the way to go
again. Perhaps by letting clients have an "identity" that they provide to a
node when it's load shedding. Clients that have been seen before, have a
track record of not being abusive etc get priority and new clients that
were never seen before get dropped. Coming up with a way to do that whilst
preserving privacy sounds like an interesting cryptographic challenge.


On Wed, Jul 17, 2013 at 12:58 PM, Peter Todd <pete@petertodd.org> wrote:

> On Tue, Jul 16, 2013 at 04:16:23PM +0200, Wendell wrote:
> > Hello everyone,
> >
> > In the previous thread, I expressed interest in seeing an SPV bitcoind,
> further stating that I would fund such work. Mike Hearn followed up with
> some of Satoshi's old code for this, which is now quite broken. The offer
> and interest on my side still stand, as more diversity in SPV options seems
> like the right way to go.
> >
> > Time-permitting, I would really appreciate feedback from knowledgable
> parties about the possible approaches to an SPV bitcoind. We at Hive
> ideally want to see something that could one be merge into master, rather
> than a fork.
>
> Keep in mind that SPV mode is newer than many realize: bloom filters are
> a 0.8 feature, itself released only last Febuary. As John Dillon posted
> earlier this week in "Protecting Bitcoin against network-wide DoS
> attack" the Bitcoin codebase will have to implement much better anti-DoS
> attack defences soon, and in a decentralized system there aren't any
> options other than requiring peers to either do work (useful or not) or
> sacrifice something of value. SPV peers can't do useful work, leaving
> only sacrifice - to what extent and how much is unknown. In addition SPV
> nodes have serious privacy issues because their peers know that any
> transaction sent to them by the SPV node is guaranteed to be from the
> node rather than relayed; bloom filters are only really helpful with
> payment protocols that don't exist yet and don't apply to merchants.
> Then you have MITM problems, vulnerability to fake blocks etc.
>
> It'll be awhile before we know how serious these issues are in practice,
> and we're likely to find new issues we didn't think of too. In any case
> Bitcoin is far better off if we make it easy to run a full node,
> donating whatever resources you can. Fortunately there's a whole
> continuum between SPV and full nodes.
>
> The way you do this is by maintaining partial UTXO sets. The trick is
> that if you have verified every block in some range i to j, every time
> you see a txout created by a transaction, and not subsequently spent,
> you can be sure that at height j the txout existed. If height j is the
> current block, you can be sure the txout exists provided that the chain
> itself is valid. Any transaction that only spends txouts in this partial
> set is a transaction you can fully verify and safely relay; for other
> transactions you just don't know and have to wait until you see them in
> a block.
>
> So what's useful about that? Basically it means your node starts with
> the same security level, and usefulness to the network, as a SPV node.
> But over time you keep downloading blocks as they are created, and with
> whatever bandwidth you have left (out of some user-configurable
> allocation) you download additional blocks going further and further
> back in time. Gradually your UTXO set becomes more complete, and over
> time you can verify a higher and higher % of all valid transactions.
> Eventually your node becomes a full node, but in the meantime it was
> still useful for the user, and still contributed to the network by
> relaying blocks and an increasingly large subset of all transactions.
> (optionally you can store a subset of the chain history too for other
> nodes to bootstrap from) You've also got better security because you
> *are* validating blocks, starting off incompletely, and increasingly
> completely until your finally validating fully. Privacy is improved, for
> both you and others, by mixing your transactions with others and adding
> to the overall anonymity set.
>
> In the future we'll have miners commit a hash of the UTXO set, and that
> gives us even more options to, for instance, have relayed transactions
> include proof that their inputs were valid, allowing all nodes to relay
> them safely.
>
>
> As for specifics, you need to maintain a UTXO set, and in addition a set
> of spent txouts (the STXO set) for which you haven't seen the
> transaction that created the txout. As download newer blocks you update
> the UTXO set; as you download older blocks you update the UTXO set and
> STXO set.
>
> Nodes now advertise this new variable to their peers:
>
> nOldestBlock - The oldest block that we've validated. (and all
> subsequent blocks)
>
> We'll also want the ability to advertise what sub-ranges of the
> blockchain data we have on hand:
>
> listArchivedBlockRanges - lists of (begin, end pairs)
>
> Nodes should drop all but the largest n pairs, say 5 or something. The
> index -1 is reserved to indicate the last block to make it easy to
> advertise that you have every block starting at some height to the most
> recent. (reserving -n with n as the last block might be a better choice
> to show intent, but still allow for specific proofs when we get node
> identities)
>
> We probably want to define a NODE_PARTIAL service bit or something; I'll
> have to re-read Pieter Wuille's proposal and think about it. Nodes
> should NOT advertize NODE_NETWORK unless they have the full chain and
> have verified it.
>
> Nodes with partial peers should only relay transactions to those peers
> if the transactions spend inputs the peers know about - remember how
> even an SPV node has that information if it's not spending unconfirmed
> inputs it didn't create. Nodes will have to update their peers
> periodically as nOldestBlock changes. That said it may also be
> worthwhile to simply relay all transactions in some cases too - a
> reasonable way to approach this might be to set a bloom filter for tx's
> that you *definitely* want, and if you are interested in everything,
> just set the filter to all 1's. If someone comes up with a reasonable
> micropayment or proof-of-work system even relaying txs that you haven't
> validated is fine - the proof-of-work and prioritization will prevent
> DoS attacks just fine.
>
> Remember that if you're running a partial node, it can get new blocks
> from any partial node, and it can retrieve historic blockchain data from
> any partial node that has archived the sequence of blocks you need next.
> On a large scale this is similar to how in BitTorrent you can serve data
> to your peers the moment you get it - a significant scalability
> improvement for the network as a whole. Even if a large % of the network
> was partial nodes running for just a few hours a day the whole system
> would work fine due to how partial nodes can serve each other the data
> they need.
>
> On startup you can act as a SPV node temporarily, grabbing asking for
> filtered blocks matching your wallet, and then go back and get the full
> blocks, or just download the full blocks right away. That's a tradeoff
> on how long the node has been off.
>
> Anyway, it's a bit more code compared to pure-SPV, but it results in a
> much more scalable Bitcoin, and if you can spare the modest bandwidth
> requirements to keep up with the blockchain it'll result in much better
> robustness against DoS attacks for you and Bitcoin in general.
>
> --
> 'peter'[:-1]@petertodd.org
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 9816 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Wendell]

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

Peter,

This sounds like a _very_ good idea for a desktop client, and probably acceptable to users so long as we take available disk space into consideration, and only ever use a fraction of it.

Will you implement this?

-wendell

grabhive.com | twitter.com/grabhive

On Jul 17, 2013, at 12:58 PM, Peter Todd wrote:

> So what's useful about that? Basically it means your node starts with
> the same security level, and usefulness to the network, as a SPV node.
> But over time you keep downloading blocks as they are created, and with
> whatever bandwidth you have left (out of some user-configurable
> allocation) you download additional blocks going further and further
> back in time. Gradually your UTXO set becomes more complete, and over
> time you can verify a higher and higher % of all valid transactions.
> Eventually your node becomes a full node, but in the meantime it was
> still useful for the user, and still contributed to the network by
> relaying blocks and an increasingly large subset of all transactions.
> (optionally you can store a subset of the chain history too for other
> nodes to bootstrap from) You've also got better security because you
> *are* validating blocks, starting off incompletely, and increasingly
> completely until your finally validating fully. Privacy is improved, for
> both you and others, by mixing your transactions with others and adding
> to the overall anonymity set.
> 
> In the future we'll have miners commit a hash of the UTXO set, and that
> gives us even more options to, for instance, have relayed transactions
> include proof that their inputs were valid, allowing all nodes to relay
> them safely.


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Michael Gronager]

[-- Attachment #1: Type: text/plain, Size: 3165 bytes --]

Hi Wendell,

What Peter describes (a hash of the current set of UTXOs as part of the coinbase) is already implemented in libcoin, on which you can easily build both a bitcoind and any client. Libcoin is a library originally based on the satoshi client, and as such it is compatible/replacable with "master". 

Have a look at github.com/libcoin/libcoin and look in the BlockChain.h/cpp and the MerkleTrie classes then you can see how it works.

What is missing from libcoin is a scheme to bootstrap the hash of UTXOs, there is some stub code for a p2pool like mining scheme ensuring several UTXO hashes every 10 minutes, but I will not have time to finalize it the first few months - anyone are of course welcome to help out ;)

Michael


On 17/07/2013, at 09:37, Wendell <w@grabhive.com> wrote:

> Peter,
> 
> This sounds like a _very_ good idea for a desktop client, and probably acceptable to users so long as we take available disk space into consideration, and only ever use a fraction of it.
> 
> Will you implement this?
> 
> -wendell
> 
> grabhive.com | twitter.com/grabhive
> 
> On Jul 17, 2013, at 12:58 PM, Peter Todd wrote:
> 
>> So what's useful about that? Basically it means your node starts with
>> the same security level, and usefulness to the network, as a SPV node.
>> But over time you keep downloading blocks as they are created, and with
>> whatever bandwidth you have left (out of some user-configurable
>> allocation) you download additional blocks going further and further
>> back in time. Gradually your UTXO set becomes more complete, and over
>> time you can verify a higher and higher % of all valid transactions.
>> Eventually your node becomes a full node, but in the meantime it was
>> still useful for the user, and still contributed to the network by
>> relaying blocks and an increasingly large subset of all transactions.
>> (optionally you can store a subset of the chain history too for other
>> nodes to bootstrap from) You've also got better security because you
>> *are* validating blocks, starting off incompletely, and increasingly
>> completely until your finally validating fully. Privacy is improved, for
>> both you and others, by mixing your transactions with others and adding
>> to the overall anonymity set.
>> 
>> In the future we'll have miners commit a hash of the UTXO set, and that
>> gives us even more options to, for instance, have relayed transactions
>> include proof that their inputs were valid, allowing all nodes to relay
>> them safely.
> 
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk_______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] SPV bitcoind?

[Andreas Schildbach]

Android apps do whatever they are programmed to do. They become active
when the user installs and inactive when they are uninstalled.
Inbetween, they are not limited in runtime.

That said, the current programming is that when receiving a block, it
stays connected for at least ~2 more minutes. This generally allows the
chain to catch up while at the same time avoiding endless battery drain
because something gets stuck. Upon sending or receiving of a
transaction, it stays connected for at least ~8 more minutes, because it
is likely the wallet will see more activity.

Additionally, on the send and request coins screens and the network
monitor it stays connected for as long as the screen is on and the app
in the foreground (= resumed state).


On 07/17/2013 02:29 PM, Mike Hearn wrote:

> Partial UTXO sets is a neat idea. Unfortunately my intuition is that
> many SPV wallets only remain open for <1 minute at a time because the
> user wants to see they received money, or to send it. It'd be neat to
> get some telemetry from the Android wallet for this - I will ask Andreas
> to let users opt in to usage statistics.

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Wendell]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

"The libcoin/bitcoind client downloads the entire block chain 3.5 times faster than the bitcoin/bitcoind client. This is less than 90 minutes on a modern laptop!"

Good lord Michael, I wish we had known about libcoin a month ago!

-wendell

grabhive.com | twitter.com/grabhive

On Jul 17, 2013, at 4:31 PM, Michael Gronager wrote:

> Hi Wendell,
> 
> What Peter describes (a hash of the current set of UTXOs as part of the coinbase) is already implemented in libcoin, on which you can easily build both a bitcoind and any client. Libcoin is a library originally based on the satoshi client, and as such it is compatible/replacable with "master". 
> 
> Have a look at github.com/libcoin/libcoin and look in the BlockChain.h/cpp and the MerkleTrie classes then you can see how it works.
> 
> What is missing from libcoin is a scheme to bootstrap the hash of UTXOs, there is some stub code for a p2pool like mining scheme ensuring several UTXO hashes every 10 minutes, but I will not have time to finalize it the first few months - anyone are of course welcome to help out ;)
> 
> Michael


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [Bitcoin-development] SPV bitcoind?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

Yeah, what I meant is, it'd be useful to know the average amount of time
that the app was holding connections open for.


On Wed, Jul 17, 2013 at 4:32 PM, Andreas Schildbach
<andreas@schildbach.de>wrote:

> Android apps do whatever they are programmed to do. They become active
> when the user installs and inactive when they are uninstalled.
> Inbetween, they are not limited in runtime.
>
> That said, the current programming is that when receiving a block, it
> stays connected for at least ~2 more minutes. This generally allows the
> chain to catch up while at the same time avoiding endless battery drain
> because something gets stuck. Upon sending or receiving of a
> transaction, it stays connected for at least ~8 more minutes, because it
> is likely the wallet will see more activity.
>
> Additionally, on the send and request coins screens and the network
> monitor it stays connected for as long as the screen is on and the app
> in the foreground (= resumed state).
>
>
> On 07/17/2013 02:29 PM, Mike Hearn wrote:
>
> > Partial UTXO sets is a neat idea. Unfortunately my intuition is that
> > many SPV wallets only remain open for <1 minute at a time because the
> > user wants to see they received money, or to send it. It'd be neat to
> > get some telemetry from the Android wallet for this - I will ask Andreas
> > to let users opt in to usage statistics.
>
>
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 2704 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

Is that still accurate Michael?


On Wed, Jul 17, 2013 at 4:58 PM, Wendell <w@grabhive.com> wrote:

> "The libcoin/bitcoind client downloads the entire block chain 3.5 times
> faster than the bitcoin/bitcoind client. This is less than 90 minutes on a
> modern laptop!"
>
> Good lord Michael, I wish we had known about libcoin a month ago!
>
> -wendell
>
> grabhive.com | twitter.com/grabhive
>
> On Jul 17, 2013, at 4:31 PM, Michael Gronager wrote:
>
> > Hi Wendell,
> >
> > What Peter describes (a hash of the current set of UTXOs as part of the
> coinbase) is already implemented in libcoin, on which you can easily build
> both a bitcoind and any client. Libcoin is a library originally based on
> the satoshi client, and as such it is compatible/replacable with "master".
> >
> > Have a look at github.com/libcoin/libcoin and look in the
> BlockChain.h/cpp and the MerkleTrie classes then you can see how it works.
> >
> > What is missing from libcoin is a scheme to bootstrap the hash of UTXOs,
> there is some stub code for a p2pool like mining scheme ensuring several
> UTXO hashes every 10 minutes, but I will not have time to finalize it the
> first few months - anyone are of course welcome to help out ;)
> >
> > Michael
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 2762 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Michael Gronager]

> Is that still accurate Michael?
> 

The 90 minutes is not - the blockchain has grown quite a lot since last year, and as for the 3.5 speed, I havn't tested it since Pieter's ultraprune - libcoin also has something similar to ultraprune, done directly in the sqlite database backend, but I should run a head to head again - could be fun. I would assume, though, that the result would be similar timings.

However, by having a merkle tree hash of all UTXOs they become downloadable in a trusted manner from any other client - something that enables bootstrap in minutes, so the old numbers becomes less relevant in this setting.

> 
> On Wed, Jul 17, 2013 at 4:58 PM, Wendell <w@grabhive.com> wrote:
> "The libcoin/bitcoind client downloads the entire block chain 3.5 times faster than the bitcoin/bitcoind client. This is less than 90 minutes on a modern laptop!"
> 
> 

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Gregory Maxwell]

On Wed, Jul 17, 2013 at 3:26 PM, Michael Gronager <gronager@mac.com> wrote:
> However, by having a merkle tree hash of all UTXOs they become downloadable in a trusted manner from any other client - something that enables bootstrap in minutes, so the old numbers becomes less relevant in this setting.

This, however, reduces the node to SPV security of the past history.
Particularly for a wallet client— as opposed to a miner or what have
you— if you are willing to accept SPV security you could simply be an
SPV client.

(I like committed UTXO trees, and I believe I was the first person to
suggest them— but I think it's good to not over-hype what they do!)

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

> The 90 minutes is not - the blockchain has grown quite a lot since last
> year, and as for the 3.5 speed, I havn't tested it since Pieter's
> ultraprune - libcoin also has something similar to ultraprune, done
> directly in the sqlite database backend, but I should run a head to head
> again - could be fun. I would assume, though, that the result would be
> similar timings.
>

ultraprune made a huge difference. I think it's very likely that this claim
is no longer true. Bitcoin got a lot more optimised since you first did
libcoin.

[-- Attachment #2: Type: text/html, Size: 770 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Bazyli Zygan]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

Hi!

I should introduce myself. I am the BitcoinKit developer. If you can call that way a dude that wrapped up already existing code for Mac developers easier to understand and use :-)

I'm replying mostly because libcoin is something that I would like to have a closer look at.
Problems I've encountered with it so far are as follows:

1. It uses QT.
Well. It's a lib. Or at least I've thought it was. But it seems that I really need it to compile it. Dunno why yet.

2. Steps to create xcodeproject doesn't work
For some reason when I've tried to follow steps to create an xcodeproject from the cmake, it failed.

3. It doesn't compile at all
Even after installing QT libs and using cmake to compile it from the terminal… it fails on bitcoind.cpp. My assumtion is that cmake or not - it uses llvm to compile the stuff.
Because of the templates that bitcoind is actually using that's not gonna work ever. That's why BitcoinKit is a separate dynamic library that's compiled with gcc (or at least llvm pretending to be gcc ;P)

Michael, have you tried to use your sources on Mac OS X recently? It seems to be a bit… outdated.

/b

[-- Attachment #2: Type: text/html, Size: 2598 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 4155 bytes --]

On Wed, Jul 17, 2013 at 02:29:26PM +0200, Mike Hearn wrote:
> Partial UTXO sets is a neat idea. Unfortunately my intuition is that many
> SPV wallets only remain open for <1 minute at a time because the user wants
> to see they received money, or to send it. It'd be neat to get some
> telemetry from the Android wallet for this - I will ask Andreas to let
> users opt in to usage statistics.

Good idea.

> So for anti-DoS I think smart prioritisation heuristics are the way to go
> again. Perhaps by letting clients have an "identity" that they provide to a
> node when it's load shedding. Clients that have been seen before, have a
> track record of not being abusive etc get priority and new clients that
> were never seen before get dropped. Coming up with a way to do that whilst
> preserving privacy sounds like an interesting cryptographic challenge.

SPV clients behaving normally are highly abusive: they use up maximum
node resources with minimum cost to themselves. (nodes doing an initial
block download are similar now, although with partial mode they can
contribute back to the network sooner)

We can't win if the attacker has more upstream bandwidth than we have
downstream, but fortunately botnets are generally comprised of computers
on asymetric residential connections. Thus our goal is to prevent the
attacker from using lots of downstream bandwidth, and more importantly,
from consuming more memory and similar resources than we posess.
Annoyingly the raw # of TCP connections is very much a limited resource
due to constraints on the # of ports a process can handle, and
constraints imposed by stateful firewalls, and memory used by kernel
buffers.

Anything that allows for more incoming connections with less memory
usage is a good thing - bloom filters are limited to 32KiB and the
per-peer test if a INV item needs to be relayed to a peer is fairly
cheap, but we also have other buffers like pending INV messages and so
on. EC2 micro instances, as an example, often need -maxconnections
limited or they run out of memory - we've probably got room for
improvement; removing mapRelay and just grabbing relayed txs from the
mempool comes to mind.


More generally a good thing to do would be to force incoming peers to
use up RAM to make a connection. We can do that with a proof-of-data
posession engineered such that unless you store the data in high-speed
memory you will have your connection dropped. Per peer a node can pick a
nonce k and define j_i=H(k+i), sending the peer a set J=(j_0...j_n) to
store in RAM. With f(k, n, i) as a pseudo-random sequence generator we
create nonce x and ask our peer to compute J'(x, m) = j_f(x, n, 0) ^ ...
^ j_f(x, n, m)) and give us the result. (^ as the XOR operator) Because
we know the nonce k we can do that cheaply, calculating it on the fly,
but our peers have no choice but to store J and retrieve it on demand.
If they store J in RAM they can do so quickly; if they store J on disk
they can't. We then prioritize peers by how fast they respond to these
requests, both measuring ping times, and forcing attackers trying to
connect to large numbers of peers to posess large amounts of relatively
expensive RAM. This is particularly nice because we've can make it
significantly more expensive for anyone to peer to every node in the
Bitcoin network simultaneously to do things like watch transaction
propagation in real-time.

A more sophisticated approach would be possible if there existed a
version of H() with a computational trap-door - that is if there existed
H'(s, i)=H(i) where H' had significantly faster running time than H(),
but required knowledge of a secret. Our peers would then be able to
answer our challenges quickly only if they stored the intermediate
results in a lookup table, while we could check those challenges cheaply
without that table.

Adam: you're our local crypto-expert, what can we use for H'? Seems that
maybe some kind of asymmetric crypto system would work by requiring the
peer to crack weak secret keys that we generate deterministicly.

-- 
'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Michael Gronager]

Hi Bazyli,

I actually do my main development on Mac OSX, so it surprises me to hear - I build Xcode projects with libcoin daily on Mac OSX and linux, on Windows it is agreeable more of a fight to build. QT is really not needed, I kept it there for BitcoinQT, that was once part of the tree too, will remove it as the qt part got split out.

Building clean on Mac requires OpenSSL, BDB and Boost - all can be installed using homebrew, also remember to use the latest cmake, and a normal cmake xcode call: cmake -GXcode should do the job. Otherwise pls send me the debug output. 

A few quick notes for building stuff there:
 - try with coinexplorer, it is the base code I am using - it splits out the wallet from the server, nice if you e.g. want to build a webcoin like server.
 - The wallet parts from bitcoind I don't use personally, so if you have problems with these I need to have a closer look.

Also note that as the first version of libcoin was a direct refactorization of bitcoin, the current one add a lot of different features and handles things quite differently - you can e.g. lookup any unspent output by script (bitcoin address) in milliseconds (nice for web wallets).

Finally: 

> 	Because of the templates that bitcoind is actually using that's not gonna work ever. That's why BitcoinKit is a separate dynamic library that's compiled with gcc (or at least llvm pretending to be gcc ;P)

As I mentioned it also compiles on Linux (gcc) - gcc is quite savvy when it comes to templates - I agree that the template stuff from Database.h is quite involved, but as I mentioned before try with coinexplorer.

- I will try to do a from scratch recompilation to see if I experience similar issues...

Also - if you are good at creating frameworks on Mac OSX using cmake, help would be appreciated! I think that libcoin by defaults build using shared libs, this configurable from ccmake using the dynamic library option.

Thanks,

Michael

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Michael Gronager]

Hi Bazyli,

Just did a fresh build based on git (Xcode) - had one issue: the paillier and account tests were missing - please comment them out in tests/CMakeLists.txt, then coinexplorer should build nicely.

Note I did a git push as well, so you need to do a git pull first.

/Michael

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

On Thu, Jul 18, 2013 at 08:13:08AM -0400, Peter Todd wrote:
> A more sophisticated approach would be possible if there existed a
> version of H() with a computational trap-door - that is if there existed
> H'(s, i)=H(i) where H' had significantly faster running time than H(),
> but required knowledge of a secret. Our peers would then be able to
> answer our challenges quickly only if they stored the intermediate
> results in a lookup table, while we could check those challenges cheaply
> without that table.
> 
> Adam: you're our local crypto-expert, what can we use for H'? Seems that
> maybe some kind of asymmetric crypto system would work by requiring the
> peer to crack weak secret keys that we generate deterministicly.

Actually, come to think of it a really easy way to create H' is for the
node to create some expensive to compute set of data associated with
their identity. The data set is then stored once by the node, cheap, but
the clients have to store one set for every unique node they connect
too, expensive. A set of the function scrypt(k | i) for i in 0..n is an
obvious way to do it.

This can equally be used as a proof-of-work to make creating lots of
nodes expensive given a cheap way to verify the POW; easily done with a
non-interactive zero-knowledge proofs. It'd be nice if that POW could
incorporate blockchain data, showing that the identity had access to
that data and thus could have computed the UTXO set honestly. (the POW
should be incrementally extendable as new data becomes available)
However that is back to using a bunch of bandwidth at startup if our
peer doesn't have access to blockchain data, so both mechanisms would
probably have to be done independently. Note how we also make MITM
attacks on encrypted P2P connections expensive this way too even without
any form of authentication. (works best when the proof-of-work is
dependent on your IP addresses)

-- 
'peter'[:-1]@petertodd.org
00000000000000762784b647ede3678f172d73dd0c72c2180ab451b00d756959

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

> SPV clients behaving normally are highly abusive: they use up maximum
> node resources with minimum cost to themselves.
>

This must be a new use of the word "abuse" I haven't come across before :)

At any rate, some of these assumptions are incorrect. Botnets of
compromised web servers are quite common, and asymmetry in node resources
is obviously biased against the kinds of devices people increasingly have
(phones, tablets) where extremely limited memory bandwidth is common and
apps routinely have just 16 or 32mb of memory to do everything including
the GUI.

A good anti-DoS strategy looks much the same as a good load shedding
strategy. There's little reason to treat them separately. Perhaps instead
of talking about DoS we should instead talk about what happens if Bitcoin
suddenly gets too popular. Now there are suddenly lots of good users all
wanting to use the network, and not enough nodes to support them all. What
do we do?

Some rules seem obvious - try to prioritise existing users over new users,
old coins over new coins (dPriority already does this) etc. If you run out
of TCP sockets prefer to disconnect recent connections (probably new users)
to long lived connections (probably high powered backbone peers). If you
run out of disk seeks prefer processing new blocks to serving old parts of
the chain, etc.

[-- Attachment #2: Type: text/html, Size: 1664 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

On Wed, Jul 17, 2013 at 03:37:41PM +0200, Wendell wrote:
> Peter,
> 
> This sounds like a _very_ good idea for a desktop client, and probably acceptable to users so long as we take available disk space into consideration, and only ever use a fraction of it.
> 
> Will you implement this?

I've got one or two orders of magnitude more good ideas than I have time
to implement, but I will say this one would have a pretty big impact -
I'm considering it.

Of course, I would accept bribes. :) But in all seriousness I also
accepted funds from John Dillon to implement replace-by-fee, although
he's been good in understanding that the scope of the project was quite
a bit bigger than originally thought. (it turned out replace-by-fee can
enable very safe zero-conf transactions, but only with mempool and
relaying changes) I'd suggest looking at my git commit track record
before you offer anything FWIW; I've been much more of an academic than
a programmer.

-- 
'peter'[:-1]@petertodd.org
0000000000000013030f49fe3eed5e7f9388c4ecc237b7a847ca93255836bc3b

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Wendell]

[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]

Heh, will do. If you have less confidence in your programming skills perhaps its best if you write documentation and we bring in someone else to do the heavy lifting? Maybe Eric Lombrozo would be interested in this, for example...

-wendell

grabhive.com | twitter.com/grabhive

On Jul 18, 2013, at 6:22 PM, Peter Todd wrote:

> I've got one or two orders of magnitude more good ideas than I have time
> to implement, but I will say this one would have a pretty big impact -
> I'm considering it.
> 
> Of course, I would accept bribes. :) But in all seriousness I also
> accepted funds from John Dillon to implement replace-by-fee, although
> he's been good in understanding that the scope of the project was quite
> a bit bigger than originally thought. (it turned out replace-by-fee can
> enable very safe zero-conf transactions, but only with mempool and
> relaying changes) I'd suggest looking at my git commit track record
> before you offer anything FWIW; I've been much more of an academic than
> a programmer.


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [Bitcoin-development] SPV bitcoind? (was: Introducing BitcoinKit.framework)

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

On Thu, Jul 18, 2013 at 06:46:16PM +0200, Wendell wrote:
> Heh, will do. If you have less confidence in your programming skills perhaps its best if you write documentation and we bring in someone else to do the heavy lifting? Maybe Eric Lombrozo would be interested in this, for example...

I have plenty of confidence in my programming skills, I just don't have
very much evidence in the Bitcoin git history to convince you my
confidence is well placed. :)

I do have a day job I love, so it will certainly get done faster if you
can get someone else to do the actual coding; I'd be willing to write
the specifications and supervise/audit/advise for a few hours a week.

> -wendell
> 
> grabhive.com | twitter.com/grabhive
> 
> On Jul 18, 2013, at 6:22 PM, Peter Todd wrote:
> 
> > I've got one or two orders of magnitude more good ideas than I have time
> > to implement, but I will say this one would have a pretty big impact -
> > I'm considering it.
> > 
> > Of course, I would accept bribes. :) But in all seriousness I also
> > accepted funds from John Dillon to implement replace-by-fee, although
> > he's been good in understanding that the scope of the project was quite
> > a bit bigger than originally thought. (it turned out replace-by-fee can
> > enable very safe zero-conf transactions, but only with mempool and
> > relaying changes) I'd suggest looking at my git commit track record
> > before you offer anything FWIW; I've been much more of an academic than
> > a programmer.

-- 
'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Pieter Wuille]

On Tue, Jul 16, 2013 at 12:59:56PM +0200, Mike Hearn wrote:
> I think that's a great approach. Here is the patch Satoshi sent me back in
> 2010. All the code has changed since but it can be a source of inspiration.
> 
> >From Satoshi:
> 
> *The simplified payment verification in the paper imagined you would
> receive transactions directly, as with sending to IP address which nobody
> uses, or a node would index all transactions by public key and you could
> download them like downloading mail from a mail server.

I'm currently working on headers-first sync, which I believe is generally
very useful (it fixes tons of edge-cases block synchronization currently
experiences), but it's also a first step towards SPV mode.

So headers-first sync means you first synchronize just the headers, and then,
when you already know (or have strong evidence for a guess on) the best chain,
start requesting blocks along that best chain - potentially in parallel from
different peers.

SPV mode is basically headers-first sync, but never do the full block sync
step, and replace it with a bloom/birthday/...-based fetching of blocks
interesting to the associated wallets. In SPV you'll also need to disable
the mempool though, and there will be more small changes, but I think
the separate headers-sync phase will be most of the work.

-- 
Pieter

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1948 bytes --]

Actually bitcoinj typically doesn't download all the headers (just from the
last checkpoint) and it throws away headers that are very old. By now
there's quite a lot of difference in how they manage things and I guess it
will diverge from bitcoind even more in future. For instance we're going to
start only storing relevant outputs in the wallet and doing other things to
try and save memory. Some people managed to get themselves wallets that
don't actually fit in ram :(
On 21 Jul 2013 17:55, "Pieter Wuille" <pieter.wuille@gmail.com> wrote:

> On Tue, Jul 16, 2013 at 12:59:56PM +0200, Mike Hearn wrote:
> > I think that's a great approach. Here is the patch Satoshi sent me back
> in
> > 2010. All the code has changed since but it can be a source of
> inspiration.
> >
> > >From Satoshi:
> >
> > *The simplified payment verification in the paper imagined you would
> > receive transactions directly, as with sending to IP address which nobody
> > uses, or a node would index all transactions by public key and you could
> > download them like downloading mail from a mail server.
>
> I'm currently working on headers-first sync, which I believe is generally
> very useful (it fixes tons of edge-cases block synchronization currently
> experiences), but it's also a first step towards SPV mode.
>
> So headers-first sync means you first synchronize just the headers, and
> then,
> when you already know (or have strong evidence for a guess on) the best
> chain,
> start requesting blocks along that best chain - potentially in parallel
> from
> different peers.
>
> SPV mode is basically headers-first sync, but never do the full block sync
> step, and replace it with a bloom/birthday/...-based fetching of blocks
> interesting to the associated wallets. In SPV you'll also need to disable
> the mempool though, and there will be more small changes, but I think
> the separate headers-sync phase will be most of the work.
>
> --
> Pieter
>
>

[-- Attachment #2: Type: text/html, Size: 2342 bytes --]

Re: [Bitcoin-development] Introducing BitcoinKit.framework

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

As an FYI, I've sent Wendell and co some example code for how to use CPPJVM
to use bitcoinj from native code. A rather rough Hello World app looks like
this:

https://github.com/mikehearn/cppjvm/blob/master/mytest/bcj-hello-world.cpp

So, fairly C++ like.

Further discussion of this should take place on the bitcoinj mailing list.

[-- Attachment #2: Type: text/html, Size: 525 bytes --]