[Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 1245 bytes --]

Hello,

This is primarily aimed at developers of SPV wallets.

The recently reported decrease in number of full nodes could have several
reasons, one of them that less people are running Bitcoin Core for the
wallet because the other wallets are getting ahead in both features and
useability.

It's great to see innovation in wallets, but it's worrying that the number
of full nodes decreases.

It may be that lots of people would support the network by running a full
node, but don't want to go through the trouble of installing bitcoin core
separately (and get confused because it's a wallet, too).

Hence I'd like to explore the idea of adding an option to popular SPV
wallets, to spin a bitcoind process in the background. This could be pretty
much transparent to the user - it would sync in the background, the wallet
could show statistics about the node, but is not dependent on it.

In exchange the user would get increased (full node level) security, as the
SPV wallet would have a local trusted node.

Does this sound like a good idea?

Is there any way that Bitcoin Core can help to accomedate this 'embedded'
usage? Specific Interfaces, special builds - maybe add a walletless
bitcoind build to gitian - bindings, dlls, etc?

Wladimir

[-- Attachment #2: Type: text/html, Size: 1465 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 2539 bytes --]

YES

Such a bitcoind is what I called border router in a previous mail. 

Yes, SPV wallets are getting ahead of features, so people will use them also because on size just does not fit all, but all want to ensure being on the same trunk of the chain.
Therefore serious user of Bitcoin run a bitcoind as a border router and connect SPV wallets with higher functionality to that trusted node(s).

This is what I think the core should focus on: Being a lightweight superfast consensus building border router and nothing more. No wallet, no GUI, no RPC calls,
no Payment protocol and the rest.

Regards,

Tamas Blummer
http://bitsofproof.com

On 09.04.2014, at 17:29, Wladimir <laanwj@gmail.com> wrote:

> Hello,
> 
> This is primarily aimed at developers of SPV wallets.
> 
> The recently reported decrease in number of full nodes could have several reasons, one of them that less people are running Bitcoin Core for the wallet because the other wallets are getting ahead in both features and useability.
> 
> It's great to see innovation in wallets, but it's worrying that the number of full nodes decreases. 
> 
> It may be that lots of people would support the network by running a full node, but don't want to go through the trouble of installing bitcoin core separately (and get confused because it's a wallet, too).
> 
> Hence I'd like to explore the idea of adding an option to popular SPV wallets, to spin a bitcoind process in the background. This could be pretty much transparent to the user - it would sync in the background, the wallet could show statistics about the node, but is not dependent on it.
> 
> In exchange the user would get increased (full node level) security, as the SPV wallet would have a local trusted node.
> 
> Does this sound like a good idea?
> 
> Is there any way that Bitcoin Core can help to accomedate this 'embedded' usage? Specific Interfaces, special builds - maybe add a walletless bitcoind build to gitian - bindings, dlls, etc?
> 
> Wladimir
> 
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment 
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees_______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #1.2: Type: text/html, Size: 6326 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Natanael]

This could probably be done fairly easily by bundling Stratum (it's
not just for pools!) and allowing SPV wallets to ask Bitcoind to start
it (if you don't use it, there's no need to waste the resources), and
then connect to it. The point of using Stratum is that it already is
being used by Electrum, and that it might be an easier way to support
SPV clients than creating a new API in bitcoind for it since Stratum
itself already relies on bitcoind to provide it's services.

On Wed, Apr 9, 2014 at 5:29 PM, Wladimir <laanwj@gmail.com> wrote:
> Hello,
>
> This is primarily aimed at developers of SPV wallets.
>
> The recently reported decrease in number of full nodes could have several
> reasons, one of them that less people are running Bitcoin Core for the
> wallet because the other wallets are getting ahead in both features and
> useability.
>
> It's great to see innovation in wallets, but it's worrying that the number
> of full nodes decreases.
>
> It may be that lots of people would support the network by running a full
> node, but don't want to go through the trouble of installing bitcoin core
> separately (and get confused because it's a wallet, too).
>
> Hence I'd like to explore the idea of adding an option to popular SPV
> wallets, to spin a bitcoind process in the background. This could be pretty
> much transparent to the user - it would sync in the background, the wallet
> could show statistics about the node, but is not dependent on it.
>
> In exchange the user would get increased (full node level) security, as the
> SPV wallet would have a local trusted node.
>
> Does this sound like a good idea?
>
> Is there any way that Bitcoin Core can help to accomedate this 'embedded'
> usage? Specific Interfaces, special builds - maybe add a walletless bitcoind
> build to gitian - bindings, dlls, etc?
>
> Wladimir
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Brian Hoffman]

[-- Attachment #1: Type: text/plain, Size: 2064 bytes --]

How would this affect the user in terms of disk storage? They're going to
get hammered on space constraints aren't they? If it's not required how
likely are users to enable this?


On Wed, Apr 9, 2014 at 11:29 AM, Wladimir <laanwj@gmail.com> wrote:

> Hello,
>
> This is primarily aimed at developers of SPV wallets.
>
> The recently reported decrease in number of full nodes could have several
> reasons, one of them that less people are running Bitcoin Core for the
> wallet because the other wallets are getting ahead in both features and
> useability.
>
> It's great to see innovation in wallets, but it's worrying that the number
> of full nodes decreases.
>
> It may be that lots of people would support the network by running a full
> node, but don't want to go through the trouble of installing bitcoin core
> separately (and get confused because it's a wallet, too).
>
> Hence I'd like to explore the idea of adding an option to popular SPV
> wallets, to spin a bitcoind process in the background. This could be pretty
> much transparent to the user - it would sync in the background, the wallet
> could show statistics about the node, but is not dependent on it.
>
> In exchange the user would get increased (full node level) security, as
> the SPV wallet would have a local trusted node.
>
> Does this sound like a good idea?
>
> Is there any way that Bitcoin Core can help to accomedate this 'embedded'
> usage? Specific Interfaces, special builds - maybe add a walletless
> bitcoind build to gitian - bindings, dlls, etc?
>
> Wladimir
>
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 2860 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mark Friedenbach]

On 04/09/2014 09:09 AM, Tamas Blummer wrote:
> Yes, SPV is a sufficient API to a trusted node to build sophisticated
> features not offered by the core.
> SPV clients of the border router will build their own archive and
> indices based on their interest of the chain therefore the
> border router core does not need to store (and process) anything not
> needed for consensus, its memory
> or disk footprint would be as low as an optimal storage of UTXO.

Storing zero full blocks does nothing to aid the network.

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Gregory Maxwell]

On Wed, Apr 9, 2014 at 8:41 AM, Natanael <natanael.l@gmail.com> wrote:
> This could probably be done fairly easily by bundling Stratum (it's
> not just for pools!) and allowing SPV wallets to ask Bitcoind to start
> it (if you don't use it, there's no need to waste the resources), and
> then connect to it. The point of using Stratum is that it already is
> being used by Electrum,

Sadly today Electrum requires more than a full node, it requires a
number of large additional indexes over what a full node has and
pruning is precluded. I don't think that increasing the resource
utilization of the node is a good way to go there for the purposes
expressed here. (not that electrum couldn't be used here, but not
unmodified without the resource usage increasing route)

> and that it might be an easier way to support
> SPV clients than creating a new API in bitcoind for it since Stratum
> itself already relies on bitcoind to provide it's services.

Bitcoin's own P2P protocol is already the API for a ordinary SPV
client. So I don't believe any new API would be require, except
perhaps for some process management stuff (which also isn't provided
for Electrum).

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Gregory Maxwell]

On Wed, Apr 9, 2014 at 8:42 AM, Brian Hoffman <brianchoffman@gmail.com> wrote:
> How would this affect the user in terms of disk storage? They're going to
> get hammered on space constraints aren't they? If it's not required how
> likely are users to enable this?

If Bitcoin core activates pruning a full node can be supported in—
say— 4GBytes or so. (That gives enough space to store the utxo about
350MB now, and a couple gigs for blocks to serve out).

I'd imagine getting information from SPV wallet developers how much
disk usage agility they think is required is part of what Wladimir is
looking for.

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Two talking points for said developers yo their user re: "Why use a full node?":

1) It's more private. Bloom filters gives away quite accurate statistical information about what coins you own to whom ever you happen to be connected too. An attacker can easily use this to deanonymize you even if you don't reuse addresses; Tor does not help much against this attack.

2) It's more secure. SPV means you are trusting miners to do validation for you. With the extremely high degree of mining centralisation we currently have it would only take one or two pools getting hacked for an attacker to be able to get enough hashing power to easily fool your SPV wallet into accepting a fake transaction.


As for what we can offer those developers, partial UTXO set mode would be a great long term goal.

On 9 April 2014 11:29:32 GMT-04:00, Wladimir <laanwj@gmail.com> wrote:
>Hello,
>
>This is primarily aimed at developers of SPV wallets.
>
>The recently reported decrease in number of full nodes could have
>several
>reasons, one of them that less people are running Bitcoin Core for the
>wallet because the other wallets are getting ahead in both features and
>useability.
>
>It's great to see innovation in wallets, but it's worrying that the
>number
>of full nodes decreases.
>
>It may be that lots of people would support the network by running a
>full
>node, but don't want to go through the trouble of installing bitcoin
>core
>separately (and get confused because it's a wallet, too).
>
>Hence I'd like to explore the idea of adding an option to popular SPV
>wallets, to spin a bitcoind process in the background. This could be
>pretty
>much transparent to the user - it would sync in the background, the
>wallet
>could show statistics about the node, but is not dependent on it.
>
>In exchange the user would get increased (full node level) security, as
>the
>SPV wallet would have a local trusted node.
>
>Does this sound like a good idea?
>
>Is there any way that Bitcoin Core can help to accomedate this
>'embedded'
>usage? Specific Interfaces, special builds - maybe add a walletless
>bitcoind build to gitian - bindings, dlls, etc?
>
>Wladimir
>
>
>------------------------------------------------------------------------
>
>------------------------------------------------------------------------------
>Put Bad Developers to Shame
>Dominate Development with Jenkins Continuous Integration
>Continuously Automate Build, Test & Deployment
>Start a new project now. Try Jenkins in the cloud.
>http://p.sf.net/sfu/13600_Cloudbees
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Bitcoin-development mailing list
>Bitcoin-development@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bitcoin-development
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRW88MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhUIBB/4tVMqj7NEHFekUgv7O
hKgUNreD5RQbVLjiG9BIKWTdOrD79qNk66QVxuH7oOCypMLCdT2ctdvsvfH8ncdG
GVpTcbXL39vEk9qkGN4wY2++pKn5unqKg1C99YbJAcC62XMFJjjES2XiwyKGsYmb
IplOEN7BXh9KE01f7gPh4nZqtqsb6XnlLQGjjRWGlX/oIMphUOdfMtBdWql59bx3
wnHUDc1pAv2bCcyiUI/nOoyNyZixZC16wXoM2pnAJ9PhPbd79lmiKVmjNmrqhu17
HIHlKXk6YD8JVlHD8NoaC1GUwG39jmt5nk1vsfvvesXF+c0VqctVNIitVkvqLNj+
mwXd
=4b5H
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Thomas Voegtlin]

Le 09/04/2014 17:54, Gregory Maxwell a écrit :

> Sadly today Electrum requires more than a full node, it requires a
> number of large additional indexes over what a full node has and
> pruning is precluded. I don't think that increasing the resource
> utilization of the node is a good way to go there for the purposes
> expressed here. (not that electrum couldn't be used here, but not
> unmodified without the resource usage increasing route)
>

Electrum uses two large indexes:

     address -> utxo

(patricia tree, aka "ultimate blockchain compression", see thread 
started by Alan Reiner in the bitcointalk forum)

     address -> spent history

The first index is not going to grow larger than what bitcoind already 
needs to store, because bitcoind will always need to store utxos.

The second index threatens to become large. However, Electrum servers do 
not keep the full histories, they prune older entries. Without adapting 
Electrum clients, it would even be possible to keep only one bit per 
address (to know whether that address has been used or not), and that 
information is only used to restore wallets from seed, not during normal 
operations.

If the first index (patricia tree) was implemented in bitcoind, that 
would obviously be a big relief for electrum servers.



>> and that it might be an easier way to support
>> SPV clients than creating a new API in bitcoind for it since Stratum
>> itself already relies on bitcoind to provide it's services.
>
> Bitcoin's own P2P protocol is already the API for a ordinary SPV
> client. So I don't believe any new API would be require, except
> perhaps for some process management stuff (which also isn't provided
> for Electrum).
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 1772 bytes --]

I am glad that SPV wallets are discussed outside the scope of mobile devices!

Yes, SPV is a sufficient API to a trusted node to build sophisticated features not offered by the core.
SPV clients of the border router will build their own archive and indices based on their interest of the chain therefore the
border router core does not need to store (and process) anything not needed for consensus, its memory
or disk footprint would be as low as an optimal storage of UTXO.

Regards,

Tamás Blummer
http://bitsofproof.com

On 09.04.2014, at 17:57, Gregory Maxwell <gmaxwell@gmail.com> wrote:

> On Wed, Apr 9, 2014 at 8:42 AM, Brian Hoffman <brianchoffman@gmail.com> wrote:
>> How would this affect the user in terms of disk storage? They're going to
>> get hammered on space constraints aren't they? If it's not required how
>> likely are users to enable this?
> 
> If Bitcoin core activates pruning a full node can be supported in—
> say— 4GBytes or so. (That gives enough space to store the utxo about
> 350MB now, and a couple gigs for blocks to serve out).
> 
> I'd imagine getting information from SPV wallet developers how much
> disk usage agility they think is required is part of what Wladimir is
> looking for.
> 
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment 
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #1.2: Type: text/html, Size: 5127 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 939 bytes --]

A border router that is not able to serve blocks is still protecting consensus rules, that SPVs do not.
If the network would only consist of SPV nodes only then e.g. a majority coalition of miner could increase their reward at will.

Archives need a different solution.

Regards,

Tamas Blummer
http://bitsofproof.com

On 09.04.2014, at 17:47, Mark Friedenbach <mark@monetize.io> wrote:

> On 04/09/2014 09:09 AM, Tamas Blummer wrote:
>> Yes, SPV is a sufficient API to a trusted node to build sophisticated
>> features not offered by the core.
>> SPV clients of the border router will build their own archive and
>> indices based on their interest of the chain therefore the
>> border router core does not need to store (and process) anything not
>> needed for consensus, its memory
>> or disk footprint would be as low as an optimal storage of UTXO.
> 
> Storing zero full blocks does nothing to aid the network.


[-- Attachment #1.2: Type: text/html, Size: 4319 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Wed, Apr 9, 2014 at 5:41 PM, Natanael <natanael.l@gmail.com> wrote:

> This could probably be done fairly easily by bundling Stratum (it's
> not just for pools!) and allowing SPV wallets to ask Bitcoind to start
> it (if you don't use it, there's no need to waste the resources), and
> then connect to it. The point of using Stratum is that it already is
> being used by Electrum, and that it might be an easier way to support
> SPV clients than creating a new API in bitcoind for it since Stratum
> itself already relies on bitcoind to provide it's services.
>

Why would a new API be needed? (beside maybe some functionality that would
make it easier to integrate)

P2P should be enough for SPV clients such as BitcoinJ to get access to
(filtered) blocks and transations, and RPC can be used to manage/query the
bitcoind instance. I'm not sure what stratum would add.

Wladimir

[-- Attachment #2: Type: text/html, Size: 1280 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Alex Mizrahi]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

>
> 1) It's more private. Bloom filters gives away quite accurate statistical
> information about what coins you own to whom ever you happen to be
> connected too. An attacker can easily use this to deanonymize you even if
> you don't reuse addresses; Tor does not help much against this attack.
>

There is also an option to download everything, but do only a very basic
surface validation (without keeping track of UTXOs).
You do not need a full node for that.

[-- Attachment #2: Type: text/html, Size: 700 bytes --]

[Bitcoin-development] Fwd: Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

On Wed, Apr 9, 2014 at 5:42 PM, Brian Hoffman <brianchoffman@gmail.com>wrote:

> How would this affect the user in terms of disk storage? They're going to
> get hammered on space constraints aren't they? If it's not required how
> likely are users to enable this?
>

If they make the (conscious) choice to run a full node they will require
the bandwidth and disk space to run it.

The difference with running Bitcoin Core as wallet will be that they can
choose their own wallet to go with the full node.

Wladimir

[-- Attachment #2: Type: text/html, Size: 1110 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

On Wed, Apr 9, 2014 at 7:33 PM, Alex Mizrahi <alex.mizrahi@gmail.com> wrote:

> 1) It's more private. Bloom filters gives away quite accurate statistical
>> information about what coins you own to whom ever you happen to be
>> connected too. An attacker can easily use this to deanonymize you even if
>> you don't reuse addresses; Tor does not help much against this attack.
>>
>
> There is also an option to download everything, but do only a very basic
> surface validation (without keeping track of UTXOs).
> You do not need a full node for that.
>

You may not *need* a full node, but the point of this (which I clearly
explained in my opening post) would be to support the network.

Wladimir

[-- Attachment #2: Type: text/html, Size: 1280 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 9 April 2014 13:33:19 GMT-04:00, Alex Mizrahi <alex.mizrahi@gmail.com> wrote:
>>
>> 1) It's more private. Bloom filters gives away quite accurate
>statistical
>> information about what coins you own to whom ever you happen to be
>> connected too. An attacker can easily use this to deanonymize you
>even if
>> you don't reuse addresses; Tor does not help much against this
>attack.
>>
>
>There is also an option to download everything, but do only a very
>basic
>surface validation (without keeping track of UTXOs).
>You do not need a full node for that.

Yup, that's part of the idea behind partial UTXO set mode. You could have a model where your node starts with no data at all, and hence SPV security. You tell your node what the oldest key birthday is that your interested in and it downloads the full block chain starting at that date, giving you your txs w/ SPV security and full node privacy.

What partial UTXO would add on top of that is then your node would gradually scan backwards until block zero, at which point it has a complete UTXO set and is a full node.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRYWUMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhaHEB/0Q0P544nJrtapnzogG
QpREeBlwR5I8fZ+BCbNcAk6afk9KLOFIsLBW0bFvFXfSO1TiyCLMezLkvhuDC5lM
vrCBKdmaXhwMaTGppgFNl9adu8yBRQ6I2BN21hQejyfmCvHTsoN6u1oeYccL+K+D
lTK2dMCmXLixcQIJjS9sGA8/TXjmiY65RzIdXl0UHeV4JUtRREMUB1HeXJWCxMEt
4KQXb5tk21vZ+BTf9kRirnCtFT+Wudyo73M2XGKbR3mzHc2jY+RjEb71J0hUmh8s
h4kVhuMMkEFjNUwU017kEYQpmXMMRwCyUydc7BoV6hZXnUx4ROTpc6pDv2IZkeLB
bGiE
=adiR
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 9 April 2014 12:27:13 GMT-04:00, Tamas Blummer <tamas@bitsofproof.com> wrote:
>A border router that is not able to serve blocks is still protecting
>consensus rules, that SPVs do not.
>If the network would only consist of SPV nodes only then e.g. a
>majority coalition of miner could increase their reward at will.
>
>Archives need a different solution.

Any collective group that has a majority of hashing power will have no major issues running enough nodes that follow their rules to make SPV insecure anyway.

There's no good reason not to have SPV security nodes distribute block chain data, particularly block headers. It helps provide redundancy in the network topology and helps provide more resources for full nodes to sync up faster. For instance in a network with a large number of partial UTXO set nodes if those nodes are forwarding block data to each other they can get enough data to become fully fledged full nodes without putting all the load on the existing full nodes.  This is a good thing.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRYdYMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhdwRB/46MAw7OHwnVkLHOD0g
Y4X6p1/QHgRisJIgpG2Y4nGVeAjOFleQWe4PWS4Wwdr4u0BDGPmJompiR3A99CaL
MUPnxJhtdiUsomn6kI704f5pkrqslQGLzejWFb7/9WuQtvGm8RwnzIs7uAqKasni
KJMn3jmqFIUcCEy9dePUdmMySCQj8qSxjGDdwZnwf8BZSdSLqd8dYiN0jQi/aA1I
2hWWyyDK9V9yQ8peAg+1dfg754Tc76lj3mEQOD39Wu3Klb9mBF3+wQCW2tJYEj2E
stzeOdFZsUNUIOKFb6mo0IoUipPOvrAkfm91ais+FIwlCf+k5KcwmAUXpV45rLHm
egCr
=2vMf
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 1477 bytes --]

Block header has to be available in SPV and also in an UTXO only storing core node, so why not serve it if bandwith allows.

Serving any additional information like known peer adresses or known full blocks is certainly beneficial and should be offered if at hand.

Regards,

Tamas Blummer
http://bitsofproof.com

On 09.04.2014, at 19:46, Peter Todd <pete@petertodd.org> wrote:

> Signed PGP part
> 
> 
> On 9 April 2014 12:27:13 GMT-04:00, Tamas Blummer <tamas@bitsofproof.com> wrote:
> >A border router that is not able to serve blocks is still protecting
> >consensus rules, that SPVs do not.
> >If the network would only consist of SPV nodes only then e.g. a
> >majority coalition of miner could increase their reward at will.
> >
> >Archives need a different solution.
> 
> Any collective group that has a majority of hashing power will have no major issues running enough nodes that follow their rules to make SPV insecure anyway.
> 
> There's no good reason not to have SPV security nodes distribute block chain data, particularly block headers. It helps provide redundancy in the network topology and helps provide more resources for full nodes to sync up faster. For instance in a network with a large number of partial UTXO set nodes if those nodes are forwarding block data to each other they can get enough data to become fully fledged full nodes without putting all the load on the existing full nodes.  This is a good thing.
> 
> 
> 


[-- Attachment #1.2: Type: text/html, Size: 5165 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

The right way to start with this, if anyone cares, is to add
instrumentation to existing SPV wallet apps to report back to home base how
long they are running for, how much disk space / RAM they have, and
possibly what kind of hardware.

I *strongly* suspect that the vast majority of SPV wallets are not left
running permanently, and run on laptops where battery life is at a premium.
These people will never want to run full nodes.

Sorry. I don't think it will ever make sense to run full nodes on consumer
hardware again. Our time is much better spent on optimising so it's cheaper
for full node operators to run them on cheap virtualised servers.

[-- Attachment #2: Type: text/html, Size: 853 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 9 April 2014 13:50:03 GMT-04:00, Tamas Blummer <tamas@bitsofproof.com> wrote:
>Block header has to be available in SPV and also in an UTXO only
>storing core node, so why not serve it if bandwith allows.
>
>Serving any additional information like known peer adresses or known
>full blocks is certainly beneficial and should be offered if at hand.

Big security advantages too. For instance if an attacker hacks, say, 10℅ of hashing power the next step for them to attack SPV clients is to try to Sybil attack them so they won't find out about the longer chain. The fewer providers of block chain data there are out there the easier that attack is - just simultaneously DoS a bunch of nodes, perhaps by a low-bandwidth exploit like the bloom io or division by zero DoS attacks. This is much harder to pull off if every SPV client is passing around block headers.

Similarly by passing around full blocks the attacker has a harder time knocking other miners off the network. Regardless of whether or not a miner's peers are fully validating chain data they still have the data they need to mine the next block and thus extend the longest correct chain.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRYu+MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwheuQCADUyClLOLP1xpG1000l
uzcfPTZuIXTpzOAmYHKs/MSb6mph/Shsu0/94eW7npQNSVeZC8wQQZ1oFQ9j1GJc
SKViYJfn5yMdNvMkaWazhC0r3jxxF0AI7oy2KlnSjasfczfOQuYICJadTCwvUHrb
GrKVDbgsKNzZYYKn86vF4hsLwtJN4moeqX85TYN1DC7//7hgNywA73Xt2/gdwfqe
LOsD4nS7mUQObQd6TcLwXDDNEGTrdS572jdYH5sykwZjPH+wqwcm2WKTnIULsJR0
OwGUi505AKJJnLcEmZ/kGbCmKB+xJ5kExjlExtcUPrJlc+xqubhnnGCMjBiGCXSY
kYCK
=HXRZ
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]

On Wed, Apr 9, 2014 at 8:00 PM, Mike Hearn <mike@plan99.net> wrote:

> The right way to start with this, if anyone cares, is to add
> instrumentation to existing SPV wallet apps to report back to home base how
> long they are running for, how much disk space / RAM they have, and
> possibly what kind of hardware.
>
> I *strongly* suspect that the vast majority of SPV wallets are not left
> running permanently, and run on laptops where battery life is at a premium.
> These people will never want to run full nodes.
>

Bitcoins stands or falls with people running full nodes.

If no one wants to volunteer resources to support the network anymore,
we'll have failed.

Sorry. I don't think it will ever make sense to run full nodes on consumer
> hardware again. Our time is much better spent on optimising so it's cheaper
> for full node operators to run them on cheap virtualised servers.
>

Most consumer hardware is much more powerful than 'cheap virtualized
servers'. More memory, disks are cheap, and at least in the Netherlands
home bandwidth is much cheaper than server bandwidth.

Also: any optimization that helps running on cheap servers will also help
running it on consumer hardware. It's not the one or the other.

Wladimir

[-- Attachment #2: Type: text/html, Size: 2026 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Justus Ranvier]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/09/2014 06:19 PM, Wladimir wrote:
> If no one wants to volunteer resources to support the network
> anymore, we'll have failed.

If the security of the network depends on a broken incentive model,
then fix the design of the network so that economics works for you
instead of against you.



- -- 
Support online privacy by using email encryption whenever possible.
Learn how here: http://www.youtube.com/watch?v=bakOKJFtB-k
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTRZLiAAoJECoisBQbQ4v0RRMIAKL84ju4b9XAls9sOxQGrLMr
xifQDhThCQKC4/qkOmGU8zseIdKRgXEq4auMF9/0BlgoSxsBcKynlRH2fFtYY7ol
sdjCy/5dk+MBu3K1GCvNn/cuGkpIJJxrom/9riSiaPtivE5ncl7cyW5hFqf2MzRd
dF6q937ocgBd8d2VuOQleQ9N5gue1+du77soxfp8oY7dXNdwk5ZrngeYxz1umsjQ
lBAI/3JYkKVhU5Rte3deJcMHe6xA+neIzvcfWbOZYrkdwWE+jLSKnDkDkCOXJnuP
vOI8CAteaFctV2mfgXX2CmNDWVeFsyCwoQwbnCmuE/uiM5y235PcTrsyr6U+Zzs=
=oQPK
-----END PGP SIGNATURE-----

[-- Attachment #2: 0x1B438BF4.asc --]
[-- Type: application/pgp-keys, Size: 21191 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Kevin]

[-- Attachment #1: Type: text/plain, Size: 2025 bytes --]

On 4/9/2014 11:29 AM, Wladimir wrote:
> Hello,
>
> This is primarily aimed at developers of SPV wallets.
>
> The recently reported decrease in number of full nodes could have 
> several reasons, one of them that less people are running Bitcoin Core 
> for the wallet because the other wallets are getting ahead in both 
> features and useability.
>
> It's great to see innovation in wallets, but it's worrying that the 
> number of full nodes decreases.
>
> It may be that lots of people would support the network by running a 
> full node, but don't want to go through the trouble of installing 
> bitcoin core separately (and get confused because it's a wallet, too).
>
> Hence I'd like to explore the idea of adding an option to popular SPV 
> wallets, to spin a bitcoind process in the background. This could be 
> pretty much transparent to the user - it would sync in the background, 
> the wallet could show statistics about the node, but is not dependent 
> on it.
>
> In exchange the user would get increased (full node level) security, 
> as the SPV wallet would have a local trusted node.
>
> Does this sound like a good idea?
>
> Is there any way that Bitcoin Core can help to accomedate this 
> 'embedded' usage? Specific Interfaces, special builds - maybe add a 
> walletless bitcoind build to gitian - bindings, dlls, etc?
>
> Wladimir
>
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
I personally like the ida.  Are you talking about a flag that could 
toggle this "in the background" mode or recoding for in the background use?


-- 
Kevin


[-- Attachment #2: Type: text/html, Size: 3586 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Wed, Apr 9, 2014 at 8:35 PM, Justus Ranvier <justusranvier@gmail.com>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 04/09/2014 06:19 PM, Wladimir wrote:
> > If no one wants to volunteer resources to support the network
> > anymore, we'll have failed.
>
> If the security of the network depends on a broken incentive model,
> then fix the design of the network so that economics works for you
> instead of against you.
>

My solution would be quick to implement and could help increase the number
of nodes in the short term. It will also smooth the way towards splitting
off the wallet from Bitcoin Core, by giving people a choice what wallet
they use with their full node.

That doesn't preclude looking for longer-term solutions that change the
incentive structure, but that is much more difficult and risky. I don't
believe it's a matter of 'fixing that instead' in a few hours, though I
nevertheless look forward to your pull request.

Wladimir

[-- Attachment #2: Type: text/html, Size: 1435 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Gregory Maxwell]

On Wed, Apr 9, 2014 at 11:35 AM, Justus Ranvier <justusranvier@gmail.com> wrote:
> If the security of the network depends on a broken incentive model,
> then fix the design of the network so that economics works for you
> instead of against you.

Who says anything about a broken incentive model. You've made past
claims about resource requirements that I think made no sense and then
failed to defend them when they were challenge.

With suitable software improvements running a full node could be done
in as little as a few gigabytes in disk space (e.g. cost 25-50 cents),
and as 50-100kbit/sec bandwidth in and out ongoing, and a moderate
amount of ram. Power costs are already just a few cents per month.  By
far the greatest cost is the figuring out and setting up part, which
bundling could fix. The exact resources could be tunable to what the
users are willing and able to contribute.

If improved marginal security and privacy in addition to supporting
the network is not enough incentive to overcome costs like these then
Bitcoin is already doomed.  I think that fundamental costs aren't an
issue at all, just implementation warts and education are.

Part of asking the question is feeling out which improvements need to
happen first, and what the prospects of getting the bundling going
once those improvements are made.

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Justus Ranvier]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/09/2014 06:50 PM, Gregory Maxwell wrote:
> Who says anything about a broken incentive model. You've made past 
> claims about resource requirements that I think made no sense and
> then failed to defend them when they were challenge.

Anyone reading the archives of the list will see about triple the
number of people independently confirming the resource usage problem
than they will see denying it, so I'm not particularly worried.


- -- 
Support online privacy by using email encryption whenever possible.
Learn how here: http://www.youtube.com/watch?v=bakOKJFtB-k
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTRZhsAAoJECoisBQbQ4v0iD8H/A1RZyLJN6f+zt5x78CFgRdp
zqZua3NBwUN2L9mI/PU1NADtxoXgkq49XusHRc+bjLu17VxGMUOsmB+AeK3VU4sN
BVC1qIcWCa+OPMkgnMFrdydz4OGxX5xiJoCNsqveZIEYc8nQoCDfsn/fwqL50/Ct
NfmPzh/cW7uUZ+h2bBgduGD3fzpQBlnnswAbHHX3FbBh9MvcaCfROeXXqUCedWj9
H2qVCNWTkIFJksJkqyF3f+KfedOTIyUwflx/0niBvzXcP9w4oK+WpBApHACWxiSA
H3KHYNF0s2/fs+mIkabPLRX1PIUk15ln29FapoDvlHW9jzDE92x6JFnMppI4rEs=
=ILks
-----END PGP SIGNATURE-----

[-- Attachment #2: 0x1B438BF4.asc --]
[-- Type: application/pgp-keys, Size: 21191 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

On Wed, Apr 9, 2014 at 6:09 PM, Thomas Voegtlin <thomasv1@gmx.de> wrote:

> Le 09/04/2014 17:54, Gregory Maxwell a écrit :
>
> > Sadly today Electrum requires more than a full node, it requires a
> > number of large additional indexes over what a full node has and
> > pruning is precluded. I don't think that increasing the resource
> > utilization of the node is a good way to go there for the purposes
> > expressed here. (not that electrum couldn't be used here, but not
> > unmodified without the resource usage increasing route)
> >
>
> Electrum uses two large indexes:
>
>      address -> utxo
>
> (patricia tree, aka "ultimate blockchain compression", see thread
> started by Alan Reiner in the bitcointalk forum)
>

Thanks for the explanation.

Adding a RPC call for a "address -> utxo" query wouldn't be a big deal. It
has been requested before for other purposes as well, all the better if it
helps for interaction with Electrum.

Spent history would be involve a much larger index, and it's not likely
that will end up in bitcoin

Wladimir

[-- Attachment #2: Type: text/html, Size: 1606 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Gregory Maxwell]

On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier <justusranvier@gmail.com> wrote:
> Anyone reading the archives of the list will see about triple the
> number of people independently confirming the resource usage problem
> than they will see denying it, so I'm not particularly worried.

The list has open membership, there is no particular qualification or
background required to post here. Optimal use of an information source
requires critical reading and understanding the limitations of the
medium. Counting comments is usually not a great way to assess
technical considerations on an open public forum.  Doubly so because
those comments were not actually talking about the same thing I am
talking about.

Existing implementations are inefficient in many known ways (and, no
doubt, some unknown ones). This list is about developing protocol and
implementations including improving their efficiency.  When talking
about incentives the costs you need to consider are the costs of the
best realistic option.  As far as I know there is no doubt from anyone
technically experienced that under the current network rules full
nodes can be operated with vastly less resources than current
implementations use, it's just a question of the relatively modest
implementation improvements.

When you argue that Bitcoin doesn't have the right incentives (and
thus something??) I retort that the actual resource _requirements_ are
for the protocol very low. I gave specific example numbers to enable
correction or clarification if I've said something wrong or
controversial. Pointing out that existing implementations are not that
currently as efficient as the underlying requirements and that some
large number of users do not like the efficiency of existing
implementations doesn't tell me anything I disagree with or didn't
already know. Whats being discussed around here contributes to
prioritizing improvements over the existing implementations.

I hope this clarifies something.

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[slush]

[-- Attachment #1: Type: text/plain, Size: 3377 bytes --]

I believe there're plenty bitcoind instances running, but they don't have
configured port forwarding properly.There's uPNP support in bitcoind, but
it works only on simple setups.

Maybe there're some not yet considered way how to expose these *existing*
instances to Internet, to strenghten the network. Maybe just self-test
indicating the node is not reachable from outside (together with short
howto like in some torrent clients).

These days IPv6 is slowly deploying to server environments, but maybe
there's some simple way how to bundle ipv6 tunnelling into bitcoind so any
instance will become ipv6-reachable automatically?

Maybe there're other ideas how to improve current situation without needs
of reworking the architecture.

Marek


On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:

> On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier <justusranvier@gmail.com>
> wrote:
> > Anyone reading the archives of the list will see about triple the
> > number of people independently confirming the resource usage problem
> > than they will see denying it, so I'm not particularly worried.
>
> The list has open membership, there is no particular qualification or
> background required to post here. Optimal use of an information source
> requires critical reading and understanding the limitations of the
> medium. Counting comments is usually not a great way to assess
> technical considerations on an open public forum.  Doubly so because
> those comments were not actually talking about the same thing I am
> talking about.
>
> Existing implementations are inefficient in many known ways (and, no
> doubt, some unknown ones). This list is about developing protocol and
> implementations including improving their efficiency.  When talking
> about incentives the costs you need to consider are the costs of the
> best realistic option.  As far as I know there is no doubt from anyone
> technically experienced that under the current network rules full
> nodes can be operated with vastly less resources than current
> implementations use, it's just a question of the relatively modest
> implementation improvements.
>
> When you argue that Bitcoin doesn't have the right incentives (and
> thus something??) I retort that the actual resource _requirements_ are
> for the protocol very low. I gave specific example numbers to enable
> correction or clarification if I've said something wrong or
> controversial. Pointing out that existing implementations are not that
> currently as efficient as the underlying requirements and that some
> large number of users do not like the efficiency of existing
> implementations doesn't tell me anything I disagree with or didn't
> already know. Whats being discussed around here contributes to
> prioritizing improvements over the existing implementations.
>
> I hope this clarifies something.
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 4330 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[slush]

[-- Attachment #1: Type: text/plain, Size: 4050 bytes --]

Another idea: Integrate torrent download of bootstrap.dat into bitcoind.
Normal user (especially a beginner) won't learn how to download bootstrap
separately and import it into bitcoind; he simply give up the
synchronization once he realize it takes too much time. From my experience
downloading the bootstrap significantly improves catching the blockchain,
which may attract some more users to run bitcoind.

Not sure about C++, but simple torrent client in python is like 30 lines of
code (using libtorrent).

Marek


On Wed, Apr 9, 2014 at 10:12 PM, slush <slush@centrum.cz> wrote:

> I believe there're plenty bitcoind instances running, but they don't have
> configured port forwarding properly.There's uPNP support in bitcoind, but
> it works only on simple setups.
>
> Maybe there're some not yet considered way how to expose these *existing*
> instances to Internet, to strenghten the network. Maybe just self-test
> indicating the node is not reachable from outside (together with short
> howto like in some torrent clients).
>
> These days IPv6 is slowly deploying to server environments, but maybe
> there's some simple way how to bundle ipv6 tunnelling into bitcoind so any
> instance will become ipv6-reachable automatically?
>
> Maybe there're other ideas how to improve current situation without needs
> of reworking the architecture.
>
> Marek
>
>
> On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell <gmaxwell@gmail.com>wrote:
>
>> On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier <justusranvier@gmail.com>
>> wrote:
>> > Anyone reading the archives of the list will see about triple the
>> > number of people independently confirming the resource usage problem
>> > than they will see denying it, so I'm not particularly worried.
>>
>> The list has open membership, there is no particular qualification or
>> background required to post here. Optimal use of an information source
>> requires critical reading and understanding the limitations of the
>> medium. Counting comments is usually not a great way to assess
>> technical considerations on an open public forum.  Doubly so because
>> those comments were not actually talking about the same thing I am
>> talking about.
>>
>> Existing implementations are inefficient in many known ways (and, no
>> doubt, some unknown ones). This list is about developing protocol and
>> implementations including improving their efficiency.  When talking
>> about incentives the costs you need to consider are the costs of the
>> best realistic option.  As far as I know there is no doubt from anyone
>> technically experienced that under the current network rules full
>> nodes can be operated with vastly less resources than current
>> implementations use, it's just a question of the relatively modest
>> implementation improvements.
>>
>> When you argue that Bitcoin doesn't have the right incentives (and
>> thus something??) I retort that the actual resource _requirements_ are
>> for the protocol very low. I gave specific example numbers to enable
>> correction or clarification if I've said something wrong or
>> controversial. Pointing out that existing implementations are not that
>> currently as efficient as the underlying requirements and that some
>> large number of users do not like the efficiency of existing
>> implementations doesn't tell me anything I disagree with or didn't
>> already know. Whats being discussed around here contributes to
>> prioritizing improvements over the existing implementations.
>>
>> I hope this clarifies something.
>>
>>
>> ------------------------------------------------------------------------------
>> Put Bad Developers to Shame
>> Dominate Development with Jenkins Continuous Integration
>> Continuously Automate Build, Test & Deployment
>> Start a new project now. Try Jenkins in the cloud.
>> http://p.sf.net/sfu/13600_Cloudbees
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>

[-- Attachment #2: Type: text/html, Size: 5309 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On Wed, Apr 9, 2014 at 10:12 PM, slush <slush@centrum.cz> wrote:

> Maybe there're other ideas how to improve current situation without needs
> of reworking the architecture.
>

Nothing I've proposed here would require larger changes to the architecture
then were already planned. After SPV lands we are going to split off the
wallet, and that will need an interface to an bitcoind to allow 'running
with full node'. If that can be generalized to be useful for other (SPV)
clients as well, that would be useful, hence I asked for input.

It of course doesn't preclude also looking for other solutions.

Wladimir

[-- Attachment #2: Type: text/html, Size: 1023 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mark Friedenbach]

I've advocated for this in the past, and reasonable counter-arguments I
was presented with are: (1) bittorrent is horribly insecure - it would
be easy to DoS the initial block download if that were the goal, and (2)
there's a reasonable pathway to doing this all in-protocol, so there's
no reason to introduce external dependencies.

On 04/09/2014 01:31 PM, slush wrote:
> Another idea: Integrate torrent download of bootstrap.dat into bitcoind.
> Normal user (especially a beginner) won't learn how to download
> bootstrap separately and import it into bitcoind; he simply give up the
> synchronization once he realize it takes too much time. From my
> experience downloading the bootstrap significantly improves catching the
> blockchain, which may attract some more users to run bitcoind.
> 
> Not sure about C++, but simple torrent client in python is like 30 lines
> of code (using libtorrent).
> 
> Marek
> 
> 
> On Wed, Apr 9, 2014 at 10:12 PM, slush <slush@centrum.cz
> <mailto:slush@centrum.cz>> wrote:
> 
>     I believe there're plenty bitcoind instances running, but they don't
>     have configured port forwarding properly.There's uPNP support in
>     bitcoind, but it works only on simple setups.
> 
>     Maybe there're some not yet considered way how to expose these
>     *existing* instances to Internet, to strenghten the network. Maybe
>     just self-test indicating the node is not reachable from outside
>     (together with short howto like in some torrent clients).
> 
>     These days IPv6 is slowly deploying to server environments, but
>     maybe there's some simple way how to bundle ipv6 tunnelling into
>     bitcoind so any instance will become ipv6-reachable automatically?
> 
>     Maybe there're other ideas how to improve current situation without
>     needs of reworking the architecture.
> 
>     Marek
> 
> 
>     On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell <gmaxwell@gmail.com
>     <mailto:gmaxwell@gmail.com>> wrote:
> 
>         On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier
>         <justusranvier@gmail.com <mailto:justusranvier@gmail.com>> wrote:
>         > Anyone reading the archives of the list will see about triple the
>         > number of people independently confirming the resource usage
>         problem
>         > than they will see denying it, so I'm not particularly worried.
> 
>         The list has open membership, there is no particular
>         qualification or
>         background required to post here. Optimal use of an information
>         source
>         requires critical reading and understanding the limitations of the
>         medium. Counting comments is usually not a great way to assess
>         technical considerations on an open public forum.  Doubly so because
>         those comments were not actually talking about the same thing I am
>         talking about.
> 
>         Existing implementations are inefficient in many known ways (and, no
>         doubt, some unknown ones). This list is about developing
>         protocol and
>         implementations including improving their efficiency.  When talking
>         about incentives the costs you need to consider are the costs of the
>         best realistic option.  As far as I know there is no doubt from
>         anyone
>         technically experienced that under the current network rules full
>         nodes can be operated with vastly less resources than current
>         implementations use, it's just a question of the relatively modest
>         implementation improvements.
> 
>         When you argue that Bitcoin doesn't have the right incentives (and
>         thus something??) I retort that the actual resource
>         _requirements_ are
>         for the protocol very low. I gave specific example numbers to enable
>         correction or clarification if I've said something wrong or
>         controversial. Pointing out that existing implementations are
>         not that
>         currently as efficient as the underlying requirements and that some
>         large number of users do not like the efficiency of existing
>         implementations doesn't tell me anything I disagree with or didn't
>         already know. Whats being discussed around here contributes to
>         prioritizing improvements over the existing implementations.
> 
>         I hope this clarifies something.
> 
>         ------------------------------------------------------------------------------
>         Put Bad Developers to Shame
>         Dominate Development with Jenkins Continuous Integration
>         Continuously Automate Build, Test & Deployment
>         Start a new project now. Try Jenkins in the cloud.
>         http://p.sf.net/sfu/13600_Cloudbees
>         _______________________________________________
>         Bitcoin-development mailing list
>         Bitcoin-development@lists.sourceforge.net
>         <mailto:Bitcoin-development@lists.sourceforge.net>
>         https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment 
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> 
> 
> 
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

On Wed, Apr 9, 2014 at 10:31 PM, slush <slush@centrum.cz> wrote:

> Another idea: Integrate torrent download of bootstrap.dat into bitcoind.
> Normal user (especially a beginner) won't learn how to download bootstrap
> separately and import it into bitcoind; he simply give up the
> synchronization once he realize it takes too much time. From my experience
> downloading the bootstrap significantly improves catching the blockchain,
> which may attract some more users to run bitcoind.
>
> Not sure about C++, but simple torrent client in python is like 30 lines
> of code (using libtorrent).
>

Parallel block download would be better for that. No need to involve
bittorrent.

But please let's not derail this thread, this is not about other solutions
for faster block download or such, let's keep it focused.

Wladimir

[-- Attachment #2: Type: text/html, Size: 1222 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[slush]

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

We definitely *need* lightweight bitcoin router / "core" which can be
easily deployed anywhere. No disagreement here.

I just wanted to remind that there're actually much more running nodes
*already* and maybe converting those hidden nodes to publicly reachable
nodes may be way easier than attracting fresh instances.

To the idea of bundled core with SPV clients - it won't improve anything,
in my opinion. SPV wallets are not running long-term (maybe Multibit team
has any better stats) so blockchain catching will turn the computer to be
unusable everytime user starts the wallet; that's exactly the reason why
people choose SPV wallets over full blockchain wallets.

Not saying that SPV clients usually aren't reachable from outside, which
turns the topic back to my ideas about motivating users to expose their
existing instances over Internet.

Marek

On Wed, Apr 9, 2014 at 10:35 PM, Wladimir <laanwj@gmail.com> wrote:

>
>
> On Wed, Apr 9, 2014 at 10:12 PM, slush <slush@centrum.cz> wrote:
>
>> Maybe there're other ideas how to improve current situation without needs
>> of reworking the architecture.
>>
>
> Nothing I've proposed here would require larger changes to the
> architecture then were already planned. After SPV lands we are going to
> split off the wallet, and that will need an interface to an bitcoind to
> allow 'running with full node'. If that can be generalized to be useful for
> other (SPV) clients as well, that would be useful, hence I asked for input.
>
> It of course doesn't preclude also looking for other solutions.
>
> Wladimir
>
>

[-- Attachment #2: Type: text/html, Size: 2572 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Laszlo Hanyecz]

On Apr 9, 2014, at 8:12 PM, slush <slush@centrum.cz> wrote:

> 
> These days IPv6 is slowly deploying to server environments, but maybe there's some simple way how to bundle ipv6 tunnelling into bitcoind so any instance will become ipv6-reachable automatically?
> 

Teredo is available by default on Microsoft systems and it's actually very common to see Teredo addresses as peers in torrents - this should work for bitcoin too, though I'm not sure if an app needs to set special flags to gain access to it.. there are probably some security settings around it.  In the US, AT&T/CenturyLink provide IPv6 by way of 6RD for DSL customers, Comcast has native IPv6 on residential (but not business) cable modems, and of course those who want to can always set up with a tunnel broker like Hurricane Electric - they even let you use your own IPv6 addresses.  IPv6 is great, but having an application running its own tunnels would not be a good way to leverage it.

Probably what's keeping a lot of them from being reachable is that most people just plug their CPE into a NAT router (without IPv6).  Teredo can help here though.
Putting an IPv6 checkbox in the settings with a link to an explanation might help with education and result in smarter operators who can make their nodes reachable.  A built in checker would be even better - something like a checklist showing red/green for different aspects of reachability (v4/teredo/v6).

-Laszlo

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Gregory Maxwell]

On Wed, Apr 9, 2014 at 1:36 PM, Mark Friedenbach <mark@monetize.io> wrote:
> (2) there's a reasonable pathway to doing this all in-protocol, so there's
> no reason to introduce external dependencies.

Not just a speculative pathway. Pieter implemented headers first:
https://github.com/sipa/bitcoin/tree/headersfirst  and it was
everything we hoped it would be— it easily can saturate residential
broadband, produces less load hot-spotting, copes with unreliable
peers, etc. It's now just slowly making its way into Bitcoin core.

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 1115 bytes --]

Hi Wladimir,

If the motivation of the SPV wallet is to radically extend functionality, as in my case, then the index is specific to the added features and the subset of the blockchain that is of interest for the wallet.
As you also point out, adding huge generic purpose indices to core would rather discourage people using full nodes due to excess requirements. 

I believe nothing would add more to the core’s popularity as a trusted background node to SPV than full validation at lowest possible memory, disk and CPU footprint.
Serving headers should be default but storing and serving full blocks configurable to ranges, so people can tailor to their bandwith and space available.

Tamas Blummer
Bits of Proof

On 09.04.2014, at 21:25, Wladimir <laanwj@gmail.com> wrote:
> 
> 
> Adding a RPC call for a "address -> utxo" query wouldn't be a big deal. It has been requested before for other purposes as well, all the better if it helps for interaction with Electrum.
> 
> Spent history would be involve a much larger index, and it's not likely that will end up in bitcoin
> 
> Wladimir
> 


[-- Attachment #1.2: Type: text/html, Size: 3699 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 4651 bytes --]

I tend to agree with slush here - counting the IPs in addr broadcasts often
gives a number like 100,000 vs just 10,000 for actually reachable nodes (or
less). It seems like optimising the NAT tunneling code would help. Starting
by adding more diagnostic stuff to the GUI. STUN support may also help.

The main constraint with home devices is not IMHO their actual power but
rather that a lot of people no longer keep computers switched on all the
time. If you don't do that then spv with bundled Core can't help your
security because the spv wallet would always be syncing from the p2p
network for performance reasons.
On 9 Apr 2014 22:13, "slush" <slush@centrum.cz> wrote:

> I believe there're plenty bitcoind instances running, but they don't have
> configured port forwarding properly.There's uPNP support in bitcoind, but
> it works only on simple setups.
>
> Maybe there're some not yet considered way how to expose these *existing*
> instances to Internet, to strenghten the network. Maybe just self-test
> indicating the node is not reachable from outside (together with short
> howto like in some torrent clients).
>
> These days IPv6 is slowly deploying to server environments, but maybe
> there's some simple way how to bundle ipv6 tunnelling into bitcoind so any
> instance will become ipv6-reachable automatically?
>
> Maybe there're other ideas how to improve current situation without needs
> of reworking the architecture.
>
> Marek
>
>
> On Wed, Apr 9, 2014 at 9:33 PM, Gregory Maxwell <gmaxwell@gmail.com>wrote:
>
>> On Wed, Apr 9, 2014 at 11:58 AM, Justus Ranvier <justusranvier@gmail.com>
>> wrote:
>> > Anyone reading the archives of the list will see about triple the
>> > number of people independently confirming the resource usage problem
>> > than they will see denying it, so I'm not particularly worried.
>>
>> The list has open membership, there is no particular qualification or
>> background required to post here. Optimal use of an information source
>> requires critical reading and understanding the limitations of the
>> medium. Counting comments is usually not a great way to assess
>> technical considerations on an open public forum.  Doubly so because
>> those comments were not actually talking about the same thing I am
>> talking about.
>>
>> Existing implementations are inefficient in many known ways (and, no
>> doubt, some unknown ones). This list is about developing protocol and
>> implementations including improving their efficiency.  When talking
>> about incentives the costs you need to consider are the costs of the
>> best realistic option.  As far as I know there is no doubt from anyone
>> technically experienced that under the current network rules full
>> nodes can be operated with vastly less resources than current
>> implementations use, it's just a question of the relatively modest
>> implementation improvements.
>>
>> When you argue that Bitcoin doesn't have the right incentives (and
>> thus something??) I retort that the actual resource _requirements_ are
>> for the protocol very low. I gave specific example numbers to enable
>> correction or clarification if I've said something wrong or
>> controversial. Pointing out that existing implementations are not that
>> currently as efficient as the underlying requirements and that some
>> large number of users do not like the efficiency of existing
>> implementations doesn't tell me anything I disagree with or didn't
>> already know. Whats being discussed around here contributes to
>> prioritizing improvements over the existing implementations.
>>
>> I hope this clarifies something.
>>
>>
>> ------------------------------------------------------------------------------
>> Put Bad Developers to Shame
>> Dominate Development with Jenkins Continuous Integration
>> Continuously Automate Build, Test & Deployment
>> Start a new project now. Try Jenkins in the cloud.
>> http://p.sf.net/sfu/13600_Cloudbees
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 6057 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

On Thu, Apr 10, 2014 at 8:38 AM, Mike Hearn <mike@plan99.net> wrote:

> I tend to agree with slush here - counting the IPs in addr broadcasts
> often gives a number like 100,000 vs just 10,000 for actually reachable
> nodes (or less). It seems like optimising the NAT tunneling code would
> help. Starting by adding more diagnostic stuff to the GUI. STUN support may
> also help.
>
> The main constraint with home devices is not IMHO their actual power but
> rather that a lot of people no longer keep computers switched on all the
> time. If you don't do that then spv with bundled Core can't help your
> security because the spv wallet would always be syncing from the p2p
> network for performance reasons.
>
I agree that there is a fundamental incompatibility in usage between
wallets and nodes. Wallets need to be online as little as possible, nodes
need to online as much as possible.

However, a full node background process could also be running if the wallet
is not open itself. Ffor example - by running as a system service.

Bitcoin Core's own wallet is also moving to SPV, so this means a general
solution is needed to get people to run a node when the wallet is not
running.

Maybe the node shouldn't be controlled from the wallet at all, it could be
a 'node control' user interface on its own (this is what -disablewallet
does currently). In this case, there is no need for packaging it with a
wallet The only drawback would be that initially, people wouldn't know why
or when to install this, hence my suggestion to pack it with wallets...

Wladimir

[-- Attachment #2: Type: text/html, Size: 1995 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 2727 bytes --]

It's an optimisation problem. Home environments are much more hostile than
servers are due to things like virus scanners, wildly varying memory
pressure as apps are started and shut down, highly asymmetrical upstream
versus downstream bandwidth,  complicated nat setups, people who only use
laptops (which I think is most people these days) and so on.

So I think the right way to go is to optimise the things that hurt server
node operators like large memory and disk  usage, and this will
automatically make it more pleasant to run on the desktop as well. If at
some point all the low hanging fruit for the server side is gone then
improving things on the desktop would be the next place to go. But we have
to be realistic. Desktop tower machines that are always on are dying and
will not be coming back. Not a single person I know uses them anymore, they
have been wiped out in favour of laptops. This is why, given the tiny size
of the bitcoin core development team, I do not think it makes sense to
spend precious coding hours chasing this goal.
On 10 Apr 2014 08:51, "Wladimir" <laanwj@gmail.com> wrote:

> On Thu, Apr 10, 2014 at 8:38 AM, Mike Hearn <mike@plan99.net> wrote:
>
>> I tend to agree with slush here - counting the IPs in addr broadcasts
>> often gives a number like 100,000 vs just 10,000 for actually reachable
>> nodes (or less). It seems like optimising the NAT tunneling code would
>> help. Starting by adding more diagnostic stuff to the GUI. STUN support may
>> also help.
>>
>> The main constraint with home devices is not IMHO their actual power but
>> rather that a lot of people no longer keep computers switched on all the
>> time. If you don't do that then spv with bundled Core can't help your
>> security because the spv wallet would always be syncing from the p2p
>> network for performance reasons.
>>
> I agree that there is a fundamental incompatibility in usage between
> wallets and nodes. Wallets need to be online as little as possible, nodes
> need to online as much as possible.
>
> However, a full node background process could also be running if the
> wallet is not open itself. Ffor example - by running as a system service.
>
> Bitcoin Core's own wallet is also moving to SPV, so this means a general
> solution is needed to get people to run a node when the wallet is not
> running.
>
> Maybe the node shouldn't be controlled from the wallet at all, it could be
> a 'node control' user interface on its own (this is what -disablewallet
> does currently). In this case, there is no need for packaging it with a
> wallet The only drawback would be that initially, people wouldn't know why
> or when to install this, hence my suggestion to pack it with wallets...
>
> Wladimir
>
>

[-- Attachment #2: Type: text/html, Size: 3388 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 526 bytes --]

You ask why people would install this ?

I find it is odd that we who hold the key to instant machine to machine micro payments do not use it to incentivise committing resources to the network.
What about serving archive blocks to peers paying for it ?

Tamas Blummer
http://bitsofproof.com

On 10.04.2014, at 08:50, Wladimir <laanwj@gmail.com> wrote:
> The only drawback would be that initially, people wouldn't know why or when to install this, hence my suggestion to pack it with wallets...
> 
> Wladimir
> 


[-- Attachment #1.2: Type: text/html, Size: 2540 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

>
> I find it is odd that we who hold the key to instant machine to machine
> micro payments do not use it to incentivise committing resources to the
> network.
>

It's not a new idea, obviously, but there are some practical consequences:

1) To pay a node for serving, you have to have bitcoins. To get bitcoins,
you need to sync with the network via a node. Catch 22.

2) If some nodes choose to charge and others choose to not charge, a smart
wallet will always use the free nodes. In the absence of any global load
balancing algorithms, this would lead to the free nodes getting overloaded
and collapsing whilst the for-pay nodes remain silent.

3) The only payment channel implementations today are bitcoinj's (Java) and
one written by Jeff in Javascript. There are no C++ implementations. And as
Matt and I can attest to, doing a real, solid, fully debugged
implementation that's integrated into a real app is .... a lot of work.

I still think the lowest hanging fruit is basic, boring optimisations
rather than architectural rethinks.

[-- Attachment #2: Type: text/html, Size: 1423 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



 But we
>have
>to be realistic. Desktop tower machines that are always on are dying
>and
>will not be coming back. Not a single person I know uses them anymore,
>they
>have been wiped out in favour of laptops. This is why, given the tiny
>size
>of the bitcoin core development team, I do not think it makes sense to
>spend precious coding hours chasing this goal.

Your social group is weird.

Nearly every coworker at my previous job had a tower computer at work and at home. Equally in my nontechnical social group lots of people, a significant minority if not majority, have Apple and PC desktops hooked up to large monitors at home for media production and games. Those who don't more often than not have laptops used as desktops, sitting in one place 95% of the time and left on.

People have found it most efficient to work at a static desk for centuries - that's not going away. Of course we're seeing desktop usage and sales falling, but that's only because previously the mobile usage was forced into suboptimal options by technical realities. The trend will bottom out a long way from zero.

Besides, even if just 1% of bitcoin users had a machine they left on that could usefully contribute to the network it would still vastly outweigh the much smaller percentage who would run nodes on expensive hosted capacity out of the goodness of their hearts. If we educated users about the privacy advantages of full nodes and gave them software that automatically contributed back within defined limits we'd have tens of thousands more useful nodes in the exact same way that user friendly filesharing software has lead to millions of users contributing bandwidth to filesharing networks. Similarly take advantage of the fault tolerance inherent in what we're doing and ensure that our software can shrug off nodes with a few % of downtime - certainly possible.

Of course, this doesn't fit in the business plans of those who might want to run full nodes to data mine and deanonymize users for marketing, tax collection, and law enforcement - one of the few profitable things to do with a full node - but screw those people.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRmVPMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhYb9B/0QWgQusFrQoPfYIzig
vpeo9sQgHAfA3gPYdqDLTtlaTgt8w3oP+/N46+Pi3lEphhCusXstzAFxi6c0XXsk
s96l9MqrUCZu55oEj1wZP0LJQx1uyUdevFv9bmocG5+94jBvGIoy3JZ3iQg+nNjL
uE9FpTnL43VOZ0WI9g6pXIE6XX6CxTx56tcxla4LTTypA1DijXa9MyYGOfYxXdPJ
w7jiRKl6Ijb3twP8+pX07GSIlL9yP7bESydnwyzwEo/RxAJxPmUpLuxluQ5DKTNY
G9TtwCpT+c6g5nXOxkI31XRcDuzhT+2kEhiDA6Neu2YNGrnyQx2WL6XuZNhi8nKB
IOMm
=rwGV
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 1885 bytes --]

I know the idea is not new. Just bringing it up to emphasize that if we don’t use it how could we expect other networks using it.
Machine to machine micro payments could become the killer application for Bitcoin.

1) There is no catch 22 as there are plenty of ways getting bitcoin without bootstrapping a full node.

2) let markets work out and not speculate what would happen.

3) Serving archive bolcks does not have to be part of core but could be a distinct service written in a language of your choice using new protocol.

As mentioned earlier I am for a stripped down core that does nothing else than consensus and stores nothing else needed for that task and offering SPV api to the wallets.

Tamas Blummer
http://bitsofproof.com

On 10.04.2014, at 11:17, Mike Hearn <mike@plan99.net> wrote:

> I find it is odd that we who hold the key to instant machine to machine micro payments do not use it to incentivise committing resources to the network.
> 
> It's not a new idea, obviously, but there are some practical consequences:
> 
> 1) To pay a node for serving, you have to have bitcoins. To get bitcoins, you need to sync with the network via a node. Catch 22.
> 
> 2) If some nodes choose to charge and others choose to not charge, a smart wallet will always use the free nodes. In the absence of any global load balancing algorithms, this would lead to the free nodes getting overloaded and collapsing whilst the for-pay nodes remain silent.
> 
> 3) The only payment channel implementations today are bitcoinj's (Java) and one written by Jeff in Javascript. There are no C++ implementations. And as Matt and I can attest to, doing a real, solid, fully debugged implementation that's integrated into a real app is .... a lot of work.
> 
> I still think the lowest hanging fruit is basic, boring optimisations rather than architectural rethinks.


[-- Attachment #1.2: Type: text/html, Size: 4242 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 10 April 2014 05:17:28 GMT-04:00, Mike Hearn <mike@plan99.net> wrote:
>>
>> I find it is odd that we who hold the key to instant machine to
>machine
>> micro payments do not use it to incentivise committing resources to
>the
>> network.
>>
>
>It's not a new idea, obviously, but there are some practical
>consequences:

You're both missing a more important issue: a core security assumption of bitcoin is that information is so easy to spread that censorship of it becomes impractical. If we're at the point where nodes are charging for their data we've failed that assumption.

More concretely, if my business is charging for block chain data and I can make a profit doing so via micro payments I have perverse incentives to drive away my competitors. If I give a peer a whole block they can sell access to that information in turn. Why would I make it easy for them if I don't have too?

Anyway, much of this discussion seems to stem from the misconception that contributing back to the network is a binary all or nothing thing - it's not. Over a year ago I myself was lamenting how I and the other "bitcoin-wizards" working on scalability had quickly solved every scaling problem *but* how to make it possible to scale up and keep mining decentralised.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRmi9MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhT/BB/98DudTV85hmruk0WRS
xVt7gGK6DJ2Isc7iJE09i9wSQc/PbHu7rZ2MYPreOdVzRmlHYhOV2ShnZaZJ7A9g
qB8pwy1wVrZgbrFeXXOLegNxGF2Xzc8OtL1E0bkNtTUUkuPIvT3UV4xn/Z+UZToR
XImXpfakfJvyRH80cbMNu4xG/t7Ym4K63nEpCCdsEKNm5a1vHTNRTNfGYMC1wrSV
XI3boZk7BQjqFDZADnonUU9zQ1WOmpdaVBYm+Xtgc+HXl3QODLcwGCY9hIRvaxek
L+IYX9yTVbgngDGy70BYG4ekWxvMtNzRU+9HG5DgA1/Er9r1KIaf/98xiFR7RVB4
Yfia
=34aD
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

>
> 1) There is no catch 22 as there are plenty of ways getting bitcoin
> without bootstrapping a full node.
>

I think I maybe wasn't clear. To spend coins you need transaction data.
Today, the dominant model is that people get that data by scanning the
block chain. If you can obtain the transaction data without doing that
then, either:

1) Someone is doing chain scanning for free. See my point about "why pay if
you can get it for free".

2) You got your tx data direct from the person you who sent you the funds,
perhaps via the payment protocol. This would resolve the catch 22 by
allowing you to spend bitcoins without actually having talked to the P2P
network first, but we're a loooooong way from this world.

And that's it. I don't think there are any other ways to get the tx data
you need. Either someone gives it to you in the act of spending, or someone
else gives it away for free, undermining the charge-for-the-p2p-network
model.

[-- Attachment #2: Type: text/html, Size: 1462 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 1228 bytes --]

Thanks, Peter and you convinced me. I run away with a thought.

It’d be great to find a spot to deploy payment channels, but I agree this is not it.

Tamas Blummer
http://bitsofproof.com

On 10.04.2014, at 12:40, Mike Hearn <mike@plan99.net> wrote:

> 1) There is no catch 22 as there are plenty of ways getting bitcoin without bootstrapping a full node.
> 
> I think I maybe wasn't clear. To spend coins you need transaction data. Today, the dominant model is that people get that data by scanning the block chain. If you can obtain the transaction data without doing that then, either:
> 
> 1) Someone is doing chain scanning for free. See my point about "why pay if you can get it for free".
> 
> 2) You got your tx data direct from the person you who sent you the funds, perhaps via the payment protocol. This would resolve the catch 22 by allowing you to spend bitcoins without actually having talked to the P2P network first, but we're a loooooong way from this world.
> 
> And that's it. I don't think there are any other ways to get the tx data you need. Either someone gives it to you in the act of spending, or someone else gives it away for free, undermining the charge-for-the-p2p-network model.


[-- Attachment #1.2: Type: text/html, Size: 3634 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Wladimir]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

On Thu, Apr 10, 2014 at 8:04 AM, Tamas Blummer <tamas@bitsofproof.com>wrote:

> Serving headers should be default but storing and serving full blocks
> configurable to ranges, so people can tailor to their bandwith and space
> available.
>

I do agree that it is important.

This does require changes to the P2P protocol, as currently there is no way
for a node to signal that they store only part of the block chain. Also,
clients will have to be modified to take this into account. Right now they
are under the assumption that every full node can send them every
(previous) block.

What would this involve?

Do you know of any previous work towards this?

Wladimir

[-- Attachment #2: Type: text/html, Size: 1125 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

>
> What would this involve?
>
> Do you know of any previous work towards this?
>

Chain pruning is a fairly complicated project, partly because it spans
codebases. For instance if you try and implement it *just* by changing
Bitcoin Core, you will break all the SPV clients based on bitcoinj (i.e.
all of them). Big changes to the P2P network like this require upgrading
both codebases simultaneously.

I think things like this may be why Gavin is now just "chief scientist"
instead of Core maintainer - in future, the changes people need will span
projects and require fairly significant planning.

From a technical perspective, it means extending addr broadcasts so nodes
broadcast how much of the chain they have, and teaching both Core and
bitcoinj how to search for nodes that have enough of the chain for them to
use. Currently bitcoinj still doesn't use addr broadcasts at all, there's
an incomplete patch available but it was never finished or merged. So that
has to be fixed first. And that probably implies improving Bitcoin Core so
the results of getaddr are more usable, ideally as high quality as what the
DNS seeds provide, because if lots of bad addresses are returned this will
slow down initial connect time, which is an important performance metric.

[-- Attachment #2: Type: text/html, Size: 1703 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Pieter Wuille]

There were earlier discussions.

The two ideas were either using one or a few service bits to indicate
availability of blocks, or to extend addr messages with some flags to
indicate this information.

I wonder whether we can't have a hybrid: bits to indicate general
degree of availability of blocks (none, only recent, everything), but
indicate actual availability only upon actually connecting (through a
"version" extension, or - preferably - a separate message). Reason is
that the actual blocks available are likely to change frequently (if
you keep the last week of blocks, a 3-day old addr entry will have
quite outdated information), and not that important to actual peer
selection - only to drive the decision which blocks to ask after
connection.

On Thu, Apr 10, 2014 at 1:09 PM, Wladimir <laanwj@gmail.com> wrote:
> On Thu, Apr 10, 2014 at 8:04 AM, Tamas Blummer <tamas@bitsofproof.com>
> wrote:
>>
>> Serving headers should be default but storing and serving full blocks
>> configurable to ranges, so people can tailor to their bandwith and space
>> available.
>
>
> I do agree that it is important.
>
> This does require changes to the P2P protocol, as currently there is no way
> for a node to signal that they store only part of the block chain. Also,
> clients will have to be modified to take this into account. Right now they
> are under the assumption that every full node can send them every (previous)
> block.
>
> What would this involve?
>
> Do you know of any previous work towards this?
>
> Wladimir
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 10 April 2014 06:44:32 GMT-04:00, Tamas Blummer <tamas@bitsofproof.com> wrote:
>Thanks, Peter and you convinced me. I run away with a thought.
>
>It’d be great to find a spot to deploy payment channels, but I agree
>this is not it.

No problem!

I'm sure we'll see payment channels implemented sooner or later
the form of "hub and spoke" payment networks. The idea there is you have one or more centralised hubs who in turn have payment channels setup to and from payors and payees. So long as the person you want to pay is connected to the same hub as you are, or in more advanced versions connected via a ripple style chain, you can push payment to the hub and get proof they did the same for the recipient. Your loss is always limited to the incremental payment amount and payment is essentially instant.

Of course, it's got some disadvantages compared to standard bitcoin transactions - its less decentralised - but when compared to other forms of off-chain payment in most situations its a strict improvement, and having the capability available is always a strict improvement. Like fidelity bonded banks the trust required in the hubs is low enough that with some minor effort applied to anti-DoS you could probably get away with using even hubs run by anonymous actors, making the centralisation less important. (hubs are essentially interchangeable) Unlike pure fidelity bonded banks the effort required to build this is relatively minor!

You can even combine it with chaum tokens for anonymity. You'll want to hold the tokens for some amount of time to thwart timing analysis, leaving you somewhat vulnerable to theft, but in that case fidelity bonded banking principles can be applied. Other than that case the idea is probably made obsolete by micropayment hubs.

Regulatory issues will be interesting... If you wind up with a few central payment hubs, without chaum tokens, those hubs learn all details about every transaction made. Obviously if a big actor like BitPay implemented this there would be a lot of pressure on them to make those records available to law enforcement and tax authorities, not to mention marketing and other data mining. Equally I suspect that if an alternative more decentralised version was implemented there would be strong government pressure for those approved hubs to not interoperate with the decentralised hubs, and equally for merchants to not accept payment from the decentralised hubs.

But all the same, if widely implemented this reduces pressure to raise the block size enormously, keeping the underlying system decentralised. So the net effect is probably positive regardless.

Oh yeah, credit goes to Mike Hearn for the payment channels, and if I'm correct, for the hub concept as well.

Amir: You should think about adding the above to dark wallet. It'd be good if the protocols are implemented in an open and decentralised fashion first, prior to vendor lock in.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRoIlMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhW26B/9A9OtYjoSHo620XZzF
VqfnnVFCPr3DpD/HuT3JYhF1kkL2vTt5wkRIHmHmfJ29Sduj8St7EFiLOyUg2mvt
q9heZgzCnqxLJm9zMiiQnb3Y/plvhTLfaONnHI+OPSfrABL6DA04nEe8OBPuFfv/
NowJ74DP/65YBq3EqbqG0dJExKm1BhdrEpWNq0v5YoCVuEYkWgFHL8SdRHnfFyxA
XTkP8avzlG82r98k55IrV0O/6VQNHjE0+xF4EHjEYBacy6OwlpEYeLrqx/VDAQ5R
RufXeAltNZI0tzLQ8nY0aMBH3YFxF0+14/sbmOAtmnD6EW49gEcV9MnSJc5ct4m7
Szq5
=aC39
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 10 April 2014 07:32:44 GMT-04:00, Pieter Wuille <pieter.wuille@gmail.com> wrote:
>There were earlier discussions.
>
>The two ideas were either using one or a few service bits to indicate
>availability of blocks, or to extend addr messages with some flags to
>indicate this information.
>
>I wonder whether we can't have a hybrid: bits to indicate general
>degree of availability of blocks (none, only recent, everything), but
>indicate actual availability only upon actually connecting (through a
>"version" extension, or - preferably - a separate message). Reason is
>that the actual blocks available are likely to change frequently (if
>you keep the last week of blocks, a 3-day old addr entry will have
>quite outdated information), and not that important to actual peer
>selection - only to drive the decision which blocks to ask after
>connection.

Why not just put an expiration date on that information and delay deletion until the expiration is reached?

Also, its worth noting how the node bit solution you proposed can be done as a gradual upgrade path for SPV client. From the perspective of nodes that don't know about it they just see the pruned nodes as SPV nodes without any chain data at all. The only issue would be if large numbers of uses turned off their full nodes, but that's a possibility regardless. Done with partial UTXO set mode this may even result in an eventual increase in the number of full nodes.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRoPZMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhetrCACA02EJQ0VpcYvafuNc
7pvqMVeirJRu3Uv7Wy8rcl9jW5irM5fmNdznARtv2vwpEZN7MU0wp3ZY1FYOCv2f
PvWC7DBCSBs2BuyGkvPuwnXEppTrYmWFT3qjg+99lF1IlOV4yWFacja2RGDuJkea
fYUkODosHJjFVcXi5aMkBPQ5sOFdlUVbC94YV4d4PDSmF2fHLGG8uEfEweYb6Pv+
gj1CsfuAWf8DWzygDeL8x/wOG9HeqYqEbjxyOb9hxlp1ByUof+4WJtz3QfGsR2Xt
fvkmgS8vkUxSIZorMdypj7oLBOnfDW1bEK5He2SlqPdYi5FEQusZ/jMMX3Fw74GV
fJKt
=Wyv8
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

>
> Oh yeah, credit goes to Mike Hearn for the payment channels, and if I'm
> correct, for the hub concept as well.
>

Actually, the design is from Satoshi and Matt did most of the
implementation work last year during a Google internship. Though I ended up
doing a lot of work on it too. We actually got pretty far: there was
Android UI for it and a couple of apps we coded up. I wish we could have
pushed it over the finishing line and got real world usage. Hopefully we
can return to it someday soon.

I think the hub/spoke concept was invented by goldsmiths in 16th century
Italy, as they started handing pieces of paper across their benches, or
*bancos* in Italian   :-)

[-- Attachment #2: Type: text/html, Size: 934 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Gregory Maxwell]

On Thu, Apr 10, 2014 at 4:32 AM, Pieter Wuille <pieter.wuille@gmail.com> wrote:
> There were earlier discussions.

On this list.

> The two ideas were either using one or a few service bits to indicate
> availability of blocks, or to extend addr messages with some flags to
> indicate this information.
>
> I wonder whether we can't have a hybrid: bits to indicate general
> degree of availability of blocks (none, only recent, everything), but
> indicate actual availability only upon actually connecting (through a
> "version" extension, or - preferably - a separate message). Reason is
> that the actual blocks available are likely to change frequently (if
> you keep the last week of blocks, a 3-day old addr entry will have
> quite outdated information), and not that important to actual peer
> selection - only to drive the decision which blocks to ask after
> connection.

I think you actually do need the kept ranges to be circulated,
otherwise you might need to hunt for a very long time to find the
right nodes with the blocks you need.  Alternatively, you give up and
don't hunt and pick some node that has them all and we get poor load
distribution. I'd rather be in a case where the nodes that have the
full history are only hit as a last resort.

WRT the changing values, I think that is pretty uniquely related to
the most recent blocks, and so instead I think that should be handled
symbolically (e.g. the hybrid approach... a flag for the "I keep the
most recent 2000 blocks", I say 2000 because thats about where the
request offset historgrams flattened out) or as a single offset range
"I keep the last N=200",  and the flag or the offset would be in
addition to whatever additional range was signaled. The latter could
be infrequently changing.

Signaling _more_ and more current range data on connect seems fine to
me, I just don't think it replaces something that gets relayed.

Based on the safety against reorgs and the block request access
patterns we observed I'm pretty sure we'd want any node serving blocks
at all to be at least the last N (for some number between 144 and 2000
or so). Based on the request patterns if just the recent blocks use up
all the space you're willing to spend, then I think thats probably
still the optimal contribution...

(Just be glad I'm not suggesting coding the entire blockchain with an
error correcting code so that it doesn't matter which subset you're
holding)

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 10 April 2014 07:45:16 GMT-04:00, Mike Hearn <mike@plan99.net> wrote:
>>
>> Oh yeah, credit goes to Mike Hearn for the payment channels, and if
>I'm
>> correct, for the hub concept as well.
>>
>
>Actually, the design is from Satoshi and Matt did most of the
>implementation work last year during a Google internship.

Ah right, of course. Along those lines we should credit Jeremy Spilman (?) for figuring out how to get rid of the dependency on nSequence, makimg the protocol trust-free.

I do recall it having an issue with malleability, semi-fixed with the P2SH trick. Be good to clear that up for good for Pieter's proposed malleability patch.

>Though I
>ended up
>doing a lot of work on it too. We actually got pretty far: there was
>Android UI for it and a couple of apps we coded up. I wish we could
>have
>pushed it over the finishing line and got real world usage. Hopefully
>we
>can return to it someday soon.
>
>I think the hub/spoke concept was invented by goldsmiths in 16th
>century
>Italy, as they started handing pieces of paper across their benches, or
>*bancos* in Italian   :-)

...and only took another five hundred years for math to catch up and make it trust free, modulo miner centralisation!
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRoYCMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhdNMB/9zFBT3nFSkasHkMVW8
01pE/VYrclH5BsxANakOqfb57Rprc4xue9H4AuppqAlIo/i/gYtOyoDy9y8oQDmC
YRiM6K5+bqUH8uC/Rjz1AASBrnb3zAasqCJlklheb5nP3+EoLpKNJ04Brk8rnlqp
CMO5GktE43r9buYL9MOMXHanB523wRmNV8JjpEF+y5tPYe9YW7rsdmpjX8F8sRga
PE1MMNy43lctoej4tR0iSBfK2ZNudsz7PdzW9+4Gvpc4NDMHp5O4JwhX/vPbSyyC
+Or1BvPKe58zVSi5kBi7AJEeFXasfuKpwBwT9r2CGmjvI62ESmkj5M9eqcb4i8Yy
i2Zr
=E4NH
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 10 April 2014 07:50:55 GMT-04:00, Gregory Maxwell <gmaxwell@gmail.com> wrote:
>(Just be glad I'm not suggesting coding the entire blockchain with an
>error correcting code so that it doesn't matter which subset you're
>holding)

I forgot to ask last night: if you do that, can you add new blocks to the chain with the encoding incrementally?
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCgA6BQJTRoZ+MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhYudCAC7ImifMnLIFHv1UifV
zRxtDkx7UxIf9dncDAcrTIyKEDhoouh0TmoZl3HKQ3KUEETAVKsMzqXLgqVe6Ezr
ny1bm0pQlkBCZFRwuZvmB27Y3mwC8PD6rT9ywtWzFjWd8PEg6/UaM547nQPw7ir0
27S3XMfE/BMiQWfWnWc/nqpbmJjd8x/dM3oiTG9SVZ7iNxotxAqfnW2X5tkhJb0q
dAV08wpu6aZ5hTyLpvDxXDFjEG119HJeLkT9QVIrg+GBG55PYORqE4gQr6uhrF4L
fGZS2EIlbk+kAiv0EjglQfxWM7KSRegplSASiKEOuX80tqLIsEugNh1em8qvG401
NOAS
=CWql
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tier Nolan]

[-- Attachment #1: Type: text/plain, Size: 3682 bytes --]

Error correction is an interesting suggestion.

If there was 10000 nodes and each stored 0.1% of the blocks, at random,
then the odds of a block not being stored is 45 in a million.

Blocks are stored on average 10 times, so there is already reasonable
redundancy.

With 1 million blocks, 45 would be lost in that case, even though most are
stored multiple times.

With error correction codes, the chances of blocks going missing is much
lower.

For example, if there was 32 out of 34 Reed-Solomon-like system, then 2
blocks out of 34 could be lost without any actual data loss for the network.

As a back of the envelop check, the odds of 2 missing blocks landing within
34 of another is 68/1000000.  That means that the odds of 2 missing blocks
falling in the same correction section is 45 * 34 / 1000000 = 0.153%.  Even
in that case, the missing blocks could be reconstructed, as long as you
know that they are missing.

The error correction code has taken it from being a near certainty that
some blocks would be lost to less than 0.153%.

A simple error correction system would just take 32 blocks in sequence and
then compute 2 extra blocks.

The extra blocks would have to be the same length as the longest block in
the 32 being corrected.

The shorter blocks would be padded with zeroes so everything is the same
size.

For each byte position in the blocks you compute the polynomial that goes
through byte (x, data(x)), for x = 0 to 31.  This could be a finite field,
or just mod 257.

You can then compute the value for x=32 and x = 33.  Those are the values
for the 2 extra blocks.

If mod 257 is used, then only the 2 extra blocks have to deal with symbols
from 0 to 256.

If you have 32 of the 34 blocks, you can compute the polynomial and thus
generate the 32 actual blocks.

This could be achieved by a soft fork by having a commitment every 32
blocks in the coinbase.

It makes the header chain much longer though.

Longer sections are more efficient, but need more calculations to recover
everything.  You could also do interleaving to handle the case where entire
sections are missing.


On Thu, Apr 10, 2014 at 12:54 PM, Peter Todd <pete@petertodd.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
>
>
> On 10 April 2014 07:50:55 GMT-04:00, Gregory Maxwell <gmaxwell@gmail.com>
> wrote:
> >(Just be glad I'm not suggesting coding the entire blockchain with an
> >error correcting code so that it doesn't matter which subset you're
> >holding)
>
> I forgot to ask last night: if you do that, can you add new blocks to the
> chain with the encoding incrementally?
> -----BEGIN PGP SIGNATURE-----
> Version: APG v1.1.1
>
> iQFQBAEBCgA6BQJTRoZ+MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
> cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhYudCAC7ImifMnLIFHv1UifV
> zRxtDkx7UxIf9dncDAcrTIyKEDhoouh0TmoZl3HKQ3KUEETAVKsMzqXLgqVe6Ezr
> ny1bm0pQlkBCZFRwuZvmB27Y3mwC8PD6rT9ywtWzFjWd8PEg6/UaM547nQPw7ir0
> 27S3XMfE/BMiQWfWnWc/nqpbmJjd8x/dM3oiTG9SVZ7iNxotxAqfnW2X5tkhJb0q
> dAV08wpu6aZ5hTyLpvDxXDFjEG119HJeLkT9QVIrg+GBG55PYORqE4gQr6uhrF4L
> fGZS2EIlbk+kAiv0EjglQfxWM7KSRegplSASiKEOuX80tqLIsEugNh1em8qvG401
> NOAS
> =CWql
> -----END PGP SIGNATURE-----
>
>
>
> ------------------------------------------------------------------------------
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 4812 bytes --]

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Gregory Maxwell]

On Thu, Apr 10, 2014 at 10:30 AM, Tier Nolan <tier.nolan@gmail.com> wrote:
> Error correction is an interesting suggestion.

Though I mentioned it, it was in jest— I think right now it would be
an over-design at least for the basic protocol.  Also, storing
'random' blocks has some locality problems, when verifying blocks you
need to obtain them contiguously, and so we can take advantage of the
locality of reference.  For the non-error-coded case I believe nodes
with random spans of blocks works out asymptotically to the same
failure rates as random.

One thing that I like to point out is that there is absolutely no need
for the entire network to use the same p2p protocol. Diversity here
would be very good.  I think it would be really good for someone to
have an alternative p2p protocol using these techniques even though I
think they aren't yet compelling enough to be table stakes in the
basic protocol.

There are some very helpful things you can do with forward error
correction for faster and more efficient block relaying too:
https://en.bitcoin.it/wiki/User:Gmaxwell/block_network_coding

(The conversation Peter Todd was referring to was one where I was
pointing out that with suitable error coding you also get an
anti-censorship effect where its very difficult to provide part of the
data without potentially providing all of it)

> If there was 10000 nodes and each stored 0.1% of the blocks, at random, then
> the odds of a block not being stored is 45 in a million.

I think in the network we have today and for the foreseeable future we
can reasonably count on there being a reasonable number of nodes that
store all the blocks... quite likely not enough to satisfy the
historical block demand from the network alone, but easily enough to
supply blocks that have otherwise gone missing.

Re: [Bitcoin-development] Bitcoind-in-background mode for SPV wallets

[Tier Nolan]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

On Fri, Apr 11, 2014 at 5:54 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:

> For the non-error-coded case I believe nodes
> with random spans of blocks works out asymptotically to the same
> failure rates as random.
>

If each "block" is really 512 blocks in sequence, then each "slot" is more
likely to be hit.  It effectively reduces the number of blocks by the
minimum run lengths.

ECC seemed cooler though.


> (The conversation Peter Todd was referring to was one where I was
> pointing out that with suitable error coding you also get an
> anti-censorship effect where its very difficult to provide part of the
> data without potentially providing all of it)
>

Interesting too.

>
> I think in the network we have today and for the foreseeable future we
> can reasonably count on there being a reasonable number of nodes that
> store all the blocks... quite likely not enough to satisfy the
> historical block demand from the network alone, but easily enough to
> supply blocks that have otherwise gone missing.
>

That's true.  Scaling up the transactions per second increases the chance
of data lost.

With side/tree chains, the odds of data loss in the less important chains
increases (though they are by definition lower value chains)

[-- Attachment #2: Type: text/html, Size: 1975 bytes --]