[Bitcoin-development] HTTP REST API for bitcoind

[Bitcoin-development] HTTP REST API for bitcoind

[Jeff Garzik]

URL: https://github.com/bitcoin/bitcoin/pull/2844

Adding an HTTP REST API for bitcoind has been occasionally tossed
about as a useful thing.  Such an API would essentially provide a
decentralized block explorer capability, enabling easy external access
to transaction/address/block indices that we maintain.

The first two implemented API calls are simple, returning a block or
TX given a simple query string based on block hash, e.g.

     GET /rest/tx/TX-HASH
or
     GET /rest/block/BLOCK-HASH

This can be easily accessed via command line cURL/wget utilities.
Output formats -- binary, hex or json -- may be selected via a
"bitcoin-format" header.

The general goal of the HTTP REST interface is to access
unauthenticated, public blockchain information.  There is no plan to
add wallet interfacing/manipulation via this API.

-- 
Jeff Garzik
Senior Software Engineer and open source evangelist
BitPay, Inc.      https://bitpay.com/

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Michael Hendricks]

[-- Attachment #1: Type: text/plain, Size: 2134 bytes --]

+1 and thank you. I've prototyped a couple different Bitcoin projects that
would benefit from this.

I'm traveling with poor 'net so I haven't read the patches yet. I echo pull
request comments about using Accept and Accept-Encoding headers. Same for
an API version number in the URL.

It'd be helpful, eventually, to have APIs corresponding to Bitcoin addr and
version messages.  Metadata about the network and the peer, respectively,
are valuable in my use cases.

Michael
On Jul 22, 2013 1:43 PM, "Jeff Garzik" <jgarzik@bitpay.com> wrote:

> URL: https://github.com/bitcoin/bitcoin/pull/2844
>
> Adding an HTTP REST API for bitcoind has been occasionally tossed
> about as a useful thing.  Such an API would essentially provide a
> decentralized block explorer capability, enabling easy external access
> to transaction/address/block indices that we maintain.
>
> The first two implemented API calls are simple, returning a block or
> TX given a simple query string based on block hash, e.g.
>
>      GET /rest/tx/TX-HASH
> or
>      GET /rest/block/BLOCK-HASH
>
> This can be easily accessed via command line cURL/wget utilities.
> Output formats -- binary, hex or json -- may be selected via a
> "bitcoin-format" header.
>
> The general goal of the HTTP REST interface is to access
> unauthenticated, public blockchain information.  There is no plan to
> add wallet interfacing/manipulation via this API.
>
> --
> Jeff Garzik
> Senior Software Engineer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 2983 bytes --]

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andreas Schildbach]

On 07/22/2013 09:42 PM, Jeff Garzik wrote:

> The general goal of the HTTP REST interface is to access
> unauthenticated, public blockchain information.  There is no plan to
> add wallet interfacing/manipulation via this API.

Is it planned to expose the UXTO set of a given address? That would be
useful for SPV wallets to be able to swipe a previously unknown private
key (e.g. paper wallet).

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Michael Gronager]

Hi Andreas / Jeff,

Access to the UTXO set can be done using libcoin (see the coinexplorer example), which also has a rest interface. Access to the UTXO set pr address/script requires indexing of all scripts, which was easy in libcoin as the blockchain is stored in a sqlite database. Integrating this in bitcoind would require setting up and maintaining such an index ad hoc.

Michael


On Jul 23, 2013, at 10:27 , Andreas Schildbach <andreas@schildbach.de> wrote:

> On 07/22/2013 09:42 PM, Jeff Garzik wrote:
> 
>> The general goal of the HTTP REST interface is to access
>> unauthenticated, public blockchain information.  There is no plan to
>> add wallet interfacing/manipulation via this API.
> 
> Is it planned to expose the UXTO set of a given address? That would be
> useful for SPV wallets to be able to swipe a previously unknown private
> key (e.g. paper wallet).
> 
> 
> 
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Pieter Wuille]

On Tue, Jul 23, 2013 at 10:27:19AM +0200, Andreas Schildbach wrote:
> On 07/22/2013 09:42 PM, Jeff Garzik wrote:
> 
> > The general goal of the HTTP REST interface is to access
> > unauthenticated, public blockchain information.  There is no plan to
> > add wallet interfacing/manipulation via this API.
> 
> Is it planned to expose the UXTO set of a given address? That would be
> useful for SPV wallets to be able to swipe a previously unknown private
> key (e.g. paper wallet).

Depends what you mean by expose.

Maintaining an address/script-indexed UTXO is generally useful, in
particular for things like sweeping addresses. I certainly have
less problems with 'exposing' this than exposing a fully-indexed
block chain history.

However, and I expect that's what your question is about, this isn't
really useful for SPV (or less) nodes, as there is no way to
authenticate this data. If you can fake a UTXO entry, you can make
a peer believe anything about their balance, potentially resulting
in creating a valid transaction that sends change it didn't know
was there as fee to miners. Other than for normal block chain data,
there is no way to detect this without at least partial validation.

The only way to do this safely at an SPV security assumption, is by
having an address-indexed committed merkle UTXO-set tree, like the
one proposed by Alan Reiner, and being implemented by Mark
Friedenback. I know Michael Gronager has something similar implemented,
but I don't know whether it is script-indexed. To be actually useful,
it likely needs to be enforced by miners - putting a significant
burden on validation nodes. Still, if it can be done efficiently,
I think this would be worth it, but more research is needed first in
any case.

Regarding sweeping keys in the first place - I think using those,
and relying on address-indexed UTXO sets or blockchains to import
them, is an idea that doesn't scale very well in the first place.
If it is for things like scratch card or physical coins, with a
pre-set value, the obvious solution IMHO is storing the crediting
transaction with its merkle path together with the key. If that's
not possible, just the txid:vout of the credit output can suffice.
Yes, that's more data than is necessary now, but it's so much more
trivial to use.

-- 
Pieter

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Michael Gronager]

> 
> The only way to do this safely at an SPV security assumption, is by
> having an address-indexed committed merkle UTXO-set tree, like the
> one proposed by Alan Reiner, and being implemented by Mark
> Friedenback. I know Michael Gronager has something similar implemented,
> but I don't know whether it is script-indexed.

The MerkleTrie I have in libcoin is indexed on UTXOs only. However, adding
an extra index for scripts would be pretty easy (half day of coding), or even having the two merged into one index.

The burden imposed on validating nodes for keeping such an index is really minimal. When using the UTXO MerkleTrie I switch off the sqlite index of these and vise versa, so there are hardly any measurable timing difference.

However, the MerkleTrie index is currently re-build on startup (which takes ~30 sec on my laptop), keeping it synced with disk would be optimal and in the long run necessary as even the UTXO set will grow over time.

> To be actually useful,
> it likely needs to be enforced by miners - putting a significant
> burden on validation nodes. Still, if it can be done efficiently,
> I think this would be worth it, but more research is needed first in
> any case.
> 

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andreas Schildbach]

On 07/23/2013 11:37 AM, Pieter Wuille wrote:

>> Is it planned to expose the UXTO set of a given address? That would be
>> useful for SPV wallets to be able to swipe a previously unknown private
>> key (e.g. paper wallet).
>
> Depends what you mean by expose.
>
> Maintaining an address/script-indexed UTXO is generally useful, in
> particular for things like sweeping addresses. I certainly have
> less problems with 'exposing' this than exposing a fully-indexed
> block chain history.
>
> However, and I expect that's what your question is about, this isn't
> really useful for SPV (or less) nodes, as there is no way to
> authenticate this data. If you can fake a UTXO entry, you can make
> a peer believe anything about their balance, potentially resulting
> in creating a valid transaction that sends change it didn't know
> was there as fee to miners. Other than for normal block chain data,
> there is no way to detect this without at least partial validation.
>
> The only way to do this safely at an SPV security assumption, is by
> having an address-indexed committed merkle UTXO-set tree, like the
> one proposed by Alan Reiner, and being implemented by Mark
> Friedenback. I know Michael Gronager has something similar implemented,
> but I don't know whether it is script-indexed. To be actually useful,
> it likely needs to be enforced by miners - putting a significant
> burden on validation nodes. Still, if it can be done efficiently,
> I think this would be worth it, but more research is needed first in
> any case.
>
> Regarding sweeping keys in the first place - I think using those,
> and relying on address-indexed UTXO sets or blockchains to import
> them, is an idea that doesn't scale very well in the first place.
> If it is for things like scratch card or physical coins, with a
> pre-set value, the obvious solution IMHO is storing the crediting
> transaction with its merkle path together with the key. If that's
> not possible, just the txid:vout of the credit output can suffice.
> Yes, that's more data than is necessary now, but it's so much more
> trivial to use.

I certainly don't want to push ideas that won't work for whatever
reason. So I fully respect whatever you decide regarding that feature.
Personally I have never felt any need for being able to sweep paper
wallets, I am more or less just relaying the need of users.

Let me just say this:

Sweeping paper wallets is a common feature request. People switch to
centralized services just for getting that.

It is my understanding that for the usecase, an address-indexed UXTO is
enough. So you probably don't need to worry about script-indexed for now.

Security issues could be mitigated by applying trust to the REST server,
e.g. because its your own or the one of your apps vendor. Of course,
link-level security would be needed for this (e.g. SSL).

Paper wallets that include the necessary additional information is
something I have been thinking about. I see some issues:

- Paper wallets are already quite widespread. You still won't be able to
sweep those.
- Some people like to "top up" a paper wallet or even just sweep a
portion of it. That would not be possible, and in some cases even lead
to loss of coins because of the "involuntary fee" you described.
- Does the necessary info fit into a QR code?

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Pieter Wuille]

On Tue, Jul 23, 2013 at 12:17 PM, Andreas Schildbach
<andreas@schildbach.de> wrote:
> Sweeping paper wallets is a common feature request. People switch to
> centralized services just for getting that.

That means they value convenience more than the trust-freeness of a
decentralized solution. The only way to avoid that is by making sure
the decentralized one is convenient enough. But relying on
unauthenticated data itself is equally bad - it means you lose
whatever benefit the decentralization had.

> It is my understanding that for the usecase, an address-indexed UXTO is
> enough. So you probably don't need to worry about script-indexed for now.

The difference between script-indexed and address-indexed is
absolutely trivial compared to the effort needed to implement and
maintain such authenticated trees by all full nodes. Restricting
things at the network level (which doesn't even know about a thing
like an address) to address-based indexes is ridiculous IMHO.

> Security issues could be mitigated by applying trust to the REST server,
> e.g. because its your own or the one of your apps vendor. Of course,
> link-level security would be needed for this (e.g. SSL).

Sure, once you introduce trust, a lot can be done. But it's not really
Bitcoin anymore in that case - it's relying on a third party to do the
heavy indexing for you. And if that is the best-scaling solution, sure
- but I don't think we should encourage that. Or at least, we should
first search for alternatives. And encourage infrastructure that
doesn't require it.

> Paper wallets that include the necessary additional information is
> something I have been thinking about. I see some issues:
>
> - Paper wallets are already quite widespread. You still won't be able to
> sweep those.
> - Some people like to "top up" a paper wallet or even just sweep a
> portion of it. That would not be possible, and in some cases even lead
> to loss of coins because of the "involuntary fee" you described.

Yeah, those are inherent problems with how there are used today. But
today there is also little problem - the UTXO set is tiny.

> - Does the necessary info fit into a QR code?

Absolutely, though a slightly bigger one.

-- 
Pieter

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andy Parkins]

On Monday 22 July 2013 20:42:45 Jeff Garzik wrote:
> URL: https://github.com/bitcoin/bitcoin/pull/2844
> 
> Adding an HTTP REST API for bitcoind has been occasionally tossed
> about as a useful thing.  Such an API would essentially provide a
> decentralized block explorer capability, enabling easy external access
> to transaction/address/block indices that we maintain.

This is excellent.

> The first two implemented API calls are simple, returning a block or
> TX given a simple query string based on block hash, e.g.
> 
>      GET /rest/tx/TX-HASH
> or
>      GET /rest/block/BLOCK-HASH

One additional URL makes this pretty much perfect:

  GET /rest/block-with-tx/TX-HASH

Construction of the transaction-hash-to-block database is something the full 
client's have to do anyway, so this query is no harder than the others for 
them to supply; but suddenly makes it possible for an SPV client to trace the 
providence of any transaction without needing to maintain the entire chain.


Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Pieter Wuille]

On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote:
> One additional URL makes this pretty much perfect:
> 
>   GET /rest/block-with-tx/TX-HASH
> 
> Construction of the transaction-hash-to-block database is something the full 
> client's have to do anyway, so this query is no harder than the others for 
> them to supply; but suddenly makes it possible for an SPV client to trace the 
> providence of any transaction without needing to maintain the entire chain.

There is actually no such index being maintained by default, and doing so is an
unnecessary burden IMHO (you need to enable -txindex since 0.8 to get this).
Of course, if enabled, it can be exposed.

-- 
Pieter

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andy Parkins]

On Tuesday 23 July 2013 10:42:05 Pieter Wuille wrote:
> On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote:
> > One additional URL makes this pretty much perfect:
> >   GET /rest/block-with-tx/TX-HASH
> > 
> > Construction of the transaction-hash-to-block database is something the
> > full client's have to do anyway, so this query is no harder than the
> > others for them to supply; but suddenly makes it possible for an SPV
> > client to trace the providence of any transaction without needing to
> > maintain the entire chain.
> 
> There is actually no such index being maintained by default, and doing so
> is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to
> get this). Of course, if enabled, it can be exposed.

Wow.  I'm surprised at that.  How does a newly received transaction have its 
inputs verified then?  Multiple linear brute force searches of the block chain 
for every new transaction?  Or is it that transactions are only recorded if 
they were in a block, and just their presence indicates they're valid?


Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Pieter Wuille]

On Tue, Jul 23, 2013 at 11:52 AM, Andy Parkins <andyparkins@gmail.com> wrote:
>> There is actually no such index being maintained by default, and doing so
>> is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to
>> get this). Of course, if enabled, it can be exposed.
>
> Wow.  I'm surprised at that.  How does a newly received transaction have its
> inputs verified then?  Multiple linear brute force searches of the block chain
> for every new transaction?  Or is it that transactions are only recorded if
> they were in a block, and just their presence indicates they're valid?

The block chain is not involved at all to verify transactions, it's
just a historical
record to serve to other nodes, and to do wallet rescans with.

For validation, a separate database with just unspent transaction
outputs is used (around 230 MB now).

-- 
Pieter

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andy Parkins]

On Tuesday 23 July 2013 10:56:02 Pieter Wuille wrote:

> The block chain is not involved at all to verify transactions, it's
> just a historical
> record to serve to other nodes, and to do wallet rescans with.

It must be involved to some extent.  Certainly during a temporary fork, there 
are two branches growing, and you have to be able, when verifying a new 
transaction, to say which branch it's one... which branch of the blockchain.


Andy
-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Pieter Wuille]

On Tue, Jul 23, 2013 at 12:02 PM, Andy Parkins <andyparkins@gmail.com> wrote:
> On Tuesday 23 July 2013 10:56:02 Pieter Wuille wrote:
>
>> The block chain is not involved at all to verify transactions, it's
>> just a historical
>> record to serve to other nodes, and to do wallet rescans with.
>
> It must be involved to some extent.  Certainly during a temporary fork, there
> are two branches growing, and you have to be able, when verifying a new
> transaction, to say which branch it's one... which branch of the blockchain.

No, not really.

The UTXO set is the state you need to validate blocks and
transactions. You can see blocks as authenticated patches to the UTXO
set (consumes some outputs, produces others). During validation, we
store "undo data", basically (non-authenticated) reverse patches to
the UTXO set, so we can walk back in case of a reorganization.

-- 
Pieter

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]

On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote:
> One additional URL makes this pretty much perfect:
> 
>   GET /rest/block-with-tx/TX-HASH
> 
> Construction of the transaction-hash-to-block database is something the full 
> client's have to do anyway, so this query is no harder than the others for 
> them to supply; but suddenly makes it possible for an SPV client to trace the 
> providence of any transaction without needing to maintain the entire chain.

On Tue, Jul 23, 2013 at 10:27:19AM +0200, Andreas Schildbach wrote:
> On 07/22/2013 09:42 PM, Jeff Garzik wrote:
> 
> > The general goal of the HTTP REST interface is to access
> > unauthenticated, public blockchain information.  There is no plan to
> > add wallet interfacing/manipulation via this API.
> 
> Is it planned to expose the UXTO set of a given address? That would be
> useful for SPV wallets to be able to swipe a previously unknown private
> key (e.g. paper wallet).

The REST API has nothing to do with SPV clients; it's similar to the RPC
interface and won't be exposed to the network as a whole.

Increasing the resource usage by SPV clients on full nodes is undesirable; we
have a lot of work to do regarding DoS attacks. John Dillon's comments here on
using micro-transactions to compensate full-nodes for maintaining expensive
blockchain indexes are worth reading:
https://github.com/bitcoin/bitcoin/pull/2802#issuecomment-20232958

In any case UTXO data currently requires you to have full trust in
whomever is providing you with it, and that situation will continue
until UTXO commitments are implemented - if they are implemented.

-- 
'peter'[:-1]@petertodd.org
000000000000007bea8b46717ec4acb05830bcb6222497366dd72b02ddc80569

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andy Parkins]

On Tuesday 23 July 2013 10:47:03 Peter Todd wrote:
> On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote:
> > One additional URL makes this pretty much perfect:
> >   GET /rest/block-with-tx/TX-HASH
> > 
> > Construction of the transaction-hash-to-block database is something the
> > full client's have to do anyway, so this query is no harder than the
> > others for them to supply; but suddenly makes it possible for an SPV
> > client to trace the providence of any transaction without needing to
> > maintain the entire chain.

> The REST API has nothing to do with SPV clients; it's similar to the RPC
> interface and won't be exposed to the network as a whole.

Yes; I know that.  I'm saying that it would make it easier for SPV (and other 
lightweight clients) for that matter.

> Increasing the resource usage by SPV clients on full nodes is undesirable;

I don't think that's thinking big enough.  What I imagine is that making it 
easier and easier to store a partial blockchain would result in lower demand 
on full nodes.

I might run a client that has only fetched blocks that contain transactions 
needed to verify my balances, right back to the genesis block.  That will be 
some small subset of the block chain and will take me very little resource to 
maintain.  I join the network and am my client is willing to verify based on 
information I have, or supply (by REST or bitcoin protocol) blocks.  Imagine 
then that everyone with a wallet were doing this.  The blockchain would be 
distributed massively.  Obviously the miners would still be keeping the entire 
chain, but we'd have a lot more nodes in the network, each contributing a 
little bit and so reducing the load on the full nodes.

> In any case UTXO data currently requires you to have full trust in
> whomever is providing you with it, and that situation will continue
> until UTXO commitments are implemented - if they are implemented.

Almost; because you can go and ask someone else the same question, it's pretty 
easy to check if you're being lied to.  Also, it's far easier to maintain a 
headers-only block chain.  When you fetch your relevant block subset, you can 
easily see that they are real blocks in your headers-only blockchain; and so 
it's pretty much impossible to lie to "give me the block containing 
transaction X".


Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 2723 bytes --]

On Tue, Jul 23, 2013 at 11:00:24AM +0100, Andy Parkins wrote:
> > Increasing the resource usage by SPV clients on full nodes is undesirable;
> 
> I don't think that's thinking big enough.  What I imagine is that making it 
> easier and easier to store a partial blockchain would result in lower demand 
> on full nodes.

Read my proposal for "Partial UTXO" mode:
http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg02511.html

> I might run a client that has only fetched blocks that contain transactions 
> needed to verify my balances, right back to the genesis block.  That will be 
> some small subset of the block chain and will take me very little resource to 
> maintain.  I join the network and am my client is willing to verify based on 
> information I have, or supply (by REST or bitcoin protocol) blocks.  Imagine 
> then that everyone with a wallet were doing this.  The blockchain would be 
> distributed massively.  Obviously the miners would still be keeping the entire 
> chain, but we'd have a lot more nodes in the network, each contributing a 
> little bit and so reducing the load on the full nodes.

Actually the really scary thing about partial UTXO mode is miners can
get away without keeping the entire chain provided they don't (often)
try to mine transactions spending UTXO's that they haven't verified
themselves.

They can get away with accepting blocks without checking that the UTXO's
exist, at least until enough miners do so that someone creates an
invalid block and the majority of hashing power never notices. Remember
that only with a complete UTXO set can you know that a UTXO *doesn't*
exist.

We're going to have to force miners to prove they possess the full UTXO
set in the future or the security of Bitcoin will be seriously
threatened.

> > In any case UTXO data currently requires you to have full trust in
> > whomever is providing you with it, and that situation will continue
> > until UTXO commitments are implemented - if they are implemented.
> 
> Almost; because you can go and ask someone else the same question, it's pretty 

How do you know they actually are someone else?

> easy to check if you're being lied to.  Also, it's far easier to maintain a 
> headers-only block chain.  When you fetch your relevant block subset, you can 
> easily see that they are real blocks in your headers-only blockchain; and so 
> it's pretty much impossible to lie to "give me the block containing 
> transaction X".

Do you think you have SPV or full security in that situation?

Do you know the difference?

-- 
'peter'[:-1]@petertodd.org
0000000000000070f3d118303a611e1f44ea6482a3b59a16056e69af088b1ffa

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andy Parkins]

On Tuesday 23 July 2013 11:17:28 Peter Todd wrote:
> On Tue, Jul 23, 2013 at 11:00:24AM +0100, Andy Parkins wrote:
> > > Increasing the resource usage by SPV clients on full nodes is
> > > undesirable;
> > 
> > I don't think that's thinking big enough.  What I imagine is that making
> > it easier and easier to store a partial blockchain would result in lower
> > demand on full nodes.
> 
> Read my proposal for "Partial UTXO" mode:
> http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg02
> 511.html

Very interesting.  I love the idea of the UTXO set being tied to a block.

> > I might run a client that has only fetched blocks that contain
> > transactions needed to verify my balances, right back to the genesis
> > block.  That will be some small subset of the block chain and will take
> > me very little resource to maintain.  I join the network and am my
> > client is willing to verify based on information I have, or supply (by
> > REST or bitcoin protocol) blocks.  Imagine then that everyone with a
> > wallet were doing this.  The blockchain would be distributed massively. 
> > Obviously the miners would still be keeping the entire chain, but we'd
> > have a lot more nodes in the network, each contributing a little bit and
> > so reducing the load on the full nodes.
> 
> Actually the really scary thing about partial UTXO mode is miners can
> get away without keeping the entire chain provided they don't (often)
> try to mine transactions spending UTXO's that they haven't verified
> themselves.

You're right.  That is scary.

> > > In any case UTXO data currently requires you to have full trust in
> > > whomever is providing you with it, and that situation will continue
> > > until UTXO commitments are implemented - if they are implemented.
> > 
> > Almost; because you can go and ask someone else the same question, it's
> > pretty
> 
> How do you know they actually are someone else?

You don't.  You can't invalidate the lie if all you have access to is lies.  
But if you have access to just one honest node; that will reveal the liars.  
I'm not claiming that headers-only nodes can ever be made as secure as a full 
node.  Just _more_ secure than they are now; and potentially able to act as 
one of those honest nodes.

> > easy to check if you're being lied to.  Also, it's far easier to maintain
> > a headers-only block chain.  When you fetch your relevant block subset,
> > you can easily see that they are real blocks in your headers-only
> > blockchain; and so it's pretty much impossible to lie to "give me the
> > block containing transaction X".
> 
> Do you think you have SPV or full security in that situation?
> Do you know the difference?

There is absolutely no need to get condescendingly shirty.  I thought this was 
a friendly list; and we were having a discussion.  If you don't want to 
respond to posts -- don't.  I also didn't realise I had to pass an exam before 
I was allowed to speak.

Yes: I know the difference between SPV and full security.  SPV is headers only 
and so has no way to verify that the transaction outputs references as inputs 
to any new as-yet-unverified transaction are valid.  Instead it relies on 
having some way of proving it's in the chain; and then looking for the number 
of blocks built on top of it as "verification".  "Full security" (which is 
itself a very poor name), is obviously just checking that every output 
referenced in the inputs is unspent; that necessarily requires full blocks.

The difference in security being that in SPV there is no way to know if the 
referenced Unspent TransaXtion Output really is unspent -- it might have been 
spent elsewhere then referenced again in this new transaction.

My suggestion was that we want to be able to fetch a block by transaction; and 
that simple nodes can all, in aggregate offer contribution to the network 
rather than just being parasitical on the full nodes.   When I ask for a block 
that contains a transaction, and I do that repeatedly, I have part of the 
block chain.  If lots of simple nodes are doing that, then the whole chain 
should be available if there are enough of them.  They would then gain the 
ability to do transaction-forwarding in some cases.  This is only possible if 
a few extra facilities are added to the protocol.  One of which is the new 
feature I suggested: block-given-transaction.  It's not enough on its own, but 
if you also add in the ability for a node to tell another about the output 
transactions (basically, what block spends it), _then_ the simple nodes are 
able to become much more secure -- not 100% of course, they're still not full 
nodes, because they have no way of knowing if they are being lied to when they 
are told (this transaction is unspent), but all it takes is one honest node to 
point them at the truth, and the lie is then exposed.

That facility is just a drain on full nodes for the most part; except if you 
start encouraging it whole-sale.  The simple node would keep cache both the 
incoming and outgoing transactions (or rather the blocks that contain them) 
for addresses to which they are paying attention.  That gives them a cache 
that contains more than just their minimal set; and then they are able to do 
just a little bit of verifying on their own.  With enough nodes of this sort, 
the verification load is reduced.

Perhaps all that effort is not worth it for the tiny reduction.  Perhaps it's 
not true that that contribution of verification adds nothing.  I can live with 
those objections.  But "do I know the difference" as a reposte?  Not so much.

Anyway; going by your post on partial UTXO's; you're well ahead of the game, 
and I'm not suggesting anything that hasn't already been thought of, and 
thought of better.  I'm not sure why you took umbridge at my idea, when it 
seems like I'm just a few steps behind what you've already thought of.  Not 
everything is an attack you know?



Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Pieter Wuille]

On Tue, Jul 23, 2013 at 12:00 PM, Andy Parkins <andyparkins@gmail.com> wrote:
>> The REST API has nothing to do with SPV clients; it's similar to the RPC
>> interface and won't be exposed to the network as a whole.
>
> Yes; I know that.  I'm saying that it would make it easier for SPV (and other
> lightweight clients) for that matter.

In what way?

>> Increasing the resource usage by SPV clients on full nodes is undesirable;
>
> I don't think that's thinking big enough.  What I imagine is that making it
> easier and easier to store a partial blockchain would result in lower demand
> on full nodes.

A partial blockchain is quite useless, as you can't build an UTXO set from it.
If you're talking simply about the storage requirements for maintaining history,
perhaps, but why rely on SPV nodes for this? Right now, those don't store any
blocks at all, and there is no reason why they should.

The only requirement is that old blocks remain available for new full
nodes to be
able to bootstrap. It's certainly not required that everyone keeps
every block ever
created. That's how the software currently works, but as soon as we get to a few
protocol changes that would allow new full nodes to find specific
peers with the data
they need, we can have fully-verifying yet (partially) pruning nodes.
I think that's a
much better idea than conflating "maintaining a wallet" with
"maintaining a subset
of historical block data".


> Obviously the miners would still be keeping the entire
> chain, but we'd have a lot more nodes in the network, each contributing a
> little bit and so reducing the load on the full nodes.

I disagree strongly here. The rules of the network are enforced by
full nodes, not by
miners - they are just trying to satisfy the rules demaned by the network.

And as far as I know, there is no way to do some "partial validation"
using just the blocks
you care about (and remember, SPV nodes currently have none at all).
One interesting
possibility here is fraud proofs, where the network can relay proofs
that certain blocks or
transactions are violating certain network rules. In this case, even
SPV nodes can just
communicate with eachother and get some "herd immunity". But storing some blocks
doesn't matter here - it is all about whether you can maintain the
UTXO set or not.

>
>> In any case UTXO data currently requires you to have full trust in
>> whomever is providing you with it, and that situation will continue
>> until UTXO commitments are implemented - if they are implemented.
>
> Almost; because you can go and ask someone else the same question, it's pretty
> easy to check if you're being lied to.  Also, it's far easier to maintain a
> headers-only block chain.  When you fetch your relevant block subset, you can
> easily see that they are real blocks in your headers-only blockchain; and so
> it's pretty much impossible to lie to "give me the block containing
> transaction X".

That's assuming there is no powerful enough attacker that can benefit from doing
a sybil attack on you. For SPV nodes currently, that risk is limited
to an attacker
that can spend enough on faking a chain with valid proof-of-work, to make you
accept a transaction that will be reversed.

If you go let SPV nodes rely on unauthenticated UTXO set data, there is no such
limitation, and they can let you believe *anything*. There are some safeguards,
like combining it with merkle paths for all outputs that credit you (which again
requires a more expensive index on the other side), but you can't ever guarantee
that a particular outputs isn't spent yet.

-- 
Pieter

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Andreas Schildbach]

On 07/23/2013 11:47 AM, Peter Todd wrote:

>> Is it planned to expose the UXTO set of a given address? That would be
>> useful for SPV wallets to be able to swipe a previously unknown private
>> key (e.g. paper wallet).
> 
> The REST API has nothing to do with SPV clients; it's similar to the RPC
> interface and won't be exposed to the network as a whole.
> 
> Increasing the resource usage by SPV clients on full nodes is undesirable; we
> have a lot of work to do regarding DoS attacks.

Yes, I understand that. For this reason, I would vote for adding the
usual HTTP authentication/SSL stuff to the REST API. That way, SPV users
can decide to run their own instance of the API (providing the needed
resources themselves).

Or, a trusted party can set up a server. For example, I would be willing
to set it up for users of Bitcoin Wallet. I don't expect shitloads of
paper wallets sweeps for the forseeable future.

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Pieter Wuille]

On Tue, Jul 23, 2013 at 12:29 PM, Andreas Schildbach
<andreas@schildbach.de> wrote:
> Yes, I understand that. For this reason, I would vote for adding the
> usual HTTP authentication/SSL stuff to the REST API. That way, SPV users
> can decide to run their own instance of the API (providing the needed
> resources themselves).
>
> Or, a trusted party can set up a server. For example, I would be willing
> to set it up for users of Bitcoin Wallet. I don't expect shitloads of
> paper wallets sweeps for the forseeable future.

I don't object to using a trusted server for this if people want that,
but I don't think the reference client should encourage this.

Apart from that, exposing this HTTP-based interface publicly has its
own problems, like security risks and potential DoS risks. If
anything, we should be reducing the attack surface rather than
increase it. IMHO, the only thing that should be exposed in the P2P
protocol, which is inevitable, and already has some DoS protections.

I like this HTTP interface, but it should really only be used for
trusted local applications and debugging.

-- 
Pieter

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Michael Hendricks]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

On Tue, Jul 23, 2013 at 4:36 AM, Pieter Wuille <pieter.wuille@gmail.com>wrote:

> Apart from that, exposing this HTTP-based interface publicly has its
> own problems, like security risks and potential DoS risks. If
> anything, we should be reducing the attack surface rather than
> increase it. IMHO, the only thing that should be exposed in the P2P
> protocol, which is inevitable, and already has some DoS protections.
>
> I like this HTTP interface, but it should really only be used for
> trusted local applications and debugging.
>

We already have a good private HTTP interface.  Most benefits of this REST
interface come from exposing it publicly.  As always, the challenge is
balancing costs and benefits.  I'm not confident that either can be known
with certainty until a well-written prototype is running in the wild.  As
some nodes expose this interface, we'll gain concrete experience from which
node operators can make informed security trade offs.

-- 
Michael

[-- Attachment #2: Type: text/html, Size: 1490 bytes --]

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Mark Friedenbach]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 7/23/13 3:29 AM, Andreas Schildbach wrote:
>
> Yes, I understand that. For this reason, I would vote for adding the
> usual HTTP authentication/SSL stuff to the REST API. That way, SPV users
> can decide to run their own instance of the API (providing the needed
> resources themselves).
>
> Or, a trusted party can set up a server. For example, I would be willing
> to set it up for users of Bitcoin Wallet. I don't expect shitloads of
> paper wallets sweeps for the forseeable future.
>
>
Anyone who wants HTTP authentication or TLS can wrap it with nginx, or
something similar. In the process they could put appropriate
restrictions in place on incoming requests, and the onus would be on
them, not us to keep it secure.

Mark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR7ttWAAoJEAdzVfsmodw4UmIP/36lK2TDc7mLTT8rbflJhl3v
TL4CFKhXj6OuzG7tyino3Djs4EQnyk+CbpfOmJ8kYr29GPaZttuDJhYXtJqQBQCi
DPq79ktudHnVMLPirEs7dUrLo+TAqhYX+8Sj+eTlW+p6YZg3JbkOAIPJG7597OK4
zzU8Oxr0XKJFfGscKfkPThxJboNqzJYGl3otHUMXM4HsbIRYmrx4QSr8y7dsVgTd
YZnD4bJO+eY4ZPzCcFdkPD/8bXQyKC5nPOH8/79lARNLESwB4OW79uf9q86EuH2O
jZQ1qwpRNHblrNWS1/U2E4+7hEidvgZBwQhj+HbWgKiPWh4Df1lEXq6bLQQwdn6/
b+jfiwg7xpb7eB2M4gPZ0uF/1TIcGJN3+LWEULFNTT/vsjyD/UU63ahZ1kVv7X0m
W1NrbKjXxDbip+x3N7HLIu3zqAAaa0ele7OysyFCL6ZlwwafwJiEZZgHn2Iw7I1L
S7lYBbFoLfXlOMVXNaKHPEV5gQEveMROJVBtnWkqShPQM0N/+Z+TXZes37up0GVo
d7ptPfNbUNDTFc8Jj3+5rIyy3dUvSyMJlHZhsLmtCUnbQ867ZOgeUS52a8XQ+nJY
8IsShLfLk6fRWmHrwo9lzZQ/TbbUNyoUje0Ns6iL7G3IZwDqJH3kAGb/bkj/piDu
tPNcN8bkYeNobTFIH+o4
=jV80
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] HTTP REST API for bitcoind

[Rune Kjær Svendsen]

[-- Attachment #1: Type: text/plain, Size: 2888 bytes --]

I think this is a useful feature, but I din't see why it should be a part
of bitcoind. I've created a simple HTTP REST wrapper around bitcoind's RPC
interface in Python:
https://github.com/runeksvendsen/btchttp/blob/master/btchttp.py

It simply runs a HTTP server that translates HTTP GET requests into the
relevant RPC request, and responds to the GET request with the relevant
data.

/Rune

On Tue, Jul 23, 2013 at 9:36 PM, Mark Friedenbach <mark@monetize.io> wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On 7/23/13 3:29 AM, Andreas Schildbach wrote:
> >
> > Yes, I understand that. For this reason, I would vote for adding the
> > usual HTTP authentication/SSL stuff to the REST API. That way, SPV users
> > can decide to run their own instance of the API (providing the needed
> > resources themselves).
> >
> > Or, a trusted party can set up a server. For example, I would be willing
> > to set it up for users of Bitcoin Wallet. I don't expect shitloads of
> > paper wallets sweeps for the forseeable future.
> >
> >
> Anyone who wants HTTP authentication or TLS can wrap it with nginx, or
> something similar. In the process they could put appropriate
> restrictions in place on incoming requests, and the onus would be on
> them, not us to keep it secure.
>
> Mark
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBAgAGBQJR7ttWAAoJEAdzVfsmodw4UmIP/36lK2TDc7mLTT8rbflJhl3v
> TL4CFKhXj6OuzG7tyino3Djs4EQnyk+CbpfOmJ8kYr29GPaZttuDJhYXtJqQBQCi
> DPq79ktudHnVMLPirEs7dUrLo+TAqhYX+8Sj+eTlW+p6YZg3JbkOAIPJG7597OK4
> zzU8Oxr0XKJFfGscKfkPThxJboNqzJYGl3otHUMXM4HsbIRYmrx4QSr8y7dsVgTd
> YZnD4bJO+eY4ZPzCcFdkPD/8bXQyKC5nPOH8/79lARNLESwB4OW79uf9q86EuH2O
> jZQ1qwpRNHblrNWS1/U2E4+7hEidvgZBwQhj+HbWgKiPWh4Df1lEXq6bLQQwdn6/
> b+jfiwg7xpb7eB2M4gPZ0uF/1TIcGJN3+LWEULFNTT/vsjyD/UU63ahZ1kVv7X0m
> W1NrbKjXxDbip+x3N7HLIu3zqAAaa0ele7OysyFCL6ZlwwafwJiEZZgHn2Iw7I1L
> S7lYBbFoLfXlOMVXNaKHPEV5gQEveMROJVBtnWkqShPQM0N/+Z+TXZes37up0GVo
> d7ptPfNbUNDTFc8Jj3+5rIyy3dUvSyMJlHZhsLmtCUnbQ867ZOgeUS52a8XQ+nJY
> 8IsShLfLk6fRWmHrwo9lzZQ/TbbUNyoUje0Ns6iL7G3IZwDqJH3kAGb/bkj/piDu
> tPNcN8bkYeNobTFIH+o4
> =jV80
> -----END PGP SIGNATURE-----
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 3998 bytes --]