[Bitcoin-development] Suggestion for enhancements to getblock

[Bitcoin-development] Suggestion for enhancements to getblock

[Andy Parkins]

Hello,

This is a suggestion with a mind to the future.  In particular, I'm slowly 
working on an alternative client and library (I know, everyone says that).  
I've got a feature that I'd like to have that would need a change in the 
protocol.  It's a change that I think would improve the official client as 
well, so I'm bringing it up here.

It's actually two changes; and both could be acheived by adding new commands 
to the protocol.  I think that would be overkill though; as they fit quite 
nicely into an extended getblocks command.

(1) The getblocks message is a list of inventory items (type MSG_BLOCK) being 
requested.

Imagine this situation though.  I am a light weight client.  I store the block 
headers only.  I am only interested in the history of my own wallet addresses.  
I receive a block broadcast with a transaction that sends coins to one of my 
addresses.  That transaction references other transactions (of course), but I 
haven't stored any transactions.  So; I want to request those transactions and 
ensure they are all valid and in blocks.  I can't.

I can request the transactions themselves; but I have no way of finding out 
what block they were in without downloading the entire full block chain 
myself.  The thing is, a peer with the full block chain is able to do this 
instantly.

Here then is my suggestion:  getblocks should accept inventory items of type 
MSG_TX as well as MSG_BLOCK.  When it finds a MSG_TX request, it shouldn't 
send the transaction (after all that is what the getdata message is for); 
instead it should return the block that contains that transaction.  It's an 
alternative way of requesting a block -- by transaction in that block.

It should be obvious then that it would be easy for a lightweight client to 
request the transaction chain transactions its interested in to create a list 
of relevant hashes for the history it's after; and then put those hashes in a 
getblocks request and have to look only at a few full blocks instead of the 
full block chain.


(2) If you are offline when new transactions are broadcast, there is no way to 
know they are pending.

Transactions that have been broadcast but not yet accepted into a block are 
never resent (nor should they be).  But if I am on a mobile client say, or a 
light-weight, intermittently used client; I have no way of checking if a 
transaction sending coins to one of my addresses is pending.

It should be possible to request the current pending transaction list.

My sugestion then is that a special virtual block request be possible.  The 
all zero block hash can never exist (it's used as the parent for the genesis 
block, so it had better not).  If I send a getblocks that contains an all-zero 
hash, then the virtual block should be returned in a block message.  That 
virtual block will list all the current pending transactions.  Obviously there 
would be a problem that it's actual hash would not be zero; so it would have 
to be marked as the virtual block in some other way (perhaps a zero timestamp 
plus a zero parent hash, or similar).



The combination of the above two protocol changes makes it possible for any 
client to have a large chunk of the facilities available in blockexplorer.com 
and the very useful http://bitcoincharts.com/bitcoin/ information.




Andy
-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: [Bitcoin-development] Suggestion for enhancements to getblock

[Mike Hearn]

On Thu, Jul 7, 2011 at 11:49 AM, Andy Parkins <andyparkins@gmail.com> wrote:
> Imagine this situation though.  I am a light weight client.  I store the block
> headers only.  I am only interested in the history of my own wallet addresses.
> I receive a block broadcast with a transaction that sends coins to one of my
> addresses.  That transaction references other transactions (of course), but I
> haven't stored any transactions.  So; I want to request those transactions and
> ensure they are all valid and in blocks.  I can't.

Everyone writing an alternative client goes through this thought
process :-) There's no point in doing it, you cannot prove your
transaction is not a double spend. That requires knowledge (ie, an
index) of all transactions.

You have to treat appearing deep in the chain as ipso-facto proof of
validity. Lightweight/SPV clients simply must have that trust, it
cannot be done any other way. See this article:

http://code.google.com/p/bitcoinj/wiki/SecurityModel

Currently this is pretty safe due to the crazy speeds. In future when
speeds are likely to be lower, it will be less safe and you'd have to
wait longer or use a trusted node.

> It should be possible to request the current pending transaction list.

I think it'd be better to implement the filtering suggestions that
have been made. It doesn't scale to download the entire memory pool -
a better approach is to give the remote node a filter to match against
transactions then have it only relay those. After setting a filter,
transactions pending and matching would be sent in one big inv and you
can then keep the connection open to learn about new transactions
without needing to "drink from the firehose". Filters can be
probabilistic and set on many different nodes to reduce the privacy
implications.

Re: [Bitcoin-development] Suggestion for enhancements to getblock

[Andy Parkins]

[-- Attachment #1: Type: Text/Plain, Size: 3029 bytes --]

On 2011 July 07 Thursday, Mike Hearn wrote:
> On Thu, Jul 7, 2011 at 11:49 AM, Andy Parkins <andyparkins@gmail.com> wrote:
> > Imagine this situation though.  I am a light weight client.  I store the
> > block headers only.  I am only interested in the history of my own
> > wallet addresses. I receive a block broadcast with a transaction that
> > sends coins to one of my addresses.  That transaction references other
> > transactions (of course), but I haven't stored any transactions.  So; I
> > want to request those transactions and ensure they are all valid and in
> > blocks.  I can't.
> 
> Everyone writing an alternative client goes through this thought
> process :-) There's no point in doing it, you cannot prove your
> transaction is not a double spend. That requires knowledge (ie, an
> index) of all transactions.

Ah; you mistake me.  I'm not interested in double spend prevention, in this 
case I'd be willing to trust the full node to return whatever block it thinks 
contains that transaction, and that it has already done double spend 
prevention.

What I want to be able to do though is calculate a balance for an aribtrary 
address.  Not every address; just the particular ones that the client is 
interested in.  It's complete overkill to require the whole block chain just 
to calculate the balance of a few addresses.

> You have to treat appearing deep in the chain as ipso-facto proof of
> validity. Lightweight/SPV clients simply must have that trust, it
> cannot be done any other way. See this article:

Not entirely.  If I ask for "the block that contains transaction with hash 
12345678abcd..." then when I get that full block, I can verify the merkle tree 
myself.  I do have to trust that the peer hasn't been adding double spends in, 
but not that the transaction is actually in the chain.

> > It should be possible to request the current pending transaction list.
> 
> I think it'd be better to implement the filtering suggestions that
> have been made. It doesn't scale to download the entire memory pool -

I'm sorry, I've only started watching this list in the last few days.  I'm not 
familiar with the filter suggestions.

I'm not entirely sure I see how a filter helps.  If I've been offline for ten 
minutes then I need all the transactions pending in the last ten minutes.  No 
amount of filtering makes that list any smaller.

> a better approach is to give the remote node a filter to match against
> transactions then have it only relay those. After setting a filter,
> transactions pending and matching would be sent in one big inv and you
> can then keep the connection open to learn about new transactions
> without needing to "drink from the firehose". Filters can be
> probabilistic and set on many different nodes to reduce the privacy
> implications.

That would be fine.  My reason for suggesting using getblocks was that it 
didn't introduce a new command.



Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Bitcoin-development] Suggestion for enhancements to getblock

[Mike Hearn]

> What I want to be able to do though is calculate a balance for an aribtrary
> address.  Not every address; just the particular ones that the client is
> interested in.  It's complete overkill to require the whole block chain just
> to calculate the balance of a few addresses.

But what is that for? You said it's for a lightweight client to do
that when it receives a transaction, to verify that all the
dependencies are in blocks recursively. But why?

> Not entirely.  If I ask for "the block that contains transaction with hash
> 12345678abcd..." then when I get that full block, I can verify the merkle tree
> myself.

Well, it's more efficient to just verify the merkle branch. But yes.

> I'm not entirely sure I see how a filter helps.  If I've been offline for ten
> minutes then I need all the transactions pending in the last ten minutes.  No
> amount of filtering makes that list any smaller.

Why do you need all of them? You just care about the ones sending
coins to you, surely?

> That would be fine.  My reason for suggesting using getblocks was that it
> didn't introduce a new command.

IMHO it's fine to introduce new commands. They'll just be ignored by
old clients in any event.

Re: [Bitcoin-development] Suggestion for enhancements to getblock

[Andy Parkins]

On Thursday 07 July 2011 17:44:48 Mike Hearn wrote:
> > What I want to be able to do though is calculate a balance for an
> > aribtrary address.  Not every address; just the particular ones that
> > the client is interested in.  It's complete overkill to require the
> > whole block chain just to calculate the balance of a few addresses.
> 
> But what is that for? You said it's for a lightweight client to do
> that when it receives a transaction, to verify that all the
> dependencies are in blocks recursively. But why?

There is no way for a client to know in advance whether any broadcast 
transaction contains a send to an address in its wallet.  So every incoming 
transaction has to be examined.

Then, there is no way to know if while you were offline any of the 
transactions in the blocks you missed contained transactions for an address 
in your wallet.

Also, a feature I am interested in supporting is a split wallet -- where the 
private key is held elsewhere.  I'd still want to be able to report the 
current balance in a particular address though.  That address can be added 
at any time.

Also, I would like to make some blockexplorer-like facilities available to 
lightweight clients.

> > Not entirely.  If I ask for "the block that contains transaction with
> > hash 12345678abcd..." then when I get that full block, I can verify
> > the merkle tree myself.
> 
> Well, it's more efficient to just verify the merkle branch. But yes.

We're only talking about one verifying one (or minimal numbers of) blocks; 
"efficient" isn't really going to matter much in that context.  Also, if 
we're talking about a situation where we don't necessarily trust the remote, 
we've got to verify the whole block, not just the one transaction we're 
interested in, since we told the remote which one we were interested in when 
we requested it.

> > I'm not entirely sure I see how a filter helps.  If I've been offline
> > for ten minutes then I need all the transactions pending in the last
> > ten minutes.  No amount of filtering makes that list any smaller.
> 
> Why do you need all of them? You just care about the ones sending
> coins to you, surely?

Is the filter going to be filter-by-address then?  I misunderstood in that 
case, I thought you were talking about filter-by-hash, which obviously tells 
you nothing about the contents of the transaction.

> > That would be fine.  My reason for suggesting using getblocks was that
> > it didn't introduce a new command.
> 
> IMHO it's fine to introduce new commands. They'll just be ignored by
> old clients in any event.

That's good to know.  I'm trying to be circumspect in what my client does; I 
want to be 100% compatible, which means if I need a new feature, it's got to 
be in the official client first.

I accept that this is all big talk, and there are plenty of people who start 
new clients and then give up; which might still happen to me.



Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: [Bitcoin-development] Suggestion for enhancements to getblock

[Gregory Maxwell]

On Thu, Jul 7, 2011 at 11:42 AM, Mike Hearn <mike@plan99.net> wrote:
[snip]
> You have to treat appearing deep in the chain as ipso-facto proof of
> validity. Lightweight/SPV clients simply must have that trust, it
> cannot be done any other way. See this article:
>
> http://code.google.com/p/bitcoinj/wiki/SecurityModel


It _could_ be done another way, with a protocol change:
http://forum.bitcoin.org/index.php?topic=21995.0