[Bitcoin-development] BIP 35: add mempool message

[Bitcoin-development] BIP 35: add mempool message

[Jeff Garzik]

Consensus was we should do a BIP for all P2P changes, so here it is...
 feedback requested.

https://en.bitcoin.it/wiki/BIP_0035

Abstract
-------------------------------------------
Make a network node's transaction memory pool accessible via a new
"mempool" message.  Extend the existing "getdata" message behavior to permit
accessing the transaction memory pool.


Motivation
-------------------------------------------
Several use cases make it desireable to expore a network node's transaction
memory pool:
* SPV clients, wishing to obtain zero-confirmation transactions sent or
  received.
* Miners, downloading existing network transactions after a restart.
* Remote network diagnostics.


Specification
-------------------------------------------
1) Upon receipt of a "mempool" message, the node will respond
   with an "inv" message containing MSG_TX hashes of all the
   transactions in the node's transaction memory pool.

   An "inv" message is always returned, even if empty.

2) The typical node behavior in response to an "inv" is "getdata".

   However, the reference Satoshi implementation ignores requests
   for transaction hashes outside that which is recently relayed.

   To support "mempool", an implementation must extend its "getdata"
   message support to querying the memory pool.

3) Feature discovery is enabled by checking two "version" message attributes:

   a) Protocol version >= 60002
   b) NODE_NETWORK bit set in nServices


Backwards compatibility
-------------------------------------------
Older clients remain 100% compatible and interoperable after this change.


Implementation
-------------------------------------------
See https://github.com/bitcoin/bitcoin/pull/1641

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com

Re: [Bitcoin-development] BIP 35: add mempool message

[Amir Taaki]

The format for "mempool" packet is missing. I'm guessing that it is an empty message, right?

Might be good to add that.



----- Original Message -----
From: Jeff Garzik <jgarzik@exmulti.com>
To: Bitcoin Development <bitcoin-development@lists.sourceforge.net>
Cc: 
Sent: Thursday, August 16, 2012 6:32 PM
Subject: [Bitcoin-development] BIP 35: add mempool message

Consensus was we should do a BIP for all P2P changes, so here it is...
feedback requested.

https://en.bitcoin.it/wiki/BIP_0035

Abstract
-------------------------------------------
Make a network node's transaction memory pool accessible via a new
"mempool" message.  Extend the existing "getdata" message behavior to permit
accessing the transaction memory pool.


Motivation
-------------------------------------------
Several use cases make it desireable to expore a network node's transaction
memory pool:
* SPV clients, wishing to obtain zero-confirmation transactions sent or
  received.
* Miners, downloading existing network transactions after a restart.
* Remote network diagnostics.


Specification
-------------------------------------------
1) Upon receipt of a "mempool" message, the node will respond
   with an "inv" message containing MSG_TX hashes of all the
   transactions in the node's transaction memory pool.

   An "inv" message is always returned, even if empty.

2) The typical node behavior in response to an "inv" is "getdata".

   However, the reference Satoshi implementation ignores requests
   for transaction hashes outside that which is recently relayed.

   To support "mempool", an implementation must extend its "getdata"
   message support to querying the memory pool.

3) Feature discovery is enabled by checking two "version" message attributes:

   a) Protocol version >= 60002
   b) NODE_NETWORK bit set in nServices


Backwards compatibility
-------------------------------------------
Older clients remain 100% compatible and interoperable after this change.


Implementation
-------------------------------------------
See https://github.com/bitcoin/bitcoin/pull/1641

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 35: add mempool message

[Jeff Garzik]

On Thu, Aug 16, 2012 at 1:40 PM, Amir Taaki <zgenjix@yahoo.com> wrote:
> The format for "mempool" packet is missing. I'm guessing that it is an empty message, right?

Yes, it is an empty message.  BIP updated.
-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com

Re: [Bitcoin-development] BIP 35: add mempool message

[Pieter Wuille]

On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote:
> Consensus was we should do a BIP for all P2P changes, so here it is...
>  feedback requested.
> 
> https://en.bitcoin.it/wiki/BIP_0035

I like the idea of being able to query the memory pool of a node; the
implementation is straightforward, which is good. Maybe effectively using the
command can be added? I suppose it is interesting in general for nodes to
get a memory pool refill at startup anyway.

> 1) Upon receipt of a "mempool" message, the node will respond
>    with an "inv" message containing MSG_TX hashes of all the
>    transactions in the node's transaction memory pool.
> 
>    An "inv" message is always returned, even if empty.

I'm not sure about this last. What is it good for? inv packets can always be
sent, even not in response to others, so it is not that this gives you an
acknowledgement the mempool is updated?

> 3) Feature discovery is enabled by checking two "version" message attributes:
> 
>    a) Protocol version >= 60002
>    b) NODE_NETWORK bit set in nServices

This seems safe, although it forces other full implementations that want to
expose protocol version 60002 (or later) to also implement this. What do they
think about this?

I would like to suggest to allocate an extra service bit for this. We still
have 63 left, and this is a well-defined and useful extra service that was
not yet provided by any earlier node. Doing that would also mean that
mempool-providing survices may be discovered before connecting to them, as
the service bits are carried around in addr messages. Any opinions about that?

-- 
Pieter

Re: [Bitcoin-development] BIP 35: add mempool message

[Jeff Garzik]

On Thu, Aug 16, 2012 at 1:56 PM, Pieter Wuille <pieter.wuille@gmail.com> wrote:
> I suppose it is interesting in general for nodes to
> get a memory pool refill at startup anyway.

Yes.

>>    An "inv" message is always returned, even if empty.
>
> I'm not sure about this last. What is it good for? inv packets can always be
> sent, even not in response to others, so it is not that this gives you an
> acknowledgement the mempool is updated?

A simple guarantee of 1:1 correspondence between request and response.
 The bitcoin protocol sometimes simply elides a response when the
response would be empty, and this makes it difficult to know whether a
request is timing out or already processed.

Sending a ping(nonce) after each P2P command is another way of achieving same :)

> This seems safe, although it forces other full implementations that want to
> expose protocol version 60002 (or later) to also implement this. What do they
> think about this?
>
> I would like to suggest to allocate an extra service bit for this. We still
> have 63 left, and this is a well-defined and useful extra service that was
> not yet provided by any earlier node. Doing that would also mean that
> mempool-providing survices may be discovered before connecting to them, as
> the service bits are carried around in addr messages. Any opinions about that?

An nServices bit would be a better fit for this optional service, but
nServices bits seemed like a scarce resource, so I elected to be
conservative.

Absent the scarce-resource concern, I'd vote for an nServices bit.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com

Re: [Bitcoin-development] BIP 35: add mempool message

[Amir Taaki]

My thoughts:

The extension is simple. It's only really useful for the use-cases listed if the majority of nodes implement it. As I view the proposal, it is perfectly simple and uncomplicated. If it's implemented, then I suggest to just increment version and make it part of the protocol.

On the flipside it is another notch in complicating an already diffuse protocol, but it seems a rather benign offense in that regard compared to other changes (past and future).



----- Original Message -----
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Jeff Garzik <jgarzik@exmulti.com>
Cc: Bitcoin Development <bitcoin-development@lists.sourceforge.net>
Sent: Thursday, August 16, 2012 6:56 PM
Subject: Re: [Bitcoin-development] BIP 35: add mempool message

On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote:
> Consensus was we should do a BIP for all P2P changes, so here it is...
>  feedback requested.
> 
> https://en.bitcoin.it/wiki/BIP_0035

I like the idea of being able to query the memory pool of a node; the
implementation is straightforward, which is good. Maybe effectively using the
command can be added? I suppose it is interesting in general for nodes to
get a memory pool refill at startup anyway.

> 1) Upon receipt of a "mempool" message, the node will respond
>    with an "inv" message containing MSG_TX hashes of all the
>    transactions in the node's transaction memory pool.
> 
>    An "inv" message is always returned, even if empty.

I'm not sure about this last. What is it good for? inv packets can always be
sent, even not in response to others, so it is not that this gives you an
acknowledgement the mempool is updated?

> 3) Feature discovery is enabled by checking two "version" message attributes:
> 
>    a) Protocol version >= 60002
>    b) NODE_NETWORK bit set in nServices

This seems safe, although it forces other full implementations that want to
expose protocol version 60002 (or later) to also implement this. What do they
think about this?

I would like to suggest to allocate an extra service bit for this. We still
have 63 left, and this is a well-defined and useful extra service that was
not yet provided by any earlier node. Doing that would also mean that
mempool-providing survices may be discovered before connecting to them, as
the service bits are carried around in addr messages. Any opinions about that?

-- 
Pieter

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 35: add mempool message

[Stefan Thomas]

> This seems safe, although it forces other full implementations that want to
> expose protocol version 60002 (or later) to also implement this. What do they
> think about this?

BitcoinJS will implement it, it's a useful feature and there is no
reason not to support it.

Two comments from my end:

- This is just a thought, but I wouldn't mind using a new inv_type for
this, e.g. MSG_MEMTX. I could conceivably see a future where broadcast
and relay txs are stored in a very fast local cache whereas the general
mempool is stored in a slower data structure. By being able to
distinguish incoming getdata requests I can save a few milliseconds by
querying the right storage right away. Might also help with things like
telling apart broadcast/relayed transactions from the response to a
mempool request for purposes like DoS scoring etc.

Not a big deal by any means, but I also don't see a downside to it.
inv_types are not a scarce resource, we have four billion of them available.

For now clients would just treat MSG_TX and MSG_MEMTX interchangeably.

- If a node doesn't have anything in it's mempool it sends back an empty
inv message. This is either ambiguous (if other things also send empty
inv messages in the future) or arbitrary (why should an empty inv be
associated with a mempool request of all things.) Instead why not
respond with an inv message that contains a single element of type
MSG_MEMTX and hash 0. That would a very direct way to indicate that this
response is associated with a mempool request.


I'm not married to either suggestion, just trying to add my perspective.
One thing you notice when reimplementing Bitcoin is that Bitcoin's
protocol leaves out a lot of information not for space reasons, but
because the reference client's implementation doesn't happen to need it.
Sometimes however this locks other clients into doing things the same
way. If we can make the protocol a bit richer, especially if this
doesn't cost any extra bytes, then we should consider it as it might
help some implementation down the road make a neat optimization.


On 8/16/2012 7:56 PM, Pieter Wuille wrote:
> On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote:
>> Consensus was we should do a BIP for all P2P changes, so here it is...
>>  feedback requested.
>>
>> https://en.bitcoin.it/wiki/BIP_0035
> I like the idea of being able to query the memory pool of a node; the
> implementation is straightforward, which is good. Maybe effectively using the
> command can be added? I suppose it is interesting in general for nodes to
> get a memory pool refill at startup anyway.
>
>> 1) Upon receipt of a "mempool" message, the node will respond
>>    with an "inv" message containing MSG_TX hashes of all the
>>    transactions in the node's transaction memory pool.
>>
>>    An "inv" message is always returned, even if empty.
> I'm not sure about this last. What is it good for? inv packets can always be
> sent, even not in response to others, so it is not that this gives you an
> acknowledgement the mempool is updated?
>
>> 3) Feature discovery is enabled by checking two "version" message attributes:
>>
>>    a) Protocol version >= 60002
>>    b) NODE_NETWORK bit set in nServices
> This seems safe, although it forces other full implementations that want to
> expose protocol version 60002 (or later) to also implement this. What do they
> think about this?
>
> I would like to suggest to allocate an extra service bit for this. We still
> have 63 left, and this is a well-defined and useful extra service that was
> not yet provided by any earlier node. Doing that would also mean that
> mempool-providing survices may be discovered before connecting to them, as
> the service bits are carried around in addr messages. Any opinions about that?
>

Re: [Bitcoin-development] BIP 35: add mempool message

[Alan Reiner]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]

On Thu, Aug 16, 2012 at 2:04 PM, Jeff Garzik <jgarzik@exmulti.com> wrote:

> On Thu, Aug 16, 2012 at 1:56 PM, Pieter Wuille <pieter.wuille@gmail.com>
> wrote:
> > I suppose it is interesting in general for nodes to
> > get a memory pool refill at startup anyway.
>
> Yes.
>
> >>    An "inv" message is always returned, even if empty.
> >
> > I'm not sure about this last. What is it good for? inv packets can
> always be
> > sent, even not in response to others, so it is not that this gives you an
> > acknowledgement the mempool is updated?
>
> A simple guarantee of 1:1 correspondence between request and response.
>  The bitcoin protocol sometimes simply elides a response when the
> response would be empty, and this makes it difficult to know whether a
> request is timing out or already processed.
>
> Sending a ping(nonce) after each P2P command is another way of achieving
> same :)
>
>

Is there a problem with sending unrecognized messages to nodes?   If we
create a new message type specifically asking for memory pool transactions,
and we broadcast it to all nodes that we are connected to, and none of them
respond, then either there are no tx in their memory pools, or they don't
recognize the message and ignore it.  Either way, you're not going to get
any extra information out of them.  If you really care, a simple ping can
identify whether they're still connected and should've responded (as Jeff
said).

As long as the older node won't cut you off for sending one unrecognized
request, it seems that you can get by fine without requiring that bit.  I
guess it depends on the utility of definitively identifying whether a node
supports the functionality.  I personally don't feel like it's critical,
especially considering that this is most useful only during the transient
period when it's not normal for nodes to support it yet.

[-- Attachment #2: Type: text/html, Size: 2440 bytes --]

Re: [Bitcoin-development] BIP 35: add mempool message

[Amir Taaki]

MSG_MEMTX solves the issue of not knowing whether a given inv is in response to a "mempool" command or not.

I don't buy the argument that always sending a response "inv" makes things easier because code should always be able to handle misbehaviour from the remote node (ommiting the "inv"). However I would argue that it is good to have it, as it makes designing flows of logic much easier (first send this, wait for response, do this, ...).



----- Original Message -----
From: Stefan Thomas <moon@justmoon.de>
To: bitcoin-development@lists.sourceforge.net
Cc: 
Sent: Thursday, August 16, 2012 8:21 PM
Subject: Re: [Bitcoin-development] BIP 35: add mempool message

> This seems safe, although it forces other full implementations that want to
> expose protocol version 60002 (or later) to also implement this. What do they
> think about this?

BitcoinJS will implement it, it's a useful feature and there is no
reason not to support it.

Two comments from my end:

- This is just a thought, but I wouldn't mind using a new inv_type for
this, e.g. MSG_MEMTX. I could conceivably see a future where broadcast
and relay txs are stored in a very fast local cache whereas the general
mempool is stored in a slower data structure. By being able to
distinguish incoming getdata requests I can save a few milliseconds by
querying the right storage right away. Might also help with things like
telling apart broadcast/relayed transactions from the response to a
mempool request for purposes like DoS scoring etc.

Not a big deal by any means, but I also don't see a downside to it.
inv_types are not a scarce resource, we have four billion of them available.

For now clients would just treat MSG_TX and MSG_MEMTX interchangeably.

- If a node doesn't have anything in it's mempool it sends back an empty
inv message. This is either ambiguous (if other things also send empty
inv messages in the future) or arbitrary (why should an empty inv be
associated with a mempool request of all things.) Instead why not
respond with an inv message that contains a single element of type
MSG_MEMTX and hash 0. That would a very direct way to indicate that this
response is associated with a mempool request.


I'm not married to either suggestion, just trying to add my perspective.
One thing you notice when reimplementing Bitcoin is that Bitcoin's
protocol leaves out a lot of information not for space reasons, but
because the reference client's implementation doesn't happen to need it.
Sometimes however this locks other clients into doing things the same
way. If we can make the protocol a bit richer, especially if this
doesn't cost any extra bytes, then we should consider it as it might
help some implementation down the road make a neat optimization.


On 8/16/2012 7:56 PM, Pieter Wuille wrote:
> On Thu, Aug 16, 2012 at 01:32:04PM -0400, Jeff Garzik wrote:
>> Consensus was we should do a BIP for all P2P changes, so here it is...
>>  feedback requested.
>>
>> https://en.bitcoin.it/wiki/BIP_0035
> I like the idea of being able to query the memory pool of a node; the
> implementation is straightforward, which is good. Maybe effectively using the
> command can be added? I suppose it is interesting in general for nodes to
> get a memory pool refill at startup anyway.
>
>> 1) Upon receipt of a "mempool" message, the node will respond
>>    with an "inv" message containing MSG_TX hashes of all the
>>    transactions in the node's transaction memory pool.
>>
>>    An "inv" message is always returned, even if empty.
> I'm not sure about this last. What is it good for? inv packets can always be
> sent, even not in response to others, so it is not that this gives you an
> acknowledgement the mempool is updated?
>
>> 3) Feature discovery is enabled by checking two "version" message attributes:
>>
>>    a) Protocol version >= 60002
>>    b) NODE_NETWORK bit set in nServices
> This seems safe, although it forces other full implementations that want to
> expose protocol version 60002 (or later) to also implement this. What do they
> think about this?
>
> I would like to suggest to allocate an extra service bit for this. We still
> have 63 left, and this is a well-defined and useful extra service that was
> not yet provided by any earlier node. Doing that would also mean that
> mempool-providing survices may be discovered before connecting to them, as
> the service bits are carried around in addr messages. Any opinions about that?
>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] BIP 35: add mempool message

[Jeff Garzik]

On MSG_MEMTX:  The current version has a much higher Just Works value.

On empty "inv":  It is generally better to do something
unconditionally, than have a response generated only under certain
conditions.

And Alan is correct to note that unknown messages are ignored
(intentionally, for expansion).  However, unconditionally returning a
response has little to do with feature probing/discovery.  It is
simply a clear, deterministic indication that processing is complete,
for each invocation.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com

Re: [Bitcoin-development] BIP 35: add mempool message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1785 bytes --]

I think MSG_TX is fine. Simply sending an inv to the other node at startup
would work, but it's better to request it explicitly as it will let the
connecting peer configure a bloom filter before requesting mempool
contents. It's already too heavy for mobile clients to download the entire
mempool contents at startup so I probably wouldn't implement/activate
support for this on the bitcoinj side until bloom filtering is done, and
then this BIP would have to be updated to reflect that the response from
mempool is filtered.

On Thu, Aug 16, 2012 at 11:05 PM, Jeff Garzik <jgarzik@exmulti.com> wrote:

> On MSG_MEMTX:  The current version has a much higher Just Works value.
>
> On empty "inv":  It is generally better to do something
> unconditionally, than have a response generated only under certain
> conditions.
>
> And Alan is correct to note that unknown messages are ignored
> (intentionally, for expansion).  However, unconditionally returning a
> response has little to do with feature probing/discovery.  It is
> simply a clear, deterministic indication that processing is complete,
> for each invocation.
>
> --
> Jeff Garzik
> exMULTI, Inc.
> jgarzik@exmulti.com
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 2510 bytes --]

Re: [Bitcoin-development] BIP 35: add mempool message

[Pieter Wuille]

On Thu, Aug 16, 2012 at 05:05:58PM -0400, Jeff Garzik wrote:
> On MSG_MEMTX:  The current version has a much higher Just Works value.
> 
> On empty "inv":  It is generally better to do something
> unconditionally, than have a response generated only under certain
> conditions.
> 
> And Alan is correct to note that unknown messages are ignored
> (intentionally, for expansion).  However, unconditionally returning a
> response has little to do with feature probing/discovery.  It is
> simply a clear, deterministic indication that processing is complete,
> for each invocation.

I disagree. Returning an empty "inv" is a very strange way of replying
"empty mempool". Bitcoin P2P is not a request-response protocol, and
"inv" messages are sent where there are inventory items to send. The
reaction to a request (for example "getblocks") can be nothing, or one
or more "inv" messages if necessary. Special casing an empty "inv" to
mean empty mempool is trying to hack a request-response system on top
of the asynchronous system.

If there is need for confirming the transmission of the mempool is
complete, the proposal to use a MSG_MEMTX sounds good to me. No client
will ever receive such an inv without requesting the mempool, and
implementing handling MSG_MEMTX is trivial.

-- 
Pieter

Re: [Bitcoin-development] BIP 35: add mempool message

[Jeff Garzik]

On Fri, Aug 17, 2012 at 9:40 AM, Pieter Wuille <pieter.wuille@gmail.com> wrote:
> On Thu, Aug 16, 2012 at 05:05:58PM -0400, Jeff Garzik wrote:
>> On MSG_MEMTX:  The current version has a much higher Just Works value.
>>
>> On empty "inv":  It is generally better to do something
>> unconditionally, than have a response generated only under certain
>> conditions.
>>
>> And Alan is correct to note that unknown messages are ignored
>> (intentionally, for expansion).  However, unconditionally returning a
>> response has little to do with feature probing/discovery.  It is
>> simply a clear, deterministic indication that processing is complete,
>> for each invocation.
>
> I disagree. Returning an empty "inv" is a very strange way of replying
> "empty mempool". Bitcoin P2P is not a request-response protocol, and
> "inv" messages are sent where there are inventory items to send. The
> reaction to a request (for example "getblocks") can be nothing, or one
> or more "inv" messages if necessary. Special casing an empty "inv" to
> mean empty mempool is trying to hack a request-response system on top
> of the asynchronous system.

OK, just updated 'mempool' branch to not return "inv" if mempool is empty.


> If there is need for confirming the transmission of the mempool is
> complete, the proposal to use a MSG_MEMTX sounds good to me. No client
> will ever receive such an inv without requesting the mempool, and
> implementing handling MSG_MEMTX is trivial.

MSG_MEMTX is not a good idea for this use case.  Just sent a ping(nonce).

Bitcoin P2P processes requests in-order, and responds accordingly.
The remote end may insert asynchronous messages into the response
stream, certainly, but responses to queries are processed and returned
in-order.  A 'getdata' response is fully sent before a 'ping' response
is sent, etc.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com