[bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Suhas Daftuar]

[-- Attachment #1: Type: text/plain, Size: 3408 bytes --]

Hi,

I'm proposing the addition of a new, optional p2p message to help improve
the way blocks are announced on the network.  The draft BIP is available
here and pasted below:
https://gist.github.com/sdaftuar/465bf008f0a4768c0def

The goal of this p2p message is to facilitate nodes being able to
optionally announce blocks with headers messages rather than with inv's,
which is particularly beneficial since the introduction of headers-first
download in Bitcoin Core 0.10.  In particular, this allows for more
efficient propagation of reorgs as it would eliminate a round trip in
network communication.

The implementation of this BIP (which includes code to directly fetch
blocks based on announced headers) is in
https://github.com/bitcoin/bitcoin/pull/6494.  For additional background,
please also see https://github.com/bitcoin/bitcoin/issues/5982.

Note that this new p2p message is optional; nodes can feel free to ignore
and continue to use inv messages to announce new blocks.

Thanks to Pieter Wuille for suggesting this idea.

Draft BIP text:

<pre>
  BIP: <unassigned>
  Title: sendheaders message
  Author: Suhas Daftuar <sdaftuar@chaincode.com>
  Status: Draft
  Type: Standards Track
  Created: 2015-05-08
</pre>

==Abstract==

Add a new message, "sendheaders", which indicates that a node prefers to
receive new block announcements via a "headers" message rather than an
"inv".

==Motivation==

Since the introduction of "headers-first" downloading of blocks in 0.10,
blocks will not be processed unless
they are able to connect to a (valid) headers chain.  Consequently, block
relay generally works
as follows:
# A node (N) announces the new tip with an "inv" message, containing the
block hash
# A peer (P) responds to the "inv" with a "getheaders" message (to request
headers up to the new tip) and a "getdata" message for the new tip itself
# N responds with a "headers" message (with the header for the new block
along with any preceding headers unknown to P) and a "block" message
containing the new block

However, in the case where a new block is being announced that builds on
the tip, it would be generally more efficient if the node N just announced
the block header for the new block, rather than just the block hash, and
saved the peer from generating and transmitting the getheaders message (and
the required block locator).

In the case of a reorg, where 1 or more blocks are disconnected, nodes
currently just send an "inv" for the new tip.  Peers currently are able to
request the new tip immediately, but wait until the headers for the
intermediate blocks are delivered before requesting those blocks.  By
announcing headers from the last fork point leading up to the new tip in
the block announcement, peers are able to request all the intermediate
blocks immediately.

==Specification==

# The sendheaders message is defined as an empty message where pchCommand
== "sendheaders"
# Upon receipt of a "sendheaders" message, the node will be permitted, but
not required, to announce new blocks by sending the header of the new block
(along with any other blocks that a node believes a peer might need in
order for the block to connect).
# Feature discovery is enabled by checking protocol version >= 70012

==Backward compatibility==

Older clients remain fully compatible and interoperable after this change.

==Implementation==

https://github.com/bitcoin/bitcoin/pull/6494

[-- Attachment #2: Type: text/html, Size: 6125 bytes --]

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Tier Nolan]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

On Thu, Sep 24, 2015 at 7:02 PM, Suhas Daftuar via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi,
>
> I'm proposing the addition of a new, optional p2p message to help improve
> the way blocks are announced on the network.  The draft BIP is available
> here and pasted below:
> https://gist.github.com/sdaftuar/465bf008f0a4768c0def
>
> The goal of this p2p message is to facilitate nodes being able to
> optionally announce blocks with headers messages rather than with inv's,
> which is particularly beneficial since the introduction of headers-first
> download in Bitcoin Core 0.10.  In particular, this allows for more
> efficient propagation of reorgs as it would eliminate a round trip in
> network communication.
>

Is there actually a requirement for the new message?  New nodes could just
unilaterally switch to sending headers and current nodes would be
compatible.

It looks like the only DOS misbehaving penalty is if the header is invalid
or if the headers don't form a chain.

[-- Attachment #2: Type: text/html, Size: 1521 bytes --]

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Suhas Daftuar]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

>
> Is there actually a requirement for the new message?  New nodes could just
> unilaterally switch to sending headers and current nodes would be
> compatible.
>

I don't believe that unilaterally switching to headers announcements would
work for all network participants -- both for users running older Bitcoin
Core versions (anything before 0.10, which I believe all ignore headers
messages) and for non-Bitcoin Core software that participates on the
network (which may ignore headers messages too, I'm not sure what all is
out there).

Even for Bitcoin Core versions 0.10 and 0.11, which process headers and use
them to determine what blocks to download, the block fetching logic is not
optimized for new block announcements via headers messages.  Part of what
is implemented in the pull request is direct fetching of blocks upon
receiving a headers message; nodes that don't implement block downloading
in response to headers announcements should continue to receive inv's, I
think -- hence this p2p message to opt-in to the new behavior.

[-- Attachment #2: Type: text/html, Size: 1639 bytes --]

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 24 September 2015 14:37:40 GMT-04:00, Suhas Daftuar via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev <
>bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>>
>> Is there actually a requirement for the new message?  New nodes could
>just
>> unilaterally switch to sending headers and current nodes would be
>> compatible.
>>
>
>I don't believe that unilaterally switching to headers announcements
>would
>work for all network participants -- both for users running older
>Bitcoin
>Core versions (anything before 0.10, which I believe all ignore headers
>messages) and for non-Bitcoin Core software that participates on the
>network (which may ignore headers messages too, I'm not sure what all
>is
>out there).

You can enable the behaviour based on advertised p2p network version.
-----BEGIN PGP SIGNATURE-----

iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBEO5
AAoJEMCF8hzn9Lncz4MH/3ztGWdFvMWWcwQsjIRH+eP6PH57WaEru1smmFYOmKrj
djdiRVdxfChxRqP3adO21RUKKchjl8DNjrFJHPFz75FSM0cDcD0QAGAHilVdnICE
LEIlTEoiIc0f1z9f/EJHSHPhiUXMnjpl/l7PYJFZV3Lt2Bl30yLsNnrp9qxjR30n
3nykZjyRad4JSavdTP6Evd3qaqwGXNUWsdObXNI+WPKlrw6hczlhFDKQ7RC1FPQU
Rbgb21pavtqLUTwbBZGUisAAc94e2Gama1p3ioUFklbVtLTdw+FtxPgV/0ZS75OR
V9pCXIbg9VM6QY4+9gYnP635+qCkqAJ4tBsYGmsT8yA=
=cF4B
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Suhas Daftuar]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

I considered that as well, but it seemed to me that other software on the
network (say, different wallet implementations) might prefer the option of
being able to bump up their protocol version in the future to pick up some
other change, without having to also opt-in to receiving
headers-announcements for blocks.

In particular, inv-based block announcements aren't going away (even in my
implementation of headers announcements, there are some edge cases where
the code would need to fall back to an inv announcement), so forcing all
software on the network to upgrade to supporting headers announcements,
whether now or in the future, seems too drastic -- I could imagine some
software not being very concerned about optimizing block relay in this
particular way.

On Thu, Sep 24, 2015 at 2:41 PM, Peter Todd <pete@petertodd.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
>
>
> On 24 September 2015 14:37:40 GMT-04:00, Suhas Daftuar via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
> >On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev <
> >bitcoin-dev@lists.linuxfoundation.org> wrote:
> >
> >>
> >> Is there actually a requirement for the new message?  New nodes could
> >just
> >> unilaterally switch to sending headers and current nodes would be
> >> compatible.
> >>
> >
> >I don't believe that unilaterally switching to headers announcements
> >would
> >work for all network participants -- both for users running older
> >Bitcoin
> >Core versions (anything before 0.10, which I believe all ignore headers
> >messages) and for non-Bitcoin Core software that participates on the
> >network (which may ignore headers messages too, I'm not sure what all
> >is
> >out there).
>
> You can enable the behaviour based on advertised p2p network version.
> -----BEGIN PGP SIGNATURE-----
>
> iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBEO5
> AAoJEMCF8hzn9Lncz4MH/3ztGWdFvMWWcwQsjIRH+eP6PH57WaEru1smmFYOmKrj
> djdiRVdxfChxRqP3adO21RUKKchjl8DNjrFJHPFz75FSM0cDcD0QAGAHilVdnICE
> LEIlTEoiIc0f1z9f/EJHSHPhiUXMnjpl/l7PYJFZV3Lt2Bl30yLsNnrp9qxjR30n
> 3nykZjyRad4JSavdTP6Evd3qaqwGXNUWsdObXNI+WPKlrw6hczlhFDKQ7RC1FPQU
> Rbgb21pavtqLUTwbBZGUisAAc94e2Gama1p3ioUFklbVtLTdw+FtxPgV/0ZS75OR
> V9pCXIbg9VM6QY4+9gYnP635+qCkqAJ4tBsYGmsT8yA=
> =cF4B
> -----END PGP SIGNATURE-----
>
>

[-- Attachment #2: Type: text/html, Size: 2999 bytes --]

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 24 September 2015 14:56:23 GMT-04:00, Suhas Daftuar <sdaftuar@gmail.com> wrote:
>I considered that as well, but it seemed to me that other software on
>the
>network (say, different wallet implementations) might prefer the option
>of
>being able to bump up their protocol version in the future to pick up
>some
>other change, without having to also opt-in to receiving
>headers-announcements for blocks.
>
>In particular, inv-based block announcements aren't going away (even in
>my
>implementation of headers announcements, there are some edge cases
>where
>the code would need to fall back to an inv announcement), so forcing
>all
>software on the network to upgrade to supporting headers announcements,
>whether now or in the future, seems too drastic -- I could imagine some
>software not being very concerned about optimizing block relay in this
>particular way.

Block headers are so small - 80 bytes - that it may be reasonable to just stop using the inv mechanism for them in favor of always sending headers. IIRC a inv is 32 bytes of digest and another four bytes or something of the inv string itself - that's already nearly half of the header.

Meanwhile reducing the amount of state in the protocol does have some value, and decreasing overall latency for headers to get around the network certainely isnt a bad thing.
-----BEGIN PGP SIGNATURE-----

iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBE60
AAoJEMCF8hzn9Lncz4MH/jybITh0VWtf+2MotWZOdMIiQtmWZ6Ly2yiDXwi3atu+
MEA6yx9vPFV8P1ZKIZzVtr/4Iu3gBHBdDxAzQW0SjreTLdzZ1+d28/A2kYD4+es7
MFD8rDV/kPtnu8ajMkS9bfmrU0WfkgSSB2fUheT+kqgH/ejIJBISo8BpQZbz7f4B
M+D+hoNadcqWcZZKBHT+o5o7v3jJwxh8qpJgMMZrtN/QfFJK5UVdU4I/hEd89XP9
XD/y29ykWAFQPDdBKMGIUj1csUGlyS5kFXp6ZLVtAZWHIgfZ1R/qOhIUcRwRxZjc
JXZEWrMGTIXr2zkX9mtLzfjAzDc6ZULoEAHCV3sVa0M=
=SLUT
-----END PGP SIGNATURE-----