From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 826E5CC2 for ; Mon, 11 Dec 2017 21:56:10 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 9574EE7 for ; Mon, 11 Dec 2017 21:56:09 +0000 (UTC) Received: by mail-it0-f50.google.com with SMTP id r6so19434056itr.3 for ; Mon, 11 Dec 2017 13:56:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=hdIGP5V3oNovMEemLh1PDM9TpfRBmdH4YC69V0/9IMQ=; b=IoOdpSZNx5X1Vf41A8/RA030+7uVd3BeTO3P61rEgq9u1j85+LriNDJDKrEOm79HkR pQ+wS+Zip/40wKGuI9NdOuV7CbomUfebbItH/Rm+IYA654eaJpQxpXKaWUA9R1muHG3z 7HkkkNDv6Bmm77pStou3tmC+e+xd/9IS+/uR8lUe3jNVuA29B8JSDvtqQB27e0BiKsH/ sYfsRch0N9pfl4mIhzrS0bbxAVePZstWYJV14tRQKhSQOZDXbNPFTjPqBE7Mgwl5wMuI PMGcKb6xGoGSxIfRvmbS4p6G4mwfXXtLzM+h+bIHgsWAdkbUQ6hwPA7q2WaKaqcns5Wg RV5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=hdIGP5V3oNovMEemLh1PDM9TpfRBmdH4YC69V0/9IMQ=; b=fXQ33+QIM2cOVQYrQ3E0rHrvohZoY1Ytby45vVfmhGCzSkBBgpbb/QhBGxVJH9r2XR kq0WZQZl/eqdozL4zNQoXiw3Y+TcHkqCnzGHcKls4gO1GjixOQF9lzr0hEX7/9fJ0mq2 SM884PRJgkhE0aqpRSQN0Etwshws1f93DCRrx58QlFPMpKa6cchEHgbO6J0sthlEjc9p 3ThpYGHYqFfZZPnKL+d2Elfk3ZMTIBy1Sf0yLCc962KUL75ITRnXeKVuCvNb6lRGlJr8 LnNX7ekS1IT/yQ2r9j7E4Ulf/tiPgRylY8qhEBtUxP15a+5Iu3b9p/RsIPwJA6DuqPAM rhYQ== X-Gm-Message-State: AKGB3mK8S0CSWtCHmQ/ZSEczwkQff/BgrMLjd3dC+K6bKFa4j1fCHEjD HKQciiPP9qUA6HqiR2HhtVRjFmxGTfhupZoOcpYL8w== X-Google-Smtp-Source: ACJfBotH1SDgggmlNJoL/ILdaXackQ/Ml8KSrG/Z0yVpNtJr8f8j1pZLUVz1fnQmDiFLo2p/Bnwn0pST+O9vIpHjhDM= X-Received: by 10.36.129.212 with SMTP id q203mr3291311itd.152.1513029368727; Mon, 11 Dec 2017 13:56:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.13.9 with HTTP; Mon, 11 Dec 2017 13:56:08 -0800 (PST) In-Reply-To: References: From: Jim Posen Date: Mon, 11 Dec 2017 13:56:08 -0800 Message-ID: To: Gregory Maxwell Content-Type: multipart/alternative; boundary="94eb2c08b2380ecd1e0560179ae5" X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Mon, 11 Dec 2017 21:57:51 +0000 Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] "Compressed" headers stream X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Dec 2017 21:56:10 -0000 --94eb2c08b2380ecd1e0560179ae5 Content-Type: text/plain; charset="UTF-8" On Mon, Dec 11, 2017 at 1:04 PM, Gregory Maxwell wrote: > On Mon, Dec 11, 2017 at 8:40 PM, Jim Posen via bitcoin-dev > wrote: > > Firstly, I don't like the idea of making the net header encoding > dependent > > on the specific header validation rules that Bitcoin uses (eg. the fact > that > > difficulty is only recalculated every 2016 blocks). This would be > coupling > > In the last proposal I recall writing up, there was a one byte flag on > each header to indicate what was included. > > Is there a link somewhere to that proposal? The only thing I could find was your forwarded email on this thread. > Nbits _never_ needs to be sent even with other consensus rules because > its more or less necessarily a strict function of the prior headers. > This still holds in every clone of Bitcoin I'm aware of; sending it > with the first header in a group probably makes sense so it can be > checked independently. > > > with insufficient benefit. > > another >18% reduction in size beyond the removal of prev. is not > insubstantial by any means. I don't think it should lightly be > ignored. > > Omitting nBits entirely seems reasonable, I wrote up a possible implementation here . The downside is that it is more complex because it leaks into the validation code. The extra 4 byte savings is certainly nice though. > Prev omission itself is not, sadly, magically compatible: I am quite > confident that if there is a bitcoin hardfork it would recover the > nbits/4-guarenteed always-zero bits of prev to use as extra nonce for > miners. This has been proposed many times, implemented at least once, > and the current requirement for mining infrastructure to reach inside > the coinbase txn to increment a nonce has been a reliable source of > failures. So I think we'd want to have the encoding able to encode > leading prev bits. > > Many altcoins also change the header structures. If the better thing > is altcoin incompatible, we should still do it. Doing otherwise would > competitively hobble Bitcoin especially considering the frequent > recklessly incompetent moves made by various altcoins and the near > total lack of useful novel development we've seen come out of the > clones. > > Probably the most important change in a new header message wouldn't be > the encoding, but it would be changing the fetching mechanism so that > header sets could be pulled in parallel, etc. > > I would rather not change the serialization of existing messages, > nodes are going to have to support speaking both messages for a long > time, and I think we already want a different protocol flow for > headers fetching in any case. > Can you elaborate on how parallel header fetching might work? getheaders requests could probably already be pipelined, where the node requests the next 2,000 headers before processing the current batch (though would make sense to check that they are all above min difficulty first). I'm open to more ideas on how to optimize the header download or design the serialization format to be more flexible, but I'm concerned that we forgo a 40-45% bandwidth savings on the current protocol for a long time because something better might be possible later on or there might be a hard fork that at some point requires another upgrade. I do recognize that supporting multiple serialization formats simultaneously adds code complexity, but in this case the change seems simple enough to me that the tradeoff is worth it. --94eb2c08b2380ecd1e0560179ae5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On M= on, Dec 11, 2017 at 1:04 PM, Gregory Maxwell <greg@xiph.org> wr= ote:
On Mon, Dec 11, 201= 7 at 8:40 PM, Jim Posen via bitcoin-dev
<bitcoin-dev@li= sts.linuxfoundation.org> wrote:
> Firstly, I don't like the idea of making the net header encoding d= ependent
> on the specific header validation rules that Bitcoin uses (eg. the fac= t that
> difficulty is only recalculated every 2016 blocks). This would be coup= ling

In the last proposal I recall writing up, there was a one byte flag = on
each header to indicate what was included.


Is there a link somewhere to that prop= osal? The only thing I could find was your forwarded em= ail=C2=A0on this thread.
=C2=A0
Nbits _never_ needs to be sent even with other consensus rules because
its more or less necessarily a strict function of the prior headers.
This still holds in every clone of Bitcoin I'm aware of; sending it
with the first header in a group probably makes sense so it can be
checked independently.

> with insufficient benefit.

another >18% reduction in size beyond the removal of prev. is not
insubstantial by any means.=C2=A0 I don't think it should lightly be ignored.


Omitting nBits entirely seems reasonab= le, I wrote up a possible implementation here. Th= e downside is that it is more complex because it leaks into the validation = code. The extra 4 byte savings is certainly nice though.
=C2=A0
Prev omission itself is not, sadly, magically compatible:=C2=A0 I am quite<= br> confident that if there is a bitcoin hardfork it would recover the
nbits/4-guarenteed always-zero bits of prev to use as extra nonce for
miners. This has been proposed many times, implemented at least once,
and the current requirement for mining infrastructure to reach inside
the coinbase txn to increment a nonce has been a reliable source of
failures.=C2=A0 So I think we'd want to have the encoding able to encod= e
leading prev bits.

Many altcoins also change the header structures. If the better thing
is altcoin incompatible, we should still do it. Doing otherwise would
competitively hobble Bitcoin especially considering the frequent
recklessly incompetent moves made by various altcoins and the near
total lack of useful novel development we've seen come out of the
clones.

Probably the most important change in a new header message wouldn't be<= br> the encoding, but it would be changing the fetching mechanism so that
header sets could be pulled in parallel, etc.

I would rather not change the serialization of existing messages,
nodes are going to have to support speaking both messages for a long
time, and I think we already want a different protocol flow for
headers fetching in any case.

Can you elaborate o= n how parallel header fetching might work? getheaders requests could probab= ly already be pipelined, where the node requests the next 2,000 headers bef= ore processing the current batch (though would make sense to check that the= y are all above min difficulty first).

=
I'm open to more ideas on how to optim= ize the header download or design the serialization format to be more flexi= ble, but I'm concerned that we forgo a 40-45% bandwidth savings on the = current protocol for a long time because something better might be possible= later on or there might be a hard fork that at some point requires another= upgrade. I do recognize that supporting multiple serialization formats sim= ultaneously adds code complexity, but in this case the change seems simple = enough to me that the tradeoff is worth it.=C2=A0
--94eb2c08b2380ecd1e0560179ae5--