From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 557F5C0051 for ; Tue, 18 Aug 2020 18:56:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3AE648525D for ; Tue, 18 Aug 2020 18:56:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xBUIZRDJv3-k for ; Tue, 18 Aug 2020 18:56:18 +0000 (UTC) X-Greylist: delayed 00:45:00 by SQLgrey-1.7.6 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) by fraxinus.osuosl.org (Postfix) with ESMTPS id EF38C851D6 for ; Tue, 18 Aug 2020 18:56:17 +0000 (UTC) Received: by mail-pj1-f65.google.com with SMTP id d4so9633401pjx.5 for ; Tue, 18 Aug 2020 11:56:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=voskuil-org.20150623.gappssmtp.com; s=20150623; h=content-transfer-encoding:from:mime-version:subject:date:message-id :references:cc:in-reply-to:to; bh=Ws5FdAk8wuXIL9zyeOB9G35pZvi14IGIz+4A9/WyHkg=; b=KrTWsd9m4yRpNP8Jj5rqqtvnOSDdktpyR1qJls7zjtlAdrI1CX3EvJlHeIjuFhHrU7 R0diQy5MEajVhen9daek+C5x7bxrQGacNSyCZPvZGsSOdJ/L5zgaogTye+of6+tjbkY3 tg/WfHuAso0TiOHI4577IuqHeXMOrGExa4ZPWVjpAW6pzNU3bsqMHb+FoLrkqLAavWtI 95Qbtdq15gZouYlKvIJozYq1xlD+qxamkfxVsk6NsR4dnrhWByEWfNSZf0hgAnh/+qDk Z7i5Lq4nrGh/HieSuHSzxha4FCovDyRTzwbJypt0nSuYjWqdsarNbWtYj5GQaoNDCVMl FwOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:content-transfer-encoding:from:mime-version :subject:date:message-id:references:cc:in-reply-to:to; bh=Ws5FdAk8wuXIL9zyeOB9G35pZvi14IGIz+4A9/WyHkg=; b=CzzFqnjKYEQdJfJ/n6xRVSTJq5As+QJz9V60YjVmOQg2EvBqz08zNIdhroZ4uaLQ2q RjeP4CxFGjUDMplPS8Sb0fi4iV10Gwz/qwbk9l8k8nibirAm3IHZMBzOWY+tkCbQ38WL tldXk5iYyWDVTjOaT8PL4LpToH/Hyb6dTWfcW8FmUFeNBk2x8V7jV1XJgT8J6TwljXtW jN0r7M4sKwAOhXlwON6E1eyQLuR8HSvMOmWbF3gJOXKQ/5McKF05IGcQw5zNdfcVNwvM Xtxu3ERzLXro4KppxLBP2O5YV08VHwmGXMwPQrwcpFgRfhaXRg2uUdgs6KLokr9rON+1 LH4g== X-Gm-Message-State: AOAM533xCiWo4FvN1Q6Yy34xap949NPzM0jTJdzoh1e6z2cH1N5LpbKK o4UyEy0BuxAoneJ5OBfh3VIEmw== X-Google-Smtp-Source: ABdhPJy2kBDU4KE9xER56SEAeOEjzO0UaIuAKKnxcd5gvtVJWApJEb/58W+VjftUydXb2noOjAWArg== X-Received: by 2002:a17:902:7605:: with SMTP id k5mr16661556pll.122.1597776977241; Tue, 18 Aug 2020 11:56:17 -0700 (PDT) Received: from ?IPv6:2600:380:7066:ce6e:35fb:26cc:838e:c70d? ([2600:380:7066:ce6e:35fb:26cc:838e:c70d]) by smtp.gmail.com with ESMTPSA id q10sm24920124pfs.75.2020.08.18.11.56.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 18 Aug 2020 11:56:16 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Eric Voskuil Mime-Version: 1.0 (1.0) Date: Tue, 18 Aug 2020 11:56:13 -0700 Message-Id: References: <995919b9-0845-3a70-4b24-9b2c1ca5fd3d@mattcorallo.com> In-Reply-To: <995919b9-0845-3a70-4b24-9b2c1ca5fd3d@mattcorallo.com> To: Matt Corallo X-Mailer: iPhone Mail (17G68) X-Mailman-Approved-At: Tue, 18 Aug 2020 19:03:38 +0000 Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Aug 2020 18:56:19 -0000 > On Aug 18, 2020, at 11:25, Matt Corallo wrote: >=20 > On 8/18/20 2:11 PM, Eric Voskuil wrote: > - snip - >>> Still, I think we're talking pedantics here, and not in a useful way. >> Not to be pedantic, but I don=E2=80=99t know what that means. >=20 > It means that part of the discussion is not useful, and not worth botherin= g to go back and figure out what was shipped before the version increase and= what wasn't, lets talk about what makes sense for the future :). When the discussion centers on backward compatibility, and there is confusio= n over what that actually implies, this is a central question. You snipped t= he bit about what actually constitutes the existing protocol. I have impleme= nted every aspect of the protocol that is widely deployed, and I can say wit= hout question that the protocol does not require a peer to accept arbitrary m= essages. In other words, your statement on the subject was a very relevant f= actual error. Furthermore no reason has been demonstrated here to accept arb= itrary traffic. >>> Ultimately we need some kind of negotiation which is flexible in allowin= g different software to negotiate different features without a global lock-s= tep version number increase. >> I have shown below how that already works. >>> Or, to put it another way, if a feature is fully optional, why should th= ere be a version number increase for it >> For the reasons previously given. >>> - the negotiation of it is independent and a version number only increas= es confusion over which change "owns" a given version number. >> Presumably this is why we have a standards process. Any new message impli= es ownership. Deconflicting that is required, which implies it can easily be= version associated (as it has been). >=20 > I think the point is, this doesn't work today, bumping the protocol versio= n requires everyone agreeing on which features make sense, As I have shown, this is not the case. While I have given my support to simp= lifying the process, we should not proceed based on an incorrect understandi= ng of actual behavior. > and as we can see from this email thread alone, that isn't a common result= in this community. People happily ignore BIPs that make no sense, of which t= here are a lot, and they should totally be able to do that! >=20 > You can say that the current world works, but there's a reason over time w= e've shifted away from the original "shove another bit on the end of the ver= sion message, and everyone agrees on the order of those bits for new feature= negotiation." Version bumping is an extension of that, really. Actually the protocol has not done this. It has used the version to signal a= new sub-protocol, and then in some cases that sub-protocol has been made op= tional through subsequent negotiation. What is being proposed here is to sim= plify that process by collapsing the secondary negotiation into the handshak= e. In fact I argued against this secondary ad-hoc negotiation when it began. No= w we are coming around to recognizing that it=E2=80=99s a handshake issue, a= s I said at the time. I=E2=80=99m glad to see that. >>> I presume you'd support a single message that lists the set of features w= hich a node (optionally) wishes to support on the connection. This proposal i= s fully equivalent to that, instead opting to list them as individual messag= es instead of one message, which is a bit nicer in that they can be handled m= ore independently or by different subsystems including even the message hash= ing. >> This presumes an implementation. As part of the handshake, collection of a= n arbitrary set of messages is a significant and unnecessary complication *o= f the protocol*. Extension of the verack is not. It is the simplest change p= ossible to implement the desired behavior. Each peer simply supplies the mat= rix of sub-protocols it supports and only those supported by both are allowe= d. There is no reason for the protocol to split that matrix up into multiple= messages, requiring termination. Independent messages exist because of timi= ng or ordering requirements. Implementing dependent messages as if they were= independent is wasteful and complicating. >=20 > Some things may need further negotiation. eg compact blocks sends multiple= redundant messages with different versions and then deduces the correct ver= sion based on the message ordering and version set supported. Doing this via= verack locks you into a very specific possible negotiation protocols. This is a moot point. Whether a list of supported optional sub-protocols is l= isted in one or multiple messages in the handshake would not change this. > You could extend it further and suggest a verack K-V list which allows for= more flexible negotiation, but I'm not sure that it isn't more complicated t= han just shoving more messages on the wire. There is no difference between a constrained set of key-value pairs and a di= stinct set of options, so there is no additional complexity here. >> I=E2=80=99m well aware of the inefficiency produced by version linearity i= n the face of optional sub-protocols. The protocol must negotiate to the ver= sion where it can then negotiate support, which has been done. I support cre= ating a simpler system, eliminating these extra messages. The existing numer= ic version can be reserved exclusively for =E2=80=9Cmust=E2=80=9D implement,= and can be used to signal an extension to the verack. The verack can then c= arry a list of =E2=80=9Cmay=E2=80=9D or =E2=80=9Cshould=E2=80=9D sub-protoco= ls for final negotiation. >=20 > I think we agree here - the current method of protocol version bumping isn= 't scalable and something more flexible is definitely a better world. To be clear this does not increase flexibility, it reduces communication and= therefore complexity, and allows peers to lock in allowed message semantics= by the end of the handshake, as opposed to allowing them to change at any t= ime. >> The format of the matrix is arbitrary, but the requirement is to list a s= et of optional sub-protocols. This implies a namespace. This implies =E2=80=9C= ownership=E2=80=9C of names. In other words, that coordination requirement i= s not eliminated. >=20 > This is true, there is some ownership requirement, we could switch to hash= es or something of the like, but human-readable names have shown to be relat= ively non-colliding in past Bitcoin protocol changes. Hashes don=E2=80=99t prevent collisions. Someone can just use the same hash.= Bitcoin uses names (message names) and numbers (version, service, relay...)= . It=E2=80=99s a protocol, coordination is the whole point. e >> e >>> Matt >>>=20 >>>> On 8/18/20 12:54 PM, Eric Voskuil wrote: >>>> =E2=80=9CBitcoin protocol has always expected clients to ignore unknown= messages=E2=80=9D >>>> This is not true. Bitcoin has long implemented version negotiation, whi= ch is the opposite expectation. Libbitcoin=E2=80=99s p2p protocol implementa= tion immediately drops a peer that sends an invalid message according to the= negotiated version. The fact that a given client does not validate the prot= ocol does not make it an expectation that the protocol not be validated. >>>> Features can clearly be optional within an actual protocol. There have b= een post-handshake negotiations implemented for optional messages which are v= alid at the negotiated version. The protocol may be flexible while remaining= validateable. There is no reason to force a client to accept unknown messag= e traffic. >>>> A generalized versioning change can be implemented in or after the hand= shake. The latter is already done on an ad-hoc basis. The former is possible= as long as the peer=E2=80=99s version is sufficient to be aware of the beha= vior. This does not imply any need to send invalid messages. The verack itse= lf can simply be extended with a matrix of feature support. There is no reas= on to complicate negotiation with an additional message(s). >>>> FWIW, bip37 did this poorly, adding a feature field to the version mess= age, resulting in bip60. Due to this design, older protocol-validating clien= ts were broken. In this case it was message length that was presumed to not b= e validated. >>>> e >>>>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev wrote: >>>>>=20 >>>>> =EF=BB=BFThis sounds like a great idea! >>>>>=20 >>>>> Bitcoin is no longer a homogeneous network of one client - it is many,= with different features implemented in each. The Bitcoin protocol hasn't (f= ully) evolved to capture that reality. Initially the Bitcoin protocol had a s= imple numerical version field, but that is wholly impractical for any divers= e network - some clients may not wish to implement every possible new relay m= echanic, and why should they have to in order to use other new features? >>>>>=20 >>>>> Bitcoin protocol changes have, many times in recent history, been made= via new dummy "negotiation" messages, which take advantage of the fact that= the Bitcoin protocol has always expected clients to ignore unknown messages= . Given that pattern, it makes sense to have an explicit negotiation phase -= after version and before verack, just send the list of features that you su= pport to negotiate what the connection will be capable of. The exact way we d= o that doesn't matter much, and sending it as a stream of messages which eac= h indicate support for a given protocol feature perfectly captures the patte= rn that has been used in several recent network upgrades, keeping consistenc= y. >>>>>=20 >>>>> Matt >>>>>=20 >>>>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote: >>>>>> Hi, >>>>>> Back in February I posted a proposal for WTXID-based transaction rela= y[1] (now known as BIP 339), which included a proposal for feature negotiati= on to take place prior to the VERACK message being received by each side. I= n my email to this list, I had asked for feedback as to whether that proposa= l was problematic, and didn't receive any responses. >>>>>> Since then, the implementation of BIP 339 has been merged into Bitcoi= n Core, though it has not yet been released. >>>>>> In thinking about the mechanism used there, I thought it would be hel= pful to codify in a BIP the idea that Bitcoin network clients should ignore u= nknown messages received before a VERACK. A draft of my proposal is availab= le here[2]. >>>>>> I presume that software upgrading past protocol version 70016 was alr= eady planning to either implement BIP 339, or ignore the wtxidrelay message p= roposed in BIP 339 (if not, then this would create network split concerns in= the future -- so I hope that someone would speak up if this were a problem)= . When we propose future protocol upgrades that would benefit from feature n= egotiation at the time of connection, I think it would be nice to be able to= use the same method as proposed in BIP 339, without even needing to bump th= e protocol version. So having an understanding that this is the standard of= how other network clients operate would be helpful. >>>>>> If, on the other hand, this is problematic for some reason, I look fo= rward to hearing that as well, so that we can be careful about how we deploy= future p2p changes to avoid disruption. >>>>>> Thanks, >>>>>> Suhas Daftuar >>>>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-Febr= uary/017648.html >>>>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature= -negotiation/bip-p2p-feature-negotiation.mediawiki >>>>>> _______________________________________________ >>>>>> bitcoin-dev mailing list >>>>>> bitcoin-dev@lists.linuxfoundation.org >>>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >>>>> _______________________________________________ >>>>> bitcoin-dev mailing list >>>>> bitcoin-dev@lists.linuxfoundation.org >>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev