[bitcoin-dev] Refreshed BIP324

[bitcoin-dev] Refreshed BIP324

[Dhruv M]

Hi all,

We have refreshed the proposal for BIP324, a new bitcoin P2P protocol
featuring opportunistic encryption, a mild bandwidth reduction, and the
ability
to negotiate upgrades before exchanging application messages. We'd like
to invite community members to review the BIP[1] and the related Bitcoin
Core
code[2].

The proposal has a rich history[3]. The big changes since the last public
appearance[4] are:

* Elligator-swift encoding for the pubkeys in the ECDH exchange to
obtain a pseudorandom bytestream
* x-only ECDH secret derivation
* Transport versioning that allows for upgradability
* Trafic shapability using decoy packets and a shapable handshake
* Complete rewrite of the BIP text

We look forward to your review and comments.

-Dhruv, Tim and Pieter


[1] BIP Pull Request: https://github.com/bitcoin/bips/pull/1378

[2] All historical and current PRs: https://bip324.com/sections/code-review/

[3] https://bip324.com/sections/bip-review/

[4] https://gist.github.com/dhruv/5b1275751bc98f3b64bcafce7876b489

Re: [bitcoin-dev] Refreshed BIP324

[Pieter Wuille]

Hi all,

On Saturday, October 8th, 2022 at 8:59 AM, Dhruv M <dhruv@bip324.com> wrote:

> We have refreshed the proposal for BIP324, a new bitcoin P2P protocol
> featuring opportunistic encryption, a mild bandwidth reduction, and the
> ability
> to negotiate upgrades before exchanging application messages. We'd like
> to invite community members to review the BIP[1] and the related Bitcoin
> Core
> code[2].

One open question we have regarding BIP324's design is how to deal with the
coordination of assigning the message type IDs.

For context, the current BIP324 draft introduces a notion of 1-byte message
type IDs, which take the place of the 12-byte command strings (in a backward
compatible way; it's still possible to send full strings). This offers a
mild bandwidth reduction (3 bytes per message overall), especially since many
messages on the network are fairly small.

However, it obviously raises the question of how the mapping table between the
1-byte IDs and the commands they represent should be maintained:

1. The most straightforward solution is using the BIP process as-is: let BIP324
   introduce a fixed initial table, and future BIPs which introduce new
   messages can introduce new mapping entries for it. In theory, this is no
   worse than the current coordination difficulty about command strings, but
   in practice the risk of collisions due to competing proposals is of course
   significantly larger with 1-byte IDs vs. 12-byte strings.

2. An alternative approach is not using 1-byte IDs but slightly longer ones;
   for example 3-byte IDs, each consisting of a 2-byte BIP number and a 1-byte
   message index introduced by that BIP, at the cost of a smaller bandwidth
   improvement. This significantly reduces collision risks, but doesn't remove
   the coordination process concerns entirely (e.g. revisions changing what a
   BIP introduces need to be taken into account and probably still mean BIPs
   need to explicitly list which assignments they introduce).

3. Yet another possibility is not having a fixed table at all, and negotiate
   the mapping dynamically. E.g. either side could send a message at
   connection time with an explicit table of entries "when I send byte X, I
   mean command Y".

4. Lastly, the whole feature could just be dropped from BIP324 (sticking with
   command strings), and left for a follow-up (or independent) protocol
   improvement. Since arguably this is purely an application-layer concern and
   not a transport-layer one, it could even be added as an optional feature to
   the (pre-BIP324) protocol today. That would however very likely mean that
   BIP324 if adopted as-is isn't actually an (albeit small) bandwidth
   reduction compared to today, and forego a possibility to fix a fairly
   gratuitous inefficiency in the protocol from day one.

Our idea is to start out with approach (1), with a mapping table effectively
managed by the BIP process directly, but if and when collisions become a
concern (maybe due to many parallel proposals, maybe because the number of
messages just grows too big), switch to approach (3), possibly even
differentially (the sent mappings are just additions/overwrites of the
BIP-defined table mappings, rather than a full mapping).

That said, we're not all that convinced this is the best approach, and feel
this more a community/process question than a technical one, so it would be
good to see more opinions on the topic.

Cheers,

-- 
Dhruv, Pieter, Tim

Re: [bitcoin-dev] Refreshed BIP324

[Vasil Dimov]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Wed, Oct 26, 2022 at 16:39:02 +0000, Pieter Wuille via bitcoin-dev wrote:
[...]
> Our idea is to start out with approach (1)
[...]
> That said, we're not all that convinced this is the best approach, and feel
> this more a community/process question than a technical one, so it would be
> good to see more opinions on the topic.
[...]

It all makes perfect sense to me.

-- 
Vasil Dimov
gro.DSBeerF@dv
%
If 10 years from now, when you are doing something quick
and dirty, you suddenly visualize that I am looking over your
shoulders and say to yourself, "Dijkstra would not have liked this",
well that would be enough immortality for me.
                -- Edsger W. Dijkstra

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1528 bytes --]

Re: [bitcoin-dev] Refreshed BIP324

[Murch]

[-- Attachment #1.1: Type: text/plain, Size: 1303 bytes --]

Hi Pieter, hello list,

On 26.10.22 12:39, Pieter Wuille via bitcoin-dev wrote:
> 1. The most straightforward solution is using the BIP process as-is: let BIP324
>     introduce a fixed initial table, and future BIPs which introduce new
>     messages can introduce new mapping entries for it. In theory, this is no
>     worse than the current coordination difficulty about command strings, but
>     in practice the risk of collisions due to competing proposals is of course
>     significantly larger with 1-byte IDs vs. 12-byte strings.

 From what I understand we'll have about 35 message types on the network 
with the addition of BIP324. 256 possible IDs sounds like plenty room to 
grow, but perhaps we can be a bit more conservative:

We could use the first bit to signal a 2-byte message ID. That allows us 
to express 128 IDs with 1 byte, but if we need more, we get a total of 
2^15 IDs across 2 bytes.

I would not be too concerned about collisions. Firstly, message types 
would probably be announced to the mailing list as part of the 
corresponding BIP, secondly, any overlooked collision should become 
apparent at implementation time. The risk could perhaps be further 
mitigated by encouraging less prevalent message types to use a 2-byte ID.

Cheers,
Murch

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [bitcoin-dev] Refreshed BIP324

[Jonas Schnelli]

> From what I understand we'll have about 35 message types on the network with the addition of BIP324. 256 possible IDs sounds like plenty room to grow, but perhaps we can be a bit more conservative:
> 
> We could use the first bit to signal a 2-byte message ID. That allows us to express 128 IDs with 1 byte, but if we need more, we get a total of 2^15 IDs across 2 bytes.

Could make sense.

There would be an alternative to preserve more 1 byte IDs on the cost of a (much) smaller 2 byte ID space:
Reserve the short ID 0xFF as an indication for a 2 bytes short ID (additional 256 short IDs with 2 bytes).
That could be done later outside BIP324.
The 0xFF approach would lead to approx. 207 unused 1 byte short IDs (while Murchs approach would give us approx. 79 unused 1 byte short IDs).
The signal bit two byte approach would however lead to ~32k more two byte message IDs.

The main (and only?) benefit of short IDs is bandwidth.
Short ID 1-12 are reserved for string based IDs and thus, new and rarely sent message types must not always use a short ID.

Maybe the BIP should state that only frequent sent messages should reserve a short ID, though, the BIP itself assigns short IDs to all(?) message types (including low frequent messages like SENDHEADERS).

Maybe exclude message types that expected to be only sent once from assigning a short ID?

/j

Re: [bitcoin-dev] Refreshed BIP324

[Anthony Towns]

On Wed, Oct 26, 2022 at 04:39:02PM +0000, Pieter Wuille via bitcoin-dev wrote:
> However, it obviously raises the question of how the mapping table between the
> 1-byte IDs and the commands they represent should be maintained:
> 
> 1. The most straightforward solution is using the BIP process as-is: let BIP324
>    introduce a fixed initial table, and future BIPs which introduce new
>    messages can introduce new mapping entries for it. [...]

> 3. Yet another possibility is not having a fixed table at all, and negotiate
>    the mapping dynamically. E.g. either side could send a message at
>    connection time with an explicit table of entries "when I send byte X, I
>    mean command Y".

FWIW, I think these two options seem fine -- maintaining a purely local
and hardcoded internal mapping of "message string C has id Y" where Y
is capped by the number of commands you actually implement (presumably
less than 65536 total) is easy, and providing a per-peer mapping from
"byte X" to "id Y" then requires at most 512 bytes per peer, along with
up to 3kB of initial setup to tell your peer what mappings you'll use.

> Our idea is to start out with approach (1), with a mapping table effectively
> managed by the BIP process directly, but if and when collisions become a
> concern (maybe due to many parallel proposals, maybe because the number of
> messages just grows too big), switch to approach (3), possibly even
> differentially (the sent mappings are just additions/overwrites of the
> BIP-defined table mappings, rather than a full mapping).

I guess I think it would make sense to not start using a novel 1-byte
message unless you've done something to introduce that message first;
whether that's via approach (3) ("I'm going to use 0xE9 to mean pkgtxns")
or via a multibyte feature support message ("I sent sendaddrv3 as a
10-byte message, that implies 0xA3 means addrv3 from now on").

I do still think it'd be better to recommend against reserving a byte for
one-shot messages, and not do it for existing one-shot messages though.

Cheers,
aj

Re: [bitcoin-dev] Refreshed BIP324

[Pieter Wuille]

Hi,

Thanks for the comments so far. I think these are all reasonable ideas.
Comments inline below.

On Thursday, November 3rd, 2022 at 1:53 PM, Murch wrote:

> From what I understand we'll have about 35 message types on the network
> with the addition of BIP324. 256 possible IDs sounds like plenty room to
> grow, but perhaps we can be a bit more conservative:
> 
> We could use the first bit to signal a 2-byte message ID. That allows us
> to express 128 IDs with 1 byte, but if we need more, we get a total of
> 2^15 IDs across 2 bytes.

Yeah, effectively treating the first 1 or 2 bytes as a simple variable
length integer is a nice way of increasing the space at low cost.

This also doesn't need to be decided now. The initial approach could just
be avoiding allocating bytes in the 128-255 range until the need for more
space arises. If and when that is the case, the choice could be to:
* Just continue treating the first byte as the command.
* Start treating the first upper bit as a sign that another command byte
  follows.
* Switch to some form of explicit signalling (option 3 is my earlier
  mail).

On Thursday, November 3rd, 2022 at 6:26 PM, Jonas Schnelli wrote:

> There would be an alternative to preserve more 1 byte IDs on the cost
> of a (much) smaller 2 byte ID space: Reserve the short ID 0xFF as an
> indication for a 2 bytes short ID (additional 256 short IDs with 2 bytes).

I don't think this is needed, because we arguably already have that! If the
first byte is 0x01, then 1 more command byte follows in the current BIP324
draft. That mechanism is designed for alphabetic 1-character commands, but
nothing prevents it from also being used for other things (by using a
non-alphabetic byte there).

> Maybe the BIP should state that only frequent sent messages should reserve
> a short ID, though, the BIP itself assigns short IDs to all(?) message
> types (including low frequent messages like SENDHEADERS).
> 
> Maybe exclude message types that expected to be only sent once from
> assigning a short ID?

I think that makes sense. Especially in combination with the idea avoiding
bytes with the upper bit set there is a bit more pressure on the 1-byte
space. Rarely-sent or at-most-once-sent commands don't really provide much
benefit. I'd suggest scrapping from the list:
* Version messages: version, verack
* Negotiation messages: sendaddrv2, sendheaders, sendcmpct, wtxidrelay
* Rarely-sent messages: mempool

I'm not sure to what extent filteradd/filterload/filterclear/merkleblock
are still actually used; perhaps they could be removed too?

On Monday, November 7th, 2022 at 10:20 PM, Anthony Towns wrote:

> I guess I think it would make sense to not start using a novel 1-byte
> message unless you've done something to introduce that message first;
> whether that's via approach (3) ("I'm going to use 0xE9 to mean pkgtxns")
> or via a multibyte feature support message ("I sent sendaddrv3 as a
> 10-byte message, that implies 0xA3 means addrv3 from now on").

That's fair, but I don't think it matters too much for allocation purposes;
protocol designs should still not assign overlapping values, unless the
protocols are known to never be used simultaneously?

Unless... the assignment works like "whenever the sendaddrv3 is sent, the
next available byte in range 48..127 gets allocated for addrv3". That means
no explicit mapping is needed, as long as the total number of messages from
simultaneously-active extensions isn't too large.

> I do still think it'd be better to recommend against reserving a byte for
> one-shot messages, and not do it for existing one-shot messages though.

Agree.

FWIW, if anyone was wondering about how much is actually saved by having
1-byte commands vs 12-byte commands, I've gathered statistics from two nodes
(one with many inbound connections, one only outbound) for two weeks. This is
obviously very dependent on network topology and local implementation choices,
but it may still give an idea:
* Outbound-only node:
  * Around 4.5% of sent bytes are bytes 2-12 of the command.
  * Sent 979.98 MiB in total.
* Outbound and inbound node:
  * Around 1.6% of sent bytes are bytes 2-12 of the command.
  * Sent 124.14 GiB in total.

Cheers,

-- 
Pieter

Re: [bitcoin-dev] Refreshed BIP324

[Pieter Wuille]

Another idea...

On Thursday, November 10th, 2022 at 4:23 PM, Pieter Wuille via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Thursday, November 3rd, 2022 at 1:53 PM, Murch wrote:
> 
> > From what I understand we'll have about 35 message types on the network
> > with the addition of BIP324. 256 possible IDs sounds like plenty room to
> > grow, but perhaps we can be a bit more conservative:
> > 
> > We could use the first bit to signal a 2-byte message ID. That allows us
> > to express 128 IDs with 1 byte, but if we need more, we get a total of
> > 2^15 IDs across 2 bytes.
> 
> Yeah, effectively treating the first 1 or 2 bytes as a simple variable
> length integer is a nice way of increasing the space at low cost.

The above would really result in having two separate variable-length encodings:
* First byte 1-12 to signify length of alphabetic command
* Otherwise first bit to signify length of short command

I think we can just merge the two and have a single variable-length command structure that can be used for both: command encodings are 1 to 12 bytes, each byte's top bit indicating whether another byte follows (the top bit of byte 11 has no special meaning).

This means:
* Every alphabetic command of L characters becomes L bytes.
* 102 non-alphabetic 1-byte commands can be assigned.
* 15708 non-alphabetic 2-byte commands can be assigned.
* etc

So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1).

WDYT?

Cheers,

-- 
Pieter

Re: [bitcoin-dev] Refreshed BIP324

[Yuval Kogman]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

On Sat, 12 Nov 2022, 11:01 Pieter Wuille via bitcoin-dev, <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> I think we can just merge the two and have a single variable-length
> command structure that can be used for both: command encodings are 1 to 12
> bytes, each byte's top bit indicating whether another byte follows (the top
> bit of byte 11 has no special meaning).
>
...

> So this gives a uniform space which commands can be assigned from, and
> there is no strict need for thinking of the short-binary and
> long-alphabetic commands as distinct.In v2, some short ones would be
> treated as aliases for old long-alphabetic ones.


This seems a bit dubious, but since command names are ASCII strings
reversing the meaning of the top bit so that 0 signifies a following byte
would allow the alphabetic commands to be reinterpreted as non-alphabetic,
so the space no longer needs to be a disjoint union of two sub-spaces which
arguably takes the "no [...] need for thinking of [them] as distinct" logic
a little bit bit farther.

The main downsides are that this does nothing to re-assign shorter codes to
existing commands, and secondly that even the non-alphabetic code path
completely supersedes the alphabetic one, the numerical values are up to 85
or 86 bits wide, which is not a reasonable word size. Perhaps this is not a
concern since as far as I know there are no collisions in the first 9 bytes
of existing commands, and constrain the non-alphabetic representation to 9
bytes would leave the last top bit free, so the space would be isomorphic
to {0,1}^64.

[-- Attachment #2: Type: text/html, Size: 2201 bytes --]

Re: [bitcoin-dev] Refreshed BIP324

[Anthony Towns]

On Sat, Nov 12, 2022 at 03:23:16AM +0000, Pieter Wuille via bitcoin-dev wrote:
> Another idea...

> This means:
> * Every alphabetic command of L characters becomes L bytes.
> * 102 non-alphabetic 1-byte commands can be assigned.
> * 15708 non-alphabetic 2-byte commands can be assigned.

(There are 128**L possible L-byte commands, 26**L alphabetic L-byte
commands, and hence 128**L-26**L non-alphabetical L-byte commands)

> * etc
> So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1).

Isn't that optimising for the wrong thing? Aren't the goals we want:

 1) it should be easy to come up with a message identifier without
    accidently conflicting with someone else's proposal

 2) commonly used messages on the wire should have a short encoding
    in order to save bandwidth

Depending on how much the p2p protocol ossifies, which messages are
"commonly used on the wire" might be expected to change; and picking an
otherwise meaningless value from a set of 102 elements seems likely to
produce conflicts...

Doing:

  -> VERSION
  <- VERSION
  <- SHORTMSG ["BIP324", "foo.org/experiment"]
  <- VERACK
  -> SHORTMSG ["BIP324"]
  -> VERACK
  -> SENDSHORTMSG "BIP324" [(13, "ADDRV3")]
  <- SENDSHORTMSG "BIP324"
  -> 34 (SENDHEADERS)
  -> 25 (GETHEADERS)
  ...

where "SHORTMSG" lets you specify an array of tables you support (such as
BIP324's), and, once you know both sides supports a table, you can send
`SENDSHORTMSG` to choose the table you'll use to abbreviate messages
you send, as well as any modifications to that table (like replacing
ADDR with ADDRV3).

Then when writing BIPs, you just choose a meaningful string ("ADDRV3"),
and when implementing you send a one-time "SENDSHORTMSG" message to
optimise the messages you'll send most often... As time goes on and the
most common messages change, issue a new BIP with a new table so that
your one-time SHORTIDs message becomes shorter too, at least when you
connect to peers that also know about the new bip..

Could potentially support that without bip324, by taking over the one
byte message space, presuming you don't have any one-character messages
you actually want to send?

Could do this /as well as/ the encoding above, I guess; then you would
have bips specify alphabetic message ids, and use SENDSHORTMSG to
dynamically (and sequentially) override/allocate non-alphabetic ids?
That would presumably limit the number of non-alphabetic ids to how
many you could specify in a single SENDSHORTIDs message, which I think
would be up to something like 749k different 1/2/3/4/5/6-byte alphabetic
message ids (encoded as 1/2/3-byte non-alphabetic messages). Probably
some more specific limit would be better though.

Cheers,
aj

Re: [bitcoin-dev] Refreshed BIP324

[Pieter Wuille]

------- Original Message -------
On Friday, November 18th, 2022 at 3:24 AM, Anthony Towns <aj@erisian.com.au> wrote:

> > * etc
> > So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1).
> 
> Isn't that optimising for the wrong thing? Aren't the goals we want:
> 
> 1) it should be easy to come up with a message identifier without
> accidently conflicting with someone else's proposal
> 
> 2) commonly used messages on the wire should have a short encoding
> in order to save bandwidth
> 
> Depending on how much the p2p protocol ossifies, which messages are
> "commonly used on the wire" might be expected to change; and picking an
> otherwise meaningless value from a set of 102 elements seems likely to
> produce conflicts...

Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now.

I just thought it would be interesting to have a uniform encoding without explicit distinction between "short commands" and "long commands" at that layer.

But maybe none of this is worth it, as it's perhaps more complexity than the alternative, and the alternative already has a working implementation and written-up specification.

Cheers,

-- 
Pieter

Re: [bitcoin-dev] Refreshed BIP324

[Anthony Towns]

On Thu, Jan 05, 2023 at 10:06:29PM +0000, Pieter Wuille via bitcoin-dev wrote:
> > > So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1).
> Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now.
> 
> I just thought it would be interesting to have a uniform encoding without explicit distinction between "short commands" and "long commands" at that layer.
> But maybe none of this is worth it, as it's perhaps more complexity than the alternative, and the alternative already has a working implementation and written-up specification.

Heh, I was just looking at this yesterday, but failing to quite reach
a conclusion.

One thing I hadn't realised about this was that it's not actually
a restriction compared to what we currently allow with p2p v1:
CMessageHeader::IsCommandValid() already rejects commands that use
characters outside of 0x20 to 0x7E, so the high bit is already available
for signalling when we reach the last byte.

The current implementation for 324 does the aliasing
as part of V2TransportDeserializer::GetMessage and
V2TransportSerializer::prepareForTransport. That makes a lot of sense,
but particularly if we were to negotiate short commands sometime around
VERSION or VERACK, it might make more sense for the aliasing to move up
to the protocol layer rather than have it close to the  wire layer. In
that case having a uniform encoding means we could just keep using
CSerializedNetMsg whether we're sending a short command or a multibyte
ascii command -- without a uniform encoding, if we wanted to move short
commands up a layer, I think we'd need to change CSerializedNetMsg to
have m_type be a `std::variant<uint8_t,std::string>` instead of just a
string, or something similar.

I think I'm leaning towards "it doesn't matter either way" though:

 * if we can negotiate short commands on a per-peer basis, then once
   negotiation's finished we'll only be using short commands so saving a
   byte on long commands doesn't matter much

 * if we've only got around 30 or 40 commands we understand anyway
   (even counting one-time-only negotiation stuff), then it doesn't
   matter if we can do 102, 126 or 242 short commands since those are
   all more than we need

 * whether we'd have to tweak an internal struct if we want to change
   the way our code is structured shouldn't really be much of an influence
   on protocol design...

Cheers,
aj

Re: [bitcoin-dev] Refreshed BIP324

[Anthony Towns]

On Fri, Jan 06, 2023 at 09:12:50AM +1000, Anthony Towns via bitcoin-dev wrote:
> On Thu, Jan 05, 2023 at 10:06:29PM +0000, Pieter Wuille via bitcoin-dev wrote:
> > Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now.

> The current implementation for 324 does the aliasing
> as part of V2TransportDeserializer::GetMessage and
> V2TransportSerializer::prepareForTransport. That makes a lot of sense,
> [...]

So I think you can make this setup work with a negotiated assignment of
shortids, perhaps starting off something like:

https://github.com/ajtowns/bitcoin/commit/6b8edd754bdcb582e293e4f5d0b41297711bdbb7

That has a 242 element array per peer giving the mappings (which
is just ~250 bytes per peer) for deserialization, which seems
workable. [0]

It also has a single global map for serialization, so we'll always shorten
CFILTER to shortid 39 for every peer that supports shortids, even, eg, for
a peer who's told us they'll send CFILTER as shortid 99 and that we should
interpret shortid 39 from them as NEWFEATUREX. That has three advantages:

 * each peer can choose a mapping that minimises their own outbound
   traffic, even potentially for asymmetric connections, and don't need
   to coordinate with the other peer to decide a common optimal mapping
   that they both use across their connection

 * you don't have to have different serialization tables per-peer,
   reducing memory usage / implementation complexity

 * you can leave V2TransportSerializer as a `const` object, and not have
   to introduce additional locking logic to be able to update its
   state...

I'm not seeing a good way to introduce shortids for future one-shot
negotiation messages though (like VERSION, VERACK, SENDADDRV2,
WTXIDRELAY, SENDTXRCNCL):

 * if you explicitly announce the mapping first, you're just wasting
   bytes ("99=FOOBAR; 99 baz quux" vs just "FOOBAR baz quux")
 * if you negotiate the tables you support between VERSION/VERACK and
   then choose a mutually supported table after VERACK, that's too late
   for pre-VERACK negotation messages
 * announcing the tables you support as part of the VERSION message
   would work, but seems a bit klunky

Also, if you did want to shift to a new table, you'd probably want to
always support sending/receiving {37, 44, 46, 47, 36} messages?

I guess I still kind-of think it'd make more sense to just reserve
shortids for post-VERACK messages that are going to be sent more
than once per connection... At that point, even if you don't have any
table in common with your peer, just following VERACK with an immediate
announcement of each shortid you want to use and its meaning would still
make reasonable sense.

If we included the ability to define your own shortids concurrently
with bip324 rollout, then I think nodes could always have a static set
of shortids they use for all their peers for outbound messages, which,
as above, seems like it would make for simpler implementations.

ie, you might send:

   VERSION
   SHORTIDTBLS ["","awesomeshortids"]
   WTXIDRELAY
   SENDADDRV2
   SENDPACKAGES 1
   VERACK
   SHORTID "" [(52,"getpkgtxns"), (53, "pkgtxns"), (54, "ancpkginfo")] 

...but you'd do all that long form, and only switch to shortids for
messages after you've declared exactly what your shortids are going to
be.

(where "" is the table name for bip324's table, and "awesomeshortids"
is an updated table that includes the package relay commands already,
perhaps)

Cheers,
aj

[0] m_deserializer is used from the SocketHandler thread in
    CNode::ReceiveMsgBytes(), but the p2p protocol is managed from the
    MessageHandler thread; with multiple messages potentially deserialized
    into vRecvMsg() at once -- but that means that if the first message
    redefines shortid decoding, and the second message uses one of the
    redefined shortids, it will have already been decoded incorrectly.
    So that would need some futzing about still.

Re: [bitcoin-dev] Refreshed BIP324

[Dhruv M]

[-- Attachment #1: Type: text/plain, Size: 5923 bytes --]

Attempting to summarize the thread thus far:

Problem:

- 1 byte message type IDs are lacking a co-ordination mechanism when multiple in-flight BIPs are proposing new message types as the id space is reduced form 12 ASCII bytes to 1 byte.
- 1 byte IDs are scarce and should be allocated judiciously, especially given that gains on bandwidth are very much non-uniform across message types.

Solutions:

- Uniform encoding using the high-bit increases the available ID space drastically, however, there's still the issue of making sure that the most frequent message types get the shorter IDs.
- Making type IDs negotiable(editable, really) per direction per connection solves that issue at the cost of some increased complexity.

Since we don't really know the extent to which the protocol will ossify over time and that BIP324 is already quite a large change, we might want to optimize for the least additional complexity that doesn't close the doors on any of the solutions. How about this:

- BIP324 restricts type IDs to [1, 127]
- We remove 1 byte allocations for messages that are sent at most once per connection per direction
- Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern.

Best,

-Dhruv

On 1/9/23 00:11, Anthony Towns via bitcoin-dev wrote:

> On Fri, Jan 06, 2023 at 09:12:50AM +1000, Anthony Towns via bitcoin-dev wrote:
>
>> On Thu, Jan 05, 2023 at 10:06:29PM +0000, Pieter Wuille via bitcoin-dev wrote:
>>
>>> Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now.
>
>> The current implementation for 324 does the aliasing
>> as part of V2TransportDeserializer::GetMessage and
>> V2TransportSerializer::prepareForTransport. That makes a lot of sense,
>> [...]
>
> So I think you can make this setup work with a negotiated assignment of
> shortids, perhaps starting off something like:
> https://github.com/ajtowns/bitcoin/commit/6b8edd754bdcb582e293e4f5d0b41297711bdbb7
> That has a 242 element array per peer giving the mappings (which
> is just ~250 bytes per peer) for deserialization, which seems
> workable. [0]
>
> It also has a single global map for serialization, so we'll always shorten
> CFILTER to shortid 39 for every peer that supports shortids, even, eg, for
> a peer who's told us they'll send CFILTER as shortid 99 and that we should
> interpret shortid 39 from them as NEWFEATUREX. That has three advantages:
>
>  * each peer can choose a mapping that minimises their own outbound
>    traffic, even potentially for asymmetric connections, and don't need
>    to coordinate with the other peer to decide a common optimal mapping
>    that they both use across their connection
>
>  * you don't have to have different serialization tables per-peer,
>    reducing memory usage / implementation complexity
>
>  * you can leave V2TransportSerializer as a `const` object, and not have
>    to introduce additional locking logic to be able to update its
>    state...
>
> I'm not seeing a good way to introduce shortids for future one-shot
> negotiation messages though (like VERSION, VERACK, SENDADDRV2,
> WTXIDRELAY, SENDTXRCNCL):
>
>  * if you explicitly announce the mapping first, you're just wasting
>    bytes ("99=FOOBAR; 99 baz quux" vs just "FOOBAR baz quux")
>  * if you negotiate the tables you support between VERSION/VERACK and
>    then choose a mutually supported table after VERACK, that's too late
>    for pre-VERACK negotation messages
>  * announcing the tables you support as part of the VERSION message
>    would work, but seems a bit klunky
>
> Also, if you did want to shift to a new table, you'd probably want to
> always support sending/receiving {37, 44, 46, 47, 36} messages?
>
> I guess I still kind-of think it'd make more sense to just reserve
> shortids for post-VERACK messages that are going to be sent more
> than once per connection... At that point, even if you don't have any
> table in common with your peer, just following VERACK with an immediate
> announcement of each shortid you want to use and its meaning would still
> make reasonable sense.
>
> If we included the ability to define your own shortids concurrently
> with bip324 rollout, then I think nodes could always have a static set
> of shortids they use for all their peers for outbound messages, which,
> as above, seems like it would make for simpler implementations.
>
> ie, you might send:
>
>    VERSION
>    SHORTIDTBLS ["","awesomeshortids"]
>    WTXIDRELAY
>    SENDADDRV2
>    SENDPACKAGES 1
>    VERACK
>    SHORTID "" [(52,"getpkgtxns"), (53, "pkgtxns"), (54, "ancpkginfo")]
>
> ...but you'd do all that long form, and only switch to shortids for
> messages after you've declared exactly what your shortids are going to
> be.
>
> (where "" is the table name for bip324's table, and "awesomeshortids"
> is an updated table that includes the package relay commands already,
> perhaps)
>
> Cheers,
> aj
>
> [0] m_deserializer is used from the SocketHandler thread in
>     CNode::ReceiveMsgBytes(), but the p2p protocol is managed from the
>     MessageHandler thread; with multiple messages potentially deserialized
>     into vRecvMsg() at once -- but that means that if the first message
>     redefines shortid decoding, and the second message uses one of the
>     redefined shortids, it will have already been decoded incorrectly.
>     So that would need some futzing about still.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 7451 bytes --]

Re: [bitcoin-dev] Refreshed BIP324

[Anthony Towns]

On Thu, Feb 16, 2023 at 05:43:22PM +0000, Dhruv M via bitcoin-dev wrote:
> Problem:
> - 1 byte message type IDs are lacking a co-ordination mechanism when multiple in-flight BIPs are proposing new message types as the id space is reduced form 12 ASCII bytes to 1 byte.
> - 1 byte IDs are scarce and should be allocated judiciously, especially given that gains on bandwidth are very much non-uniform across message types.

ACK.

> Solutions:
> - Uniform encoding using the high-bit increases the available ID space drastically, however, there's still the issue of making sure that the most frequent message types get the shorter IDs.
> - Making type IDs negotiable(editable, really) per direction per connection solves that issue at the cost of some increased complexity.
> 
> Since we don't really know the extent to which the protocol will ossify over time and that BIP324 is already quite a large change, we might want to optimize for the least additional complexity that doesn't close the doors on any of the solutions. 

I think it's probably less complex to close *some* of the doors?

In particular, I think there's two questions that have to get answered:

 1) how do you distinguish the command from the payload for
    non short-ids -- by a length prefix, or by setting the high-bit
    of the final command byte?

 2) are short ids available/meaningful to send prior to VERACK being
    completed?

> How about this:
> - BIP324 restricts type IDs to [1, 127]

Is this for short ids (currently [13-255] per the bip) or for every byte
in a non-short-id command (for p2p v1, IsCommandValid() restricts each
byte to being in the printable ascii range, ie [32-126])?

Here's another approach:

 idea: we use short ids to minimise bandwidth, and don't care about
       bandwidth for long ids

 implementation:
       short id 0 is reserved for long commands. when received, we
       decode the first 12 bytes of the payload and treat them
       exactly the same as a v1 p2p message (trailing 0-bytes, etc)
       (if there's not 12 bytes of payload, it's just treated as an
       invalid command and dropped)

       short ids 1-255 are available for use as aliases of particular
       long commands

(That's exactly compatible with p2p v1, and also avoids the temptation
to try to choose short command names rather than descriptive ones -- the
0-padding to 12 bytes prevents you from saving any bandwidth that way;
but that's what we have short ids for anyway)

If we decide we want >255 short ids, we can figure out how to extend
them later, in a fairly open ended way I think, eg by having [128-255]
imply a 2 byte short id, so that seems fine?

> - We remove 1 byte allocations for messages that are sent at most once per connection per direction

I think this leaves 32 commands that get short ids initially:

    misc: ADDR, ADDRV2, BLOCK, FEEFILTER, GETBLOCKS, GETDATA, GETHEADERS,
          HEADERS, INV, NOTFOUND, PING, PONG, TX
    bip 35/37: FILTERADD, FILTERCLEAR, FILTERLOAD, MEMPOOL, MERKLEBLOCK
    bip 152: BLOCKTXN, CMPCTBLOCK, GETBLOCKTXN
    bip 157: CFCHECKPT, CFHEADERS, CFILTER, GETCFCHCKPT, GETCFHEADERS,
        GETCFILTERS
    bip 330: RECONCILDIFF, REQRECON, REQSKETCHEXT, SENDCMPCT, SKETCH

which drops:

    VERSION, VERACK, GETADDR, SENDADDRV2, SENDHEADERS, SENDTXRCNCL,
    WTXIDRELAY  

compared to bip 324 currently.

I think the things missing from the current list (and not currently in
use by bitcoin core) are:

    bip 61: REJECT
    bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO

> - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern.

I agree that's fine, though I expect that we'll probably want to do it
not long after bip 331 is ready for merge (or some other p2p improvement
comes along)...

Cheers,
aj

Re: [bitcoin-dev] Refreshed BIP324

[Pieter Wuille]

On Friday, February 17th, 2023 at 10:51 AM, Anthony Towns via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> I think it's probably less complex to close some of the doors?
 
> 2) are short ids available/meaningful to send prior to VERACK being
> completed?

Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for *re*-negotiating.

> Here's another approach:
> 
> idea: we use short ids to minimise bandwidth, and don't care about
> bandwidth for long ids
> 
> implementation:
> short id 0 is reserved for long commands. when received, we
> decode the first 12 bytes of the payload and treat them
> exactly the same as a v1 p2p message (trailing 0-bytes, etc)
> (if there's not 12 bytes of payload, it's just treated as an
> invalid command and dropped)
> 
> short ids 1-255 are available for use as aliases of particular
> long commands
> 
> (That's exactly compatible with p2p v1, and also avoids the temptation
> to try to choose short command names rather than descriptive ones -- the
> 0-padding to 12 bytes prevents you from saving any bandwidth that way;
> but that's what we have short ids for anyway)

I like this idea. It avoids the variable-length encoding question and related complexity entirely for things where we admittedly don't care about the bandwidth impact anyway.

It may also have another (rather weak) advantage, in that it may reduce how much information a passive observe may learn about application level features (sendheaders, sendaddrv2, ...) from the packet size sent (which would otherwise depend on command lengths), even when decoys are not in use, if no short commands are included for these messages.

> > - We remove 1 byte allocations for messages that are sent at most once per connection per direction
> 
> I think this leaves 32 commands that get short ids initially:
> 
> misc: ADDR, ADDRV2, BLOCK, FEEFILTER, GETBLOCKS, GETDATA, GETHEADERS,
> HEADERS, INV, NOTFOUND, PING, PONG, TX
> bip 35/37: FILTERADD, FILTERCLEAR, FILTERLOAD, MEMPOOL, MERKLEBLOCK
> bip 152: BLOCKTXN, CMPCTBLOCK, GETBLOCKTXN
> bip 157: CFCHECKPT, CFHEADERS, CFILTER, GETCFCHCKPT, GETCFHEADERS,
> GETCFILTERS
> bip 330: RECONCILDIFF, REQRECON, REQSKETCHEXT, SENDCMPCT, SKETCH

Sounds right.

> which drops:
> 
> VERSION, VERACK, GETADDR, SENDADDRV2, SENDHEADERS, SENDTXRCNCL,
> WTXIDRELAY

Indeed.

> compared to bip 324 currently.
> 
> I think the things missing from the current list (and not currently in
> use by bitcoin core) are:
> 
> bip 61: REJECT
> bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO

Do you feel REJECT should be included?

> > - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern.
>
> I agree that's fine, though I expect that we'll probably want to do it
> not long after bip 331 is ready for merge (or some other p2p improvement
> comes along)...

I do prefer that as well; it feels like the transport layer shouldn't be aware of the different command names that exist, but this is very much just an implementation issue.

Perhaps a possibility is having the transport layer translate short-command-number-N to the 12-byte command "\x00\x00..." + byte(N), and hand that to the application layer, which could then do the mapping?

Cheers,

-- 
Pieter

Re: [bitcoin-dev] Refreshed BIP324

[Anthony Towns]

On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote:
> > I think it's probably less complex to close some of the doors?
> > 2) are short ids available/meaningful to send prior to VERACK being
> > completed?
> Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for *re*-negotiating.

I think you still need/want two negotiation steps -- once to tell each
other what tables you know about, once to choose a mutually recognised
table and specify any additions.

> > I think the things missing from the current list (and not currently in
> > use by bitcoin core) are:
> > bip 61: REJECT
> > bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO
> Do you feel REJECT should be included?

I don't think it matters much; reject messages are both rare and include
a reason so you'd only be saving maybe 12 bytes out of 62 (~20%)
for maybe 6000 messages a day per peer that sends reject messages,
so 72kB/day/reject-peer?

> Perhaps a possibility is having the transport layer translate short-command-number-N to the 12-byte command "\x00\x00..." + byte(N), and hand that to the application layer, which could then do the mapping?

Presuming the transport layer also continues to reject commands that
have a '\x00' byte at the start or in the middle (ie !IsCommandValid()),
that seems pretty reasonable...

Cheers,
aj

Re: [bitcoin-dev] Refreshed BIP324

[Pieter Wuille]

On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns <aj@erisian.com.au> wrote:

> On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote:
> 
> > > I think it's probably less complex to close some of the doors?
> > > 2) are short ids available/meaningful to send prior to VERACK being
> > > completed?
> > > Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for re-negotiating.
> 
> I think you still need/want two negotiation steps -- once to tell each
> other what tables you know about, once to choose a mutually recognised
> table and specify any additions.

Right, I wasn't talking about how many steps/messages the negotiation takes. I just meant that if all negotiation of the mapping table happens just once (before VERACK) and that negotiation itself happens without use of short commands, then there is no need for re-negotiating short commands after they are already in use. Nothing concrete, but I can imagine that that may simplify some implementations.

Cheers,

-- 
Pieter

Re: [bitcoin-dev] Refreshed BIP324

[Anthony Towns]

On Mon, Feb 20, 2023 at 03:22:30PM +0000, Pieter Wuille via bitcoin-dev wrote:
> On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns <aj@erisian.com.au> wrote:
> > On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote:
> > > > I think it's probably less complex to close some of the doors?
> > > > 2) are short ids available/meaningful to send prior to VERACK being
> > > > completed?
> > > > Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for re-negotiating.
> > I think you still need/want two negotiation steps -- once to tell each
> > other what tables you know about, once to choose a mutually recognised
> > table and specify any additions.
> Right, I wasn't talking about how many steps/messages the negotiation takes. I just meant that if all negotiation of the mapping table happens just once (before VERACK) and that negotiation itself happens without use of short commands, then there is no need for re-negotiating short commands after they are already in use. Nothing concrete, but I can imagine that that may simplify some implementations.

Yeah; I was just thinking of the fact that currently the negotiation is:

 * send your VERSION message
 * see what their VERSION message is

 * announce a bunch of features, depending on the version (or service
   flags)
 * send the VERACK (and GETADDR and final ALERT)

 * wait for their announcements and VERACK
 * negotiation is finished; we know everything; we're ready to go

which only gets you two steps if you send the short id stuff as part of
the VERSION message. Obviously you could just add an extra phase either
just before or just after the VERACK, though.

I suppose being able to choose your own short id mapping from day 0 would
mean that every bip324 node could use a single short id mapping for all
outgoing messages, which might also make implementation marginally easier
(no need to use one table for modern nodes, but also support the original
table for old bip324 implementations)...

Cheers,
aj

Re: [bitcoin-dev] Refreshed BIP324

[Dhruv M]

The relevant changes from this discussion about short 1-byte message
type IDs are now in a PR for the bips repo:
https://github.com/bitcoin/bips/pull/1428

On 2/21/23 08:03, Anthony Towns via bitcoin-dev wrote:
> On Mon, Feb 20, 2023 at 03:22:30PM +0000, Pieter Wuille via bitcoin-dev wrote:
>> On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns <aj@erisian.com.au> wrote:
>>> On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote:
>>>>> I think it's probably less complex to close some of the doors?
>>>>> 2) are short ids available/meaningful to send prior to VERACK being
>>>>> completed?
>>>>> Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for re-negotiating.
>>> I think you still need/want two negotiation steps -- once to tell each
>>> other what tables you know about, once to choose a mutually recognised
>>> table and specify any additions.
>> Right, I wasn't talking about how many steps/messages the negotiation takes. I just meant that if all negotiation of the mapping table happens just once (before VERACK) and that negotiation itself happens without use of short commands, then there is no need for re-negotiating short commands after they are already in use. Nothing concrete, but I can imagine that that may simplify some implementations.
> Yeah; I was just thinking of the fact that currently the negotiation is:
>
>   * send your VERSION message
>   * see what their VERSION message is
>
>   * announce a bunch of features, depending on the version (or service
>     flags)
>   * send the VERACK (and GETADDR and final ALERT)
>
>   * wait for their announcements and VERACK
>   * negotiation is finished; we know everything; we're ready to go
>
> which only gets you two steps if you send the short id stuff as part of
> the VERSION message. Obviously you could just add an extra phase either
> just before or just after the VERACK, though.
>
> I suppose being able to choose your own short id mapping from day 0 would
> mean that every bip324 node could use a single short id mapping for all
> outgoing messages, which might also make implementation marginally easier
> (no need to use one table for modern nodes, but also support the original
> table for old bip324 implementations)...
>
> Cheers,
> aj
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Refreshed BIP324

[Erik Aronesty]

[-- Attachment #1: Type: text/plain, Size: 3314 bytes --]

you can always do what protocols usually do.   1 byte is fine for now, but
reserve the top bit for "this is a two byte id" (128 choices).   then when
you run out of room, set the top bit which means "this is a 2 byte id
(again with one reserved) and so-on.   ie: how protobuf stores integers.

On Tue, Feb 28, 2023 at 1:42 PM Dhruv M via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> The relevant changes from this discussion about short 1-byte message
> type IDs are now in a PR for the bips repo:
> https://github.com/bitcoin/bips/pull/1428
>
> On 2/21/23 08:03, Anthony Towns via bitcoin-dev wrote:
> > On Mon, Feb 20, 2023 at 03:22:30PM +0000, Pieter Wuille via bitcoin-dev
> wrote:
> >> On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns <
> aj@erisian.com.au> wrote:
> >>> On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via
> bitcoin-dev wrote:
> >>>>> I think it's probably less complex to close some of the doors?
> >>>>> 2) are short ids available/meaningful to send prior to VERACK being
> >>>>> completed?
> >>>>> Ah, I hadn't considered this nuance. If we don't care about them
> being available before VERACK negotiation, then it may be possible to
> introduce a way to negotiate a different short id mapping table without
> needing a mechanism for re-negotiating.
> >>> I think you still need/want two negotiation steps -- once to tell each
> >>> other what tables you know about, once to choose a mutually recognised
> >>> table and specify any additions.
> >> Right, I wasn't talking about how many steps/messages the negotiation
> takes. I just meant that if all negotiation of the mapping table happens
> just once (before VERACK) and that negotiation itself happens without use
> of short commands, then there is no need for re-negotiating short commands
> after they are already in use. Nothing concrete, but I can imagine that
> that may simplify some implementations.
> > Yeah; I was just thinking of the fact that currently the negotiation is:
> >
> >   * send your VERSION message
> >   * see what their VERSION message is
> >
> >   * announce a bunch of features, depending on the version (or service
> >     flags)
> >   * send the VERACK (and GETADDR and final ALERT)
> >
> >   * wait for their announcements and VERACK
> >   * negotiation is finished; we know everything; we're ready to go
> >
> > which only gets you two steps if you send the short id stuff as part of
> > the VERSION message. Obviously you could just add an extra phase either
> > just before or just after the VERACK, though.
> >
> > I suppose being able to choose your own short id mapping from day 0 would
> > mean that every bip324 node could use a single short id mapping for all
> > outgoing messages, which might also make implementation marginally easier
> > (no need to use one table for modern nodes, but also support the original
> > table for old bip324 implementations)...
> >
> > Cheers,
> > aj
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 4507 bytes --]

Re: [bitcoin-dev] Refreshed BIP324

[Tim Ruffing]

Hello,

We'd like to announce two recent updates to BIP324 ("Version 2 P2P
Encrypted Transport Protocol"). Some of these changes affect semantics
and some are backwards-incompatible.

While we are not aware of any implementations of BIP324 except the one
in Bitcoin Core (see https://github.com/bitcoin/bitcoin/issues/27634 ),
the purpose of the email is to inform anyone involved in other
implementation efforts. At this point, we don't expect any further
backwards-incompatible changes.

https://github.com/bitcoin/bips/pull/1496 did multiple small changes:
 * Incoming v1 connections are now detected based on first 16 bytes
   they sent (instead of 12), which improves accuracy. If the incoming
   v1 connection appears to come from a wrong network (due to non-
   matching "network magic" bytes), responders may now drop the
   connection immediately.
 * The BIP330 message types have been dropped from the short encodings
   list in the BIP. It feels like it shouldn't be BIP324's goal to
   predict future protocol improvements.

https://github.com/bitcoin/bips/pull/1498 introduced a backwards-
incompatible change:
 * The garbage authentication packet is removed by merging it with the
   version packet. This simplifies the protocol implementation by
   consolidating the states and removing the special case of "ignoring
   the ignore bit." The freedom to choose the contents of the garbage
   authentication packet has also been removed, leading to easier
   testing and implementation.

We also did some editorial improvements. The most recent revision of
the BIP324 can be found at: 

https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki

Best,
Dhruv, Tim, and Pieter

On Sat, 2022-10-08 at 12:59 +0000, Dhruv M wrote:
> Hi all,
> 
> We have refreshed the proposal for BIP324, a new bitcoin P2P protocol
> featuring opportunistic encryption, a mild bandwidth reduction, and
> the
> ability
> to negotiate upgrades before exchanging application messages. We'd
> like
> to invite community members to review the BIP[1] and the related
> Bitcoin
> Core
> code[2].
> 
> The proposal has a rich history[3]. The big changes since the last
> public
> appearance[4] are:
> 
> * Elligator-swift encoding for the pubkeys in the ECDH exchange to
> obtain a pseudorandom bytestream
> * x-only ECDH secret derivation
> * Transport versioning that allows for upgradability
> * Trafic shapability using decoy packets and a shapable handshake
> * Complete rewrite of the BIP text
> 
> We look forward to your review and comments.
> 
> -Dhruv, Tim and Pieter
> 
> 
> [1] BIP Pull Request: https://github.com/bitcoin/bips/pull/1378
> 
> [2] All historical and current PRs:
> https://bip324.com/sections/code-review/
> 
> [3] https://bip324.com/sections/bip-review/
> 
> [4] https://gist.github.com/dhruv/5b1275751bc98f3b64bcafce7876b489
> 
> 
>