From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6B78BC000E for ; Tue, 26 Oct 2021 18:17:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 5B66F60A65 for ; Tue, 26 Oct 2021 18:17:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ycBj-THDixxN for ; Tue, 26 Oct 2021 18:17:04 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-yb1-xb2e.google.com (mail-yb1-xb2e.google.com [IPv6:2607:f8b0:4864:20::b2e]) by smtp3.osuosl.org (Postfix) with ESMTPS id 1B5EB6067F for ; Tue, 26 Oct 2021 18:17:04 +0000 (UTC) Received: by mail-yb1-xb2e.google.com with SMTP id v200so37078503ybe.11 for ; Tue, 26 Oct 2021 11:17:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=n8kPHKVUU5yuLM0ZxqqfNlW3lBy93wzRINd7AtGaATY=; b=q64m1UAbVhpEhYhxd84c4Q+cll9I/XTwJlPL6ka5/RYLYMur0ioXoT9ETYI8iMDkOf DJpTcMX5vNLHmyDJIam6hi8Z/+hozXqDlNqZ3ZkGpZxSp+XQGPJIM0M6ZvkAOGgFBVH6 DdkN20xrNollF49qQ1iMri2Zj2liDC//25wS7Wu+6duQ9fQtPP0L6VpGhB9+/Q1fCTjl fIq0DCCeW/CCtBL3OXT7yqloZS4f/ZfVgBgFBxe0UcPVU39T5UinAPckM79/Y/sL28Eb q28uF3ywU5DOSdz+HP65/bj7d4oy6avoeIfSawdiQtIzygBQ8xADs4DNWx+g5IiI0AQh /hZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=n8kPHKVUU5yuLM0ZxqqfNlW3lBy93wzRINd7AtGaATY=; b=JRQdk590sid524P840Hm/u7dpBnHRCpgOZ+czUu309cC6VyeIS8H9UyshB7rjg+ugf LRVz+fwC58i7I6ySv68VQtG5fzttCIoQKJA55O0hZ7s0p7VjHYLmG47q0xZT6U6d/YYY dsjpzPkaBKON9BrCKvaJbspVSSojE6iZaEqoga7iqevhmqY+uSlhIQJKemZqocEB6DzJ vXghv3HuQo3Le2a1tnSBcI4ZFA2ICU0e2LdEFZ3eFt+ofKV4590sXBBR4/E8eFvTvl6y VcKp4hHxKIQ3gjuJPU2EYUiE5g7qadR4iP5mElWFr9fGL0aarV6pr1wKox8m60fG+e4/ 0OhQ== X-Gm-Message-State: AOAM531hMBwV2OKrIOdOfzf3q8/4+EdvEeISLfueX2qlLNVFcUvuU2dp oaDW/GbymoMSxLFaKMJQ7Y5aO6RJ458iq3Ca190BS/z2weJbSw== X-Google-Smtp-Source: ABdhPJy9jabJjI/jax7rRCzrURmuZ/+7jaZXvr5pRD10XnJ+Oa63bdTJ0CSxtNFt+FBYG2rRbNS2ih4Sci5oXZLwg0I= X-Received: by 2002:a25:b946:: with SMTP id s6mr27519772ybm.368.1635272222908; Tue, 26 Oct 2021 11:17:02 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Gloria Zhao Date: Tue, 26 Oct 2021 19:16:51 +0100 Message-ID: To: lisa neigut , Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="000000000000f5256705cf457bd0" X-Mailman-Approved-At: Tue, 26 Oct 2021 18:31:03 +0000 Subject: Re: [bitcoin-dev] death to the mempool, long live the mempool X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Oct 2021 18:17:06 -0000 --000000000000f5256705cf457bd0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Lisa, Some background for people who are not familiar with mempool: The mempool is a cache of unconfirmed transactions, designed in a way to help miners efficiently pick the highest feerate packages to include in new blocks. It stores more than a block's worth of transactions because transaction volume fluctuates and any rational miner would try to maximize the fees in their blocks; in a reorg, we also don't want to completely forget what transactions were in the now-stale tip. In Bitcoin Core, full nodes keep a mempool by default. The additional requirements for keeping a mempool are minimal (300MB storage, can be configured to be lower) because anyone, anywhere in the world, should be able to run a node and broadcast a Bitcoin payment without special connectivity to some specific set of people or expensive/inaccessible hardware. Perhaps connecting directly to miners can be a solution for some people, but I don't think it's healthy for the network. Some benefits of keeping a mempool as a non-mining node include: - Fee estimation based on your node's knowledge of unconfirmed transactions + historical data. - Dramatically increased block validation (and thus propagation) speed, since you cache signature and script results of transactions before they are confirmed. - Reduced block relay bandwidth usage (Bitcoin Core nodes use BIP152 compact block relay), as you don't need to re-request the block transactions you already have in your mempool. - Wallet ability to send/receive transactions that spend unconfirmed output= s. > I had the realization that the mempool is obsolete and should be eliminat= ed. I assume you mean that the mempool should still exist but be turned off for non-mining nodes. A block template producer needs to keep unconfirmed transactions somewhere. On Bitcoin Core today, you can use the -blocksonly config option to ignore incoming transactions (effectively switching off your mempool), but there are strong reasons not to do this: - It is trivial for your peers to detect that all transactions broadcasted by your node =3D from your wallet. Linking your node to your transactions is a very bad privacy leak. - You must query someone else for what feerate to put on your transaction. - You can't use BIP152 compact block relay, so your network bandwidth usage spikes at every block. You also can't cache validation results, so your block validation speed slows down. > Removing the mempool would greatly reduce the bandwidth requirement for r= unning a node... If you're having problems with your individual node's bandwidth usage, you can also decrease the number of connections you make or turn off incoming connections. There are efforts to reduce transaction relay bandwidth usage network-wide [1]. > Removing the mempool would... keep intentionality of transactions private= until confirmed/irrevocable... I'm confused - what is the purpose of keeping a transaction private until it confirms? Don't miners still see your transaction? A confirmed transaction is not irrevocable; reorgs happen. > Removing the mempool would... naturally resolve all current issues inhere= nt in package relay and rbf rules. Removing the mempool does not help with this. How does a miner decide whether a conflicting transaction is an economically-advantageous replacement or a DoS attack? How do you submit your CPFP if the parent is below the mempool minimum feerate? Do they already have a different relay/mempool implementation handling all of these problems but don't aren't sharing it with the rest of the community? > Initial feerate estimation would need to be based on published blocks, no= t pending transactions (as this information would no longer be available), = or from direct interactions with block producers. There are many reasons why using only published blocks for fee estimates is a flawed design, including: - The miner of a block can artificially inflate the feerate of the transactions in their mempool simply by including a few of their own transactions that pay extremely high feerates. This costs them nothing, as they collect the fees. - A miner constructs a block based on the transactions in their mempool. Your transaction's feerate may have been enough to be included 2 blocks ago or a week ago, but it will be compared to the other unconfirmed transactions available to the miner now. They can tell you what's in their mempool or what the next-block feerate is, but you would be a fool to believe them. See also [2],[3]. > Provided the number of block template producing actors remains beneath, s= ay 1000, it=E2=80=99d be quite feasible to publish a list of tor endpoints = that nodes can independently + directly submit their transactions to. In f= act, merely allowing users to select their own list of endpoints to use alt= ernatively to the mempool would be a low effort starting point for the even= tual replacement. As a thought experiment, let's imagine we have some public registry of mining nodes' tor endpoints and we use it for this secondary direct-to-miner transaction relay network. If the registry is maintained (by who?) and accurate (based on whose word?), it is a point of failure for transaction censorship and deanonymization, as well as an additional barrier to becoming a miner, encouraging centralization. The other possibility is that the registry is not accurate. In fact, unless the registry requires miners to identify themselves (which others on this thread have already pointed out is ill-advised), this should be treated similarly to regular addr gossip. We would never automatically trust that the entity behind the endpoint provides the service it advertises, is an honest node that won't simply blackhole our transaction, or even belongs to a Bitcoin node at all. Best, Gloria [1]: https://arxiv.org/pdf/1905.10518.pdf [2]: https://bitcointechtalk.com/an-introduction-to-bitcoin-core-fee-estima= tion-27920880ad0 [3]: https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41 On Tue, Oct 26, 2021 at 8:38 AM lisa neigut via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Hi all, > > In a recent conversation with @glozow, I had the realization that the > mempool is obsolete and should be eliminated. > > Instead, users should submit their transactions directly to mining pools, > preferably over an anonymous communication network such as tor. This can > easily be achieved by mining pools running a tor onion node for this > express purpose (or via a lightning network extension etc) > > Mempools make sense in a world where mining is done by a large number of > participating nodes, eg where the block template is constructed by a > majority of the participants on the network. In this case, it is necessar= y > to socialize pending transaction data to all participants, as you don=E2= =80=99t > know which participant will be constructing the winning block template. > > In reality however, mempool relay is unnecessary where the majority of > hashpower and thus block template creation is concentrated in a > semi-restricted set. > > Removing the mempool would greatly reduce the bandwidth requirement for > running a node, keep intentionality of transactions private until > confirmed/irrevocable, and naturally resolve all current issues inherent = in > package relay and rbf rules. It also resolves the recent minimum relay > questions, as relay is no longer a concern for unmined transactions. > > Provided the number of block template producing actors remains beneath, > say 1000, it=E2=80=99d be quite feasible to publish a list of tor endpoin= ts that > nodes can independently + directly submit their transactions to. In fact= , > merely allowing users to select their own list of endpoints to use > alternatively to the mempool would be a low effort starting point for the > eventual replacement. > > On the other hand, removing the mempool would greatly complicate solo > mining and would also make BetterHash proposals, which move the block > template construction away from a centralized mining pool back to the > individual miner, much more difficult. It also makes explicit the target > for DoS attacks. > > A direct communication channel between block template construction venues > and transaction proposers also provides a venue for direct feedback wrt > acceptable feerates at the time, which both makes transaction confirmatio= n > timelines less variable as well as provides block producers a mechanism f= or > (independently) enforcing their own minimum security budget. In other > words, expressing a minimum acceptable feerate for continued operation. > > Initial feerate estimation would need to be based on published blocks, no= t > pending transactions (as this information would no longer be available), = or > from direct interactions with block producers. > > > ~niftynei > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --000000000000f5256705cf457bd0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 
Hi Lisa,

Some background for people who are no=
t familiar with mempool:

The mempool is a cache of unconfirmed transactions, designed in a way to he=
lp miners efficiently pick the highest feerate packages to include in new b=
locks. It stores more than a block's worth of transactions because tran=
saction volume fluctuates and any rational miner would try to maximize the =
fees in their blocks; in a reorg, we also don't want to completely forg=
et what transactions were in the now-stale tip.

In Bitcoin Core, full nodes keep a mempool by default. The additional requi=
rements for keeping a mempool are minimal (300MB storage, can be configured=
 to be lower) because anyone, anywhere in the world, should be able to run =
a node and broadcast a Bitcoin payment without special connectivity to some=
 specific set of people or expensive/inaccessible hardware. Perhaps connect=
ing directly to miners can be a solution for some people, but I don't t=
hink it's healthy for the network.

Some benefits of keeping a mempool as a non-mining node include:
- Fee estimation based on your node's knowledge of unconfirmed transact=
ions + historical data.
- Dramatically increased block validation (and thus propagation) speed, sin=
ce you cache signature and script results of transactions before they are c=
onfirmed.
- Reduced block relay bandwidth usage (Bitcoin Core nodes use BIP152 compac=
t block relay), as you don't need to re-request the block transactions =
you already have in your mempool.
- Wallet ability to send/receive transactions that spend unconfirmed output=
s.

> I had the realization that the mempool is obsolete and should be elimi=
nated.

I assume you mean that the mempool should still exist but be turned off for=
 non-mining nodes. A block template producer needs to keep unconfirmed tran=
sactions somewhere.
On Bitcoin Core today, you can use the -blocksonly config option to ignore =
incoming transactions (effectively switching off your mempool), but there a=
re strong reasons not to do this:
- It is trivial for your peers to detect that all transactions broadcasted =
by your node =3D from your wallet. Linking your node to your transactions i=
s a very bad privacy leak.
- You must query someone else for what feerate to put on your transaction.
- You can't use BIP152 compact block relay, so your network bandwidth u=
sage spikes at every block. You also can't cache validation results, so=
 your block validation speed slows down.

> Removing the mempool would greatly reduce the bandwidth requirement fo=
r running a node...

If you're having problems with your individual node's bandwidth usa=
ge, you can also decrease the number of connections you make or turn off in=
coming connections. There are efforts to reduce transaction relay bandwidth=
 usage network-wide [1].

> Removing the mempool would... keep intentionality of transactions priv=
ate until confirmed/irrevocable...

I'm confused - what is the purpose of keeping a transaction private unt=
il it confirms? Don't miners still see your transaction? A confirmed tr=
ansaction is not irrevocable; reorgs happen.

> Removing the mempool would... naturally resolve all current issues inh=
erent in package relay and rbf rules.

Removing the mempool does not help with this. How does a miner decide wheth=
er a conflicting transaction is an economically-advantageous replacement or=
 a DoS attack? How do you submit your CPFP if the parent is below the mempo=
ol minimum feerate? Do they already have a different relay/mempool implemen=
tation handling all of these problems but don't aren't sharing it w=
ith the rest of the community?

> Initial feerate estimation would need to be based on published blocks,=
 not pending transactions (as this information would no longer be available=
), or from direct interactions with block producers.

There are many reasons why using only published blocks for fee estimates is=
 a flawed design, including:

- The miner of a block can artificially inflate the feerate of the transact=
ions in their mempool simply by including a few of their own transactions t=
hat pay extremely high feerates. This costs them nothing, as they collect t=
he fees.
- A miner constructs a block based on the transactions in their mempool. Yo=
ur transaction's feerate may have been enough to be included 2 blocks a=
go or a week ago, but it will be compared to the other unconfirmed transact=
ions available to the miner now. They can tell you what's in their memp=
ool or what the next-block feerate is, but you would be a fool to believe t=
hem.

See also [2],[3].

> Provided the number of block template producing actors remains beneath=
, say 1000, it=E2=80=99d be quite feasible to publish a list of tor endpoin=
ts that nodes can independently  + directly submit their transactions to. I=
n fact, merely allowing users to select their own list of endpoints to use =
alternatively to the mempool would be a low effort starting point for the e=
ventual replacement.

As a thought experiment, let's imagine we have some public registry of =
mining nodes' tor endpoints and we use it for this secondary direct-to-=
miner transaction relay network. If the registry is maintained (by who?) an=
d accurate (based on whose word?), it is a point of failure for transaction=
 censorship and deanonymization, as well as an additional barrier to becomi=
ng a miner, encouraging centralization.
The other possibility is that the registry is not accurate. In fact, unless=
 the registry requires miners to identify themselves (which others on this =
thread have already pointed out is ill-advised), this should be treated sim=
ilarly to regular addr gossip. We would never automatically trust that the =
entity behind the endpoint provides the service it advertises, is an honest=
 node that won't simply blackhole our transaction, or even belongs to a=
 Bitcoin node at all.

Best,
Gloria

[1]: https://arxiv.org/pdf=
/1905.10518.pdf
[2]: https://bitcointechtalk.com/an-introduction-to=
-bitcoin-core-fee-estimation-27920880ad0
[3]: https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41

On Tue, Oct 26, 2021 at 8:38 AM lisa neigut via bitcoin-dev <bitcoin-dev@lists.linuxfo= undation.org> wrote:
Hi all,

In a recent conversation with @glozow, I had the realiza= tion that the mempool is obsolete and should be eliminated.

Instead, users should submit their tr= ansactions directly to mining pools, preferably over an anonymous communica= tion network such as tor. This can easily be achieved by mining pools runni= ng a tor onion node for this express purpose (or via a lightning network ex= tension etc)

Mempools ma= ke sense in a world where mining is done by a large number of participating= nodes, eg where the block template is constructed by a majority of the par= ticipants on the network. In this case, it is necessary to socialize pendin= g transaction data to all participants, as you don=E2=80=99t know which par= ticipant will be constructing the winning block template.

In reality however, mempool relay is unne= cessary where the majority of hashpower and thus block template creation is= concentrated in a semi-restricted set.=C2=A0

Removing the mempool would greatly reduce the bandwid= th requirement for running a node, keep intentionality of transactions priv= ate until confirmed/irrevocable, and naturally resolve all current issues i= nherent in package relay and rbf rules. It also resolves the recent minimum= relay questions, as relay is no longer a concern for unmined transactions.=

Provided the number of = block template producing actors remains beneath, say 1000, it=E2=80=99d be = quite feasible to publish a list of tor endpoints that nodes can independen= tly =C2=A0+ directly submit their transactions to. In fact, merely allowing= users to select their own list of endpoints to use alternatively to the me= mpool would be a low effort starting point for the eventual replacement.

On the other hand, removin= g the mempool would greatly complicate solo mining and would also make Bett= erHash proposals, which move the block template construction away from a ce= ntralized mining pool back to the individual miner, much more difficult. It= also makes explicit the target for DoS attacks.
A direct communication channel between block templ= ate construction venues and transaction proposers also provides a venue for= direct feedback wrt acceptable feerates at the time, which both makes tran= saction confirmation timelines less variable as well as provides block prod= ucers a mechanism for (independently) enforcing their own minimum security = budget. In other words, expressing a minimum acceptable feerate for continu= ed operation.

Initial fe= erate estimation would need to be based on published blocks, not pending tr= ansactions (as this information would no longer be available), or from dire= ct interactions with block producers.


~niftynei
_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--000000000000f5256705cf457bd0--