Re: [bitcoin-dev] Improving RBF Policy

Re: [bitcoin-dev] Improving RBF Policy

[Prayank]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

Hi Bastein,

> This work will highly improve the security of any multi-party contract trying to build on top of bitcoin
Do you think such multi party contracts are vulnerable by design considering they rely on policy that cannot be enforced?

> For starters, let me quickly explain why the current rules are hard to work with in the context of lightning
Using the term 'rules' can be confusing sometimes because it's just a policy and different from consensus rules. I wish we could change this in the BIP with something else.

> I'm actually paying a high fee twice instead of once (and needlessly using on-chain space, our scarcest asset, because we could have avoided that additional transaction
Not sure I understand this part because if a transaction is on-chain it can't be replaced. 

> The second biggest pain point is rule 3. It prevents me from efficiently using my capital while it's unconfirmed
> I'm curious to hear other people's thoughts on that. If it makes sense, I would propose the following very simple rules
Looks interesting however not sure about X and Y.

-- 
Prayank

A3B1 E430 2298 178F

[-- Attachment #2: Type: text/html, Size: 1828 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Bastien TEINTURIER]

[-- Attachment #1: Type: text/plain, Size: 4381 bytes --]

Hi AJ, Prayank,

> I think that's backwards: we're trying to discourage people from wasting
> the network's bandwidth, which they would do by publishing transactions
> that will never get confirmed -- if they were to eventually get confirmed
> it wouldn't be a waste of bandwith, after all. But if the original
> descendent txs were that sort of spam, then they may well not be
> submitted again if the ancestor tx reaches a fee rate that's actually
> likely to confirm.

But do you agree that descendants only matter for DoS resistance then,
not for miner incentives?

I'm asking this because I think a new set of policies should separate
policies that address the miner incentives from policies that address
the DoS issues.

The two policies I proposed address miner incentives. I think they're
insufficient to address DoS issues. But adding a 3rd policy to address
DoS issues may be a good solution?

I think that rate-limiting p2p as you suggest (and Gloria also mentioned
it) is likely a better way of fixing the DoS concerns than a descendant
rule like BIP 125 rule 5 (which as I mentioned earlier, is problematic
because the descendent set varies from one mempool to another).

I would like to add a small update to my policy suggestions. The X and Y
percentage increase should be met for both the ancestor scores AND the
transaction in isolation. Otherwise I could replace txA with txA' that
uses a new ancestor txB that has a high fee and high feerate, while txA'
has a low fee and low feerate. It's then possible for txB to confirm
without txA', and what would remain then in the mempool would be worse
than before the replacement.

> All you need is for there to be *a* path that follows the new relay rules
> and gets from your node/wallet to perhaps 10% of hashpower, which seems
> like something wallet providers could construct relatively quickly?

That's true, maybe we can be more optimistic about the timeline for
using an updated set of policies ;)

> Do you think such multi party contracts are vulnerable by design
> considering they rely on policy that cannot be enforced?

It's a good question. Even though these policies cannot be enforced, if
they are rational to apply by nodes, I think it's ok to rely on them.
Others may disagree with that, but I guess it's worth a separate thread.

> Not sure I understand this part because if a transaction is on-chain
> it can't be replaced.

Sorry, that was a bit unclear.

Suppose I have txA that I want to RBF, but I only have unconfirmed utxos
and I can't simply lower its existing outputs to reach my desired
feerate.

I must make one of my unconfirmed utxos confirm asap just to be able to
use it to RBF txA. That means I'll need to pay fees a first time just to
convert one of my unconfirmed utxos to a confirmed one. Then I'll pay
the fees to bump txA. I had to overpay fees compared to just using my
unconfirmed utxo in the first place (and manage more complexity to track
the confirmation of my unconfirmed utxo).

Thanks for your feedback!
Bastien

Le mar. 1 févr. 2022 à 03:47, Prayank <prayank@tutanota.de> a écrit :

> Hi Bastein,
>
> > This work will highly improve the security of any multi-party contract
> trying to build on top of bitcoin
>
> Do you think such multi party contracts are vulnerable by design
> considering they rely on policy that cannot be enforced?
>
> > For starters, let me quickly explain why the current rules are hard to
> work with in the context of lightning
>
> Using the term 'rules' can be confusing sometimes because it's just a
> policy and different from consensus rules. I wish we could change this in
> the BIP with something else.
>
> > I'm actually paying a high fee twice instead of once (and needlessly
> using on-chain space, our scarcest asset, because we could have avoided
> that additional transaction
>
> Not sure I understand this part because if a transaction is on-chain it
> can't be replaced.
>
> > The second biggest pain point is rule 3. It prevents me from efficiently
> using my capital while it's unconfirmed
>
> > I'm curious to hear other people's thoughts on that. If it makes sense,
> I would propose the following very simple rules
>
> Looks interesting however not sure about X and Y.
>
>
> --
> Prayank
>
> A3B1 E430 2298 178F
>

[-- Attachment #2: Type: text/html, Size: 5455 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Anthony Towns]

On Tue, Feb 01, 2022 at 10:30:12AM +0100, Bastien TEINTURIER via bitcoin-dev wrote:
> But do you agree that descendants only matter for DoS resistance then,
> not for miner incentives?

There's an edge case where you're replacing tx A with tx X, and X's fee
rate is higher than A's, but you'd be obsoleting descendent txs (B, C,
D...) and thus replacing them with unrelated txs (L, M, N...), and the
total feerate/fees of A+B+C+D... is nevertheless higher than X+L+M+N...

But I think that's probably unusual (transactions D and L are adjacent
in the mempool, that's why L is chosen for the block; but somehow
there's a big drop off in value somewhere between B/C/D and L/M/N),
and at least today, I don't think miners consider it essential to eke
out every possible sat in fee income.

(If, as per your example, you're actually replacing {A,B,C,D} with
{X,Y,Z,W} where X pays higher fees than A and the package in total pays
either the same or higher fees, that's certainly incentive compatible.
The tricky question is what happens when X arrives on its own and it
might be that no one ever sends a replacement for B,C,D)

> The two policies I proposed address miner incentives. I think they're
> insufficient to address DoS issues. But adding a 3rd policy to address
> DoS issues may be a good solution?

>>> 1. The transaction's ancestor absolute fees must be X% higher than the
>>> previous transaction's ancestor fees
>>> 2. The transaction's ancestor feerate must be Y% higher than the
>>> previous transaction's ancestor feerate

Absolute fees only matter if your backlog's feerate drops off. If you've
got 100MB of txs offering 5sat/vb, then exchanging 50kB at 5sat/vb for
1kB at 6sat/vb is still a win: your block gains 1000 sats in fees even
though your mempool loses 245,000 sats in fees.

But if your backlog's feerate does drop off, *and* that matters, then
I don't think you can ignore the impact of the descendent transactions
that you might not get a replacement for.

I think "Y% higher" rather than just "higher" is only useful for
rate-limiting, not incentive compatibility. (Though maybe it helps
stabilise a greedy algorithm in some cases?)

Cheers,
aj

Re: [bitcoin-dev] Improving RBF Policy

[Billy Tetrud]

[-- Attachment #1: Type: text/plain, Size: 22909 bytes --]

@Antoine
>  B overrides A and starts to replace package A in the network mempools
nearest to Alice. I think those peers won't have bandwidth saving from
adopting a replacement staggering strategy.

That's an interesting point, but even with that fact, the method would be
effective at limiting spam. While yes, considering just a single unit of
potential spam, only the nodes that didn't relay the spam in the first
place would save bandwidth. However, the point is not to prevent a single
unit, but to prevent thousands of units of spam. Even if in the situation
you describe Bob and Alice sent 100 replacement transaction per seconds, it
would lead to only 1 transaction sent by Bob's peers and only 2
transactions sent by Alice's peers (within a given stagger/cooldown
window). That seems pretty effective to me.

> I wonder if a "diff-only" strategy.. would be more interesting

I think a diff-only relay strategy is definitely interesting. But its
completely orthogonal. A diff only strategy isn't really a spam reduction
mechanism, but rather a relay optimization that reduces bandwidth on all
relay. Both can be done - seems like it could be argued that both should be
done.

>> For example, some nth bit of nSequence with nVersion 3 means "this
transaction won't have more than X vbytes of descendants"
>what if the pinning transaction has a parent with a junk feerate ?

I think you're right that this scheme would also be susceptible to pinning.

One thing I've identified as pretty much always suboptimal in any kind of
policy is cliffs. Hard cut offs very often cause problems. You see this in
welfare cliffs where the cliff disincentivizes people from earning more
money for example. Its almost always better to build in a smooth gradient.

Rate limiting where a node would relay replacement transaction data up to a
certain point and then stop is a cliff like this. The descendant byte limit
is a cliff like this. If such things were to be actually done, I'd
recommend building in some kind of gradient where, say, every X vbytes of
descendents requires Y additional feerate, or something to that effect.
That way you can always add more descendents as long as you're willing to
pay a higher and higher feerate for them. However, I think simply removing
the absolute feerate rule is a better solution to that kind of RBF pinning.



On Thu, Mar 17, 2022 at 4:32 AM Antoine Riard via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi Mempoololic Anonymous fellow,
>
> > 2. Staggered broadcast of replacement transactions: within some time
> > interval, maybe accept multiple replacements for the same prevout, but
> only
> > relay the original transaction.
>
> If the goal of replacement staggering is to save on bandwidth, I'm not
> sure it's going to be effective if you consider replacement done from a
> shared-utxo. E.g, Alice broadcasts a package to confirm her commitment,
> relay is staggered until T. At the same time, Bob broadcasts a package to
> confirm his version of the commitment at a slightly better feerate, relay
> is staggered until T.
>
> At T, package A gradually floods from Alice's peers and package B does the
> same from Bob's peers. When there is an intersection. B overrides A and
> starts to replace package A in the network mempools nearest to Alice. I
> think those peers won't have bandwidth saving from adopting a replacement
> staggering strategy.
>
> Or maybe that's something completely different if you have in mind ? I
> think it's worth more staggering detail to guess if it's robust against all
> the replacement propagations patterns.
>
> Though if we aim to save on replacement bandwidth I wonder if a
> "diff-only" strategy, assuming some new p2p mechanism, would be more
> interesting (as discussed in the recent "Thoughts on fee bumping thread").
>
> > A lingering concern that I have about this idea is it would then be
> > possible to impact the propagation of another person’s transaction, i.e.,
> > an attacker can censor somebody’s transaction from ever being announced
> by
> > a node if they send enough transactions to fill up the rate limit.
> > Obviously this would be expensive since they're spending a lot on fees,
> but
> > I imagine it could be profitable in some situations to spend a few
> thousand
> > dollars to prevent anyone from hearing about a transaction for a few
> hours.
> > This might be a non-issue in practice if the rate limit is generous and
> > traffic isn’t horrendous, but is this a problem?
>
> I think I share the concern too about an attacker exhausting a node
> transaction relay ressources to prevent another person's transaction to
> propagate, especially if the transaction targeted is a L2's time-sensitive
> one. In that latter context, an attacker would aim to delay the relay of a
> time-sensitive transaction (e.g a HTLC-success) to the miners, until the
> timelock expires. The malicious delay period should swallow the go-to-chain
> HTLC deadline ("the deadline for received HTLCs this node fulfilled" in
> bolt 2 parlance), in that current example 18 blocks.
>
> Let's say we allocate 10 MB of bandwidth per-block period. Once the 10 MB
> are exhausted, there is no more bandwidth allocated until the next block is
> issued. If the top mempool feerate is 1 sat/vb, such naive design would
> allow an attacker to buy all the p2p network bandwidth period for 0.1 BTC.
> If an attacker aims to jam a HTLC transaction for the 18 blocks period, the
> cost is of 1,8 BTC. If the attacker is a LN counterparty to a HTLC worth
> more than 1.8 BTC, the attack sounds economically profitable.
>
> Worst, the p2p network bandwidth is a public resource while a HTLC is a
> private, off-chain contract. An attacker could be counterparty to many
> HTLCs, where each HTLC individual value is far inferior to the global p2p
> bandwidth cost but the sum, only known to the attacker, is superior to.
> Therefore, it sounds to me that p2p network bandwidth might be attractive
> if the stealing are batched.
>
> Is the attacker scenario described credible ? Are the numbers sketched out
> realistic ?
>
> If yes, I think one design insight for eventual transaction relay rate
> limiting would be to make them "dynamic", and not naively fixed for a
> period. By making them dynamic, an attacker would have to compete with the
> effective feerate proposed by the victim transaction. E.g, if the
> HTLC-success feerate is of 10 sat/vb, an attacker would have to propose a
> stream of malicious transaction of more than 10 sat/vb during the whole
> HTLC deadline period for the transaction-relay jamming to be effective.
>
> Further, the attack might be invisible from the victim standpoint, the
> malicious flow of feerate competitive transactions can be hard to
> dissociate from an honest one. Thus, you can expect the
> HTLC transaction issuer to only slowly increase the feerate at each block,
> and those moves to be anticipated by the attacker. Even if the transaction
> issuer adopts a scorched-earth approach for the latest blocks of the
> deadline, the absolute value of the HTLC burnt in fees might still be less
> than the transaction relay bandwidth exhaustion paid by the attacker
> because the attack is batched by the attacker.
>
> I'm not sure if this reasoning is correct. Though if yes, the issue sounds
> really similar to "flood&loot" attack affecting LN previously researched on
> [0]. What worries me more with this "exhaust&loot" is that if we introduce
> bounded transaction relay rate limiting, it sounds a cheaper public
> ressource to buy than the mempool..
>
> [0] https://arxiv.org/pdf/2006.08513.pdf
>
> Anyway, I would say it's worthy to investigate more transaction relay rate
> limiting designs and especially carefully weigh the implications for L2s.
> Those ones might have to adjust their fee-bumping and transaction
> rebroadcast strategies in consequence.
>
> > Suhas and Matt [proposed][0] adding a policy rule allowing users to
> specify
> > descendant limits on their transactions. For example, some nth bit of
> > nSequence with nVersion 3 means "this transaction won't have more than X
> > vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
> > solves the pinning problem with package RBF where the attacker's package
> > contains a very large and high-fee descendant.
>
> Hey, what if the pinning transaction has a parent with a junk feerate ?
>
> Let's say you have commitment tx for a HTLC of value 500000 sats, with top
> mempool feerate of 50 sat/vbyte. The commitment tx is pinned by a malicious
> tx of size 1000 vbytes, matching top mempool feerate. This malicious tx has
> a second unconfirmed parent (in addition to the commitment) of size
> MAX_STANDARD_TX_WEIGHT offering a 1 sat/vb. I think the pinning transaction
> ancestor score would be less than 2 sat/vb and thus considered irrelevant
> for block template inclusion ? At the same time, as the pinning transaction
> is attached with a top mempool feerate, the honest user wouldn't be able to
> replace it with a better-feerate proposal ? Unless adopting a
> scorched-earth approach,  although economically I don't think this
> fee-bumping strategy is safe in case of batch-pinning.
>
> It might be fixable if we make one additional requirement "The child
> transaction subject to the user-elected descendant limit must have only one
> unconfirmed parent" (here the commitment
> transaction) ? Though I'm not even sure of the robustness of this fix. The
> commitment transaction itself could be used as a junk parent to downgrade
> the pinning transaction ancestor score. E.g, using a revoked commitment
> transaction with `max_accepted_htlcs` on both sides, pre-signed with a
> feerate of 1 sat/vb. We might restrict the maximum number of pending HTLCs
> network-wise to make the worst commitment transaction size reasonable,
> though not sure if my LN colleagues are going to like the idea..
>
> Is that reasoning correct and conform to our Ancestor Set Based algorithm
> approach ? Maybe more details are needed.
>
> > Also, coming back to the idea of "we can't just use {individual,
> ancestor}
> > feerate," I'm interested in soliciting feedback on adding a “mining
> score”
> > calculator. I've implemented one [here][2] which takes the transaction in
> > question, grabs all of the connected mempool transactions (including
> > siblings, coparents, etc., as they wouldn’t be in the ancestor nor
> > descendant sets), and builds a “block template” using our current mining
> > algorithm. The mining score of a transaction is the ancestor feerate at
> > which it is included.
>
> I don't have a strong opinion there yet, though if we make this "block
> template" construction the default one, I would be really conservative to
> avoid malicious child attachment on multi-party transactions downgrading
> the block inclusion efficiency.
>
> Antoine
>
> Le mer. 9 mars 2022 à 10:37, Gloria Zhao via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> a écrit :
>
>> Hi RBF friends,
>>
>> Posting a summary of RBF discussions at coredev (mostly on transaction
>> relay rate-limiting), user-elected descendant limit as a short term
>> solution to unblock package RBF, and mining score, all open for feedback:
>>
>> One big concept discussed was baking DoS protection into the p2p level
>> rather than policy level. TLDR: The fees are not paid to the node operator,
>> but to the miner. While we can use fees to reason about the cost of an
>> attack, if we're ultimately interested in preventing resource exhaustion,
>> maybe we want to "stop the bleeding" when it happens and bound the amount
>> of resources used in general. There were two main ideas:
>>
>> 1. Transaction relay rate limiting (i.e. the one you proposed above or
>> some variation) with a feerate-based priority queue
>> 2. Staggered broadcast of replacement transactions: within some time
>> interval, maybe accept multiple replacements for the same prevout, but only
>> relay the original transaction.
>>
>> Looking to solicit feedback on these ideas and the concept in general. Is
>> it a good idea (separate from RBF) to add rate-limiting in transaction
>> relay? And is it the right direction to think about RBF DoS protection this
>> way?
>>
>> A lingering concern that I have about this idea is it would then be
>> possible to impact the propagation of another person’s transaction, i.e.,
>> an attacker can censor somebody’s transaction from ever being announced by
>> a node if they send enough transactions to fill up the rate limit.
>> Obviously this would be expensive since they're spending a lot on fees, but
>> I imagine it could be profitable in some situations to spend a few thousand
>> dollars to prevent anyone from hearing about a transaction for a few hours.
>> This might be a non-issue in practice if the rate limit is generous and
>> traffic isn’t horrendous, but is this a problem?
>>
>> And if we don't require an increase in (i.e. addition of "new") absolute
>> fees, users are essentially allowed to “recycle” fees. In the scenario
>> where we prioritize relay based on feerate, users could potentially be
>> placed higher in the queue, ahead of other users’ transactions, multiple
>> times, without ever adding more fees to the transaction. Again, maybe this
>> isn’t a huge deal in practice if we set the parameters right, but it seems…
>> not great, in principle.
>>
>> ---------
>>
>> It's probably also a good idea to point out that there's been some
>> discussion happening on the gist containing my original post on this thread
>> (https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).
>>
>> Suhas and Matt [proposed][0] adding a policy rule allowing users to
>> specify descendant limits on their transactions. For example, some nth bit
>> of nSequence with nVersion 3 means "this transaction won't have more than X
>> vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
>> solves the pinning problem with package RBF where the attacker's package
>> contains a very large and high-fee descendant.
>>
>> We could add this policy and deploy it with package RBF/package relay so
>> that LN can use it by setting the user-elected descendant limit flag on
>> commitment transactions. (Otherwise package RBF is blocked until we find a
>> more comprehensive solution to the pinning attack).
>>
>> It's simple to [implement][1] as a mempool policy, but adds some
>> complexity for wallets that use it, since it limits their use of UTXOs from
>> transactions with this bit set.
>>
>> ---------
>>
>> Also, coming back to the idea of "we can't just use {individual,
>> ancestor} feerate," I'm interested in soliciting feedback on adding a
>> “mining score” calculator. I've implemented one [here][2] which takes the
>> transaction in question, grabs all of the connected mempool transactions
>> (including siblings, coparents, etc., as they wouldn’t be in the ancestor
>> nor descendant sets), and builds a “block template” using our current
>> mining algorithm. The mining score of a transaction is the ancestor feerate
>> at which it is included.
>>
>> This would be helpful for something like ancestor-aware funding and
>> fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
>> priority queue for transaction relay, we'd want to use something like this
>> as the priority value. And for RBF, we probably want to require that a
>> replacement have a higher mining score than the original transactions. This
>> could be computationally expensive to do all the time; it could be good to
>> cache it but that could make mempool bookkeeping more complicated. Also, if
>> we end up trying to switch to a candidate set-based algorithm for mining,
>> we'd of course need a new calculator.
>>
>> [0]:
>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4058140#gistcomment-4058140
>> [1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
>> [2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
>> [3]: https://github.com/bitcoin/bitcoin/issues/9645
>> [4]: https://github.com/bitcoin/bitcoin/issues/15553
>>
>> Best,
>> Gloria
>>
>> On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:
>>
>>> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
>>> > @aj:
>>> > > I wonder sometimes if it could be sufficient to just have a relay
>>> rate
>>> > > limit and prioritise by ancestor feerate though. Maybe something
>>> like:
>>> > > - instead of adding txs to each peers setInventoryTxToSend
>>> immediately,
>>> > >   set a mempool flag "relayed=false"
>>> > > - on a time delay, add the top N (by fee rate) "relayed=false" txs to
>>> > >   each peer's setInventoryTxToSend and mark them as "relayed=true";
>>> > >   calculate how much kB those txs were, and do this again after
>>> > >   SIZE/RATELIMIT seconds
>>>
>>> > > - don't include "relayed=false" txs when building blocks?
>>>
>>> The "?" was me not being sure that point is a good suggestion...
>>>
>>> Miners might reasonably decide to have no rate limit, and always relay,
>>> and never exclude txs -- but the question then becomes is whether they
>>> hear about the tx at all, so rate limiting behaviour could still be a
>>> potential problem for whoever made the tx.
>>>
>>> > Wow cool! I think outbound tx relay size-based rate-limiting and
>>> > prioritizing tx relay by feerate are great ideas for preventing
>>> spammers
>>> > from wasting bandwidth network-wide. I agree, this would slow the low
>>> > feerate spam down, preventing a huge network-wide bandwidth spike. And
>>> it
>>> > would allow high feerate transactions to propagate as they should,
>>> > regardless of how busy traffic is. Combined with inbound tx request
>>> > rate-limiting, might this be sufficient to prevent DoS regardless of
>>> the
>>> > fee-based replacement policies?
>>>
>>> I think you only want to do outbound rate limits, ie, how often you send
>>> INV, GETDATA and TX messages? Once you receive any of those, I think
>>> you have to immediately process / ignore it, you can't really sensibly
>>> defer it (beyond the existing queues we have that just build up while
>>> we're busy processing other things first)?
>>>
>>> > One point that I'm not 100% clear on: is it ok to prioritize the
>>> > transactions by ancestor feerate in this scheme? As I described in the
>>> > original post, this can be quite different from the actual feerate we
>>> would
>>> > consider a transaction in a block for. The transaction could have a
>>> high
>>> > feerate sibling bumping its ancestor.
>>> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB).
>>> If
>>> > we just received C, it would be incorrect to give it a priority equal
>>> to
>>> > its ancestor feerate (3sat/vB) because if we constructed a block
>>> template
>>> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
>>> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If we
>>> > also exclude C when building blocks, we're missing out on good fees.
>>>
>>> I think you're right that this would be ugly. It's something of a
>>> special case:
>>>
>>>  a) you really care about C getting into the next block; but
>>>  b) you're trusting B not being replaced by a higher fee tx that
>>>     doesn't have A as a parent; and
>>>  c) there's a lot of txs bidding the floor of the next block up to a
>>>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>>>     rate of 5sat/vB
>>>
>>> Without (a), maybe you don't care about it getting to a miner quickly.
>>> If your trust in (b) was misplaced, then your tx's effective fee rate
>>> will drop and (because of (c)), you'll lose anyway. And if the spam ends
>>> up outside of (c)'s range, either the rate limiting won't take effect
>>> (spam's too cheap) and you'll be fine, or you'll miss out on the block
>>> anyway (spam's paying more than your tx rate) and you never had any hope
>>> of making it in.
>>>
>>> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
>>> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
>>> per 10 minutes for outbound connections. This would be a weight based
>>> rate limit instead-of/in-addition-to that, I guess.
>>>
>>> As far as a non-ugly approach goes, I think you'd have to be smarter
>>> about
>>> tracking the "effective fee rate" than the ancestor fee rate manages;
>>> maybe that's something that could fall out of Murch and Clara's candidate
>>> set blockbuilding ideas [0] ?
>>>
>>> Perhaps that same work would also make it possible to come up with
>>> a better answer to "do I care that this replacement would invalidate
>>> these descendents?"
>>>
>>> [0] https://github.com/Xekyo/blockbuilding
>>>
>>> > > - keep high-feerate evicted txs around for a while in case they get
>>> > >   mined by someone else to improve compact block relay, a la the
>>> > >   orphan pool?
>>> > Replaced transactions are already added to vExtraTxnForCompact :D
>>>
>>> I guess I was thinking that it's just a 100 tx LRU cache, which might
>>> not be good enough?
>>>
>>> Maybe it would be more on point to have a rate limit apply only to
>>> replacement transactions?
>>>
>>> > For wallets, AJ's "All you need is for there to be *a* path that
>>> follows
>>> > the new relay rules and gets from your node/wallet to perhaps 10% of
>>> > hashpower" makes sense to me (which would be the former).
>>>
>>> Perhaps a corollarly of that is that it's *better* to have the mempool
>>> acceptance rule only consider economic incentives, and have the spam
>>> prevention only be about "shall I tell my peers about this?"
>>>
>>> If you don't have that split; then the anti-spam rules can prevent you
>>> from getting the tx in the mempool at all; whereas if you do have the
>>> split, then even if the bitcoind anti-spam rules are blocking you at
>>> every turn, you can still send your tx to miners by some other route,
>>> and then they can add it to their mempool directly without any hassle.
>>>
>>> Cheers,
>>> aj
>>>
>>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 25984 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Antoine Riard]

[-- Attachment #1: Type: text/plain, Size: 19666 bytes --]

Hi Mempoololic Anonymous fellow,

> 2. Staggered broadcast of replacement transactions: within some time
> interval, maybe accept multiple replacements for the same prevout, but
only
> relay the original transaction.

If the goal of replacement staggering is to save on bandwidth, I'm not sure
it's going to be effective if you consider replacement done from a
shared-utxo. E.g, Alice broadcasts a package to confirm her commitment,
relay is staggered until T. At the same time, Bob broadcasts a package to
confirm his version of the commitment at a slightly better feerate, relay
is staggered until T.

At T, package A gradually floods from Alice's peers and package B does the
same from Bob's peers. When there is an intersection. B overrides A and
starts to replace package A in the network mempools nearest to Alice. I
think those peers won't have bandwidth saving from adopting a replacement
staggering strategy.

Or maybe that's something completely different if you have in mind ? I
think it's worth more staggering detail to guess if it's robust against all
the replacement propagations patterns.

Though if we aim to save on replacement bandwidth I wonder if a "diff-only"
strategy, assuming some new p2p mechanism, would be more interesting (as
discussed in the recent "Thoughts on fee bumping thread").

> A lingering concern that I have about this idea is it would then be
> possible to impact the propagation of another person’s transaction, i.e.,
> an attacker can censor somebody’s transaction from ever being announced by
> a node if they send enough transactions to fill up the rate limit.
> Obviously this would be expensive since they're spending a lot on fees,
but
> I imagine it could be profitable in some situations to spend a few
thousand
> dollars to prevent anyone from hearing about a transaction for a few
hours.
> This might be a non-issue in practice if the rate limit is generous and
> traffic isn’t horrendous, but is this a problem?

I think I share the concern too about an attacker exhausting a node
transaction relay ressources to prevent another person's transaction to
propagate, especially if the transaction targeted is a L2's time-sensitive
one. In that latter context, an attacker would aim to delay the relay of a
time-sensitive transaction (e.g a HTLC-success) to the miners, until the
timelock expires. The malicious delay period should swallow the go-to-chain
HTLC deadline ("the deadline for received HTLCs this node fulfilled" in
bolt 2 parlance), in that current example 18 blocks.

Let's say we allocate 10 MB of bandwidth per-block period. Once the 10 MB
are exhausted, there is no more bandwidth allocated until the next block is
issued. If the top mempool feerate is 1 sat/vb, such naive design would
allow an attacker to buy all the p2p network bandwidth period for 0.1 BTC.
If an attacker aims to jam a HTLC transaction for the 18 blocks period, the
cost is of 1,8 BTC. If the attacker is a LN counterparty to a HTLC worth
more than 1.8 BTC, the attack sounds economically profitable.

Worst, the p2p network bandwidth is a public resource while a HTLC is a
private, off-chain contract. An attacker could be counterparty to many
HTLCs, where each HTLC individual value is far inferior to the global p2p
bandwidth cost but the sum, only known to the attacker, is superior to.
Therefore, it sounds to me that p2p network bandwidth might be attractive
if the stealing are batched.

Is the attacker scenario described credible ? Are the numbers sketched out
realistic ?

If yes, I think one design insight for eventual transaction relay rate
limiting would be to make them "dynamic", and not naively fixed for a
period. By making them dynamic, an attacker would have to compete with the
effective feerate proposed by the victim transaction. E.g, if the
HTLC-success feerate is of 10 sat/vb, an attacker would have to propose a
stream of malicious transaction of more than 10 sat/vb during the whole
HTLC deadline period for the transaction-relay jamming to be effective.

Further, the attack might be invisible from the victim standpoint, the
malicious flow of feerate competitive transactions can be hard to
dissociate from an honest one. Thus, you can expect the
HTLC transaction issuer to only slowly increase the feerate at each block,
and those moves to be anticipated by the attacker. Even if the transaction
issuer adopts a scorched-earth approach for the latest blocks of the
deadline, the absolute value of the HTLC burnt in fees might still be less
than the transaction relay bandwidth exhaustion paid by the attacker
because the attack is batched by the attacker.

I'm not sure if this reasoning is correct. Though if yes, the issue sounds
really similar to "flood&loot" attack affecting LN previously researched on
[0]. What worries me more with this "exhaust&loot" is that if we introduce
bounded transaction relay rate limiting, it sounds a cheaper public
ressource to buy than the mempool..

[0] https://arxiv.org/pdf/2006.08513.pdf

Anyway, I would say it's worthy to investigate more transaction relay rate
limiting designs and especially carefully weigh the implications for L2s.
Those ones might have to adjust their fee-bumping and transaction
rebroadcast strategies in consequence.

> Suhas and Matt [proposed][0] adding a policy rule allowing users to
specify
> descendant limits on their transactions. For example, some nth bit of
> nSequence with nVersion 3 means "this transaction won't have more than X
> vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
> solves the pinning problem with package RBF where the attacker's package
> contains a very large and high-fee descendant.

Hey, what if the pinning transaction has a parent with a junk feerate ?

Let's say you have commitment tx for a HTLC of value 500000 sats, with top
mempool feerate of 50 sat/vbyte. The commitment tx is pinned by a malicious
tx of size 1000 vbytes, matching top mempool feerate. This malicious tx has
a second unconfirmed parent (in addition to the commitment) of size
MAX_STANDARD_TX_WEIGHT offering a 1 sat/vb. I think the pinning transaction
ancestor score would be less than 2 sat/vb and thus considered irrelevant
for block template inclusion ? At the same time, as the pinning transaction
is attached with a top mempool feerate, the honest user wouldn't be able to
replace it with a better-feerate proposal ? Unless adopting a
scorched-earth approach,  although economically I don't think this
fee-bumping strategy is safe in case of batch-pinning.

It might be fixable if we make one additional requirement "The child
transaction subject to the user-elected descendant limit must have only one
unconfirmed parent" (here the commitment
transaction) ? Though I'm not even sure of the robustness of this fix. The
commitment transaction itself could be used as a junk parent to downgrade
the pinning transaction ancestor score. E.g, using a revoked commitment
transaction with `max_accepted_htlcs` on both sides, pre-signed with a
feerate of 1 sat/vb. We might restrict the maximum number of pending HTLCs
network-wise to make the worst commitment transaction size reasonable,
though not sure if my LN colleagues are going to like the idea..

Is that reasoning correct and conform to our Ancestor Set Based algorithm
approach ? Maybe more details are needed.

> Also, coming back to the idea of "we can't just use {individual, ancestor}
> feerate," I'm interested in soliciting feedback on adding a “mining score”
> calculator. I've implemented one [here][2] which takes the transaction in
> question, grabs all of the connected mempool transactions (including
> siblings, coparents, etc., as they wouldn’t be in the ancestor nor
> descendant sets), and builds a “block template” using our current mining
> algorithm. The mining score of a transaction is the ancestor feerate at
> which it is included.

I don't have a strong opinion there yet, though if we make this "block
template" construction the default one, I would be really conservative to
avoid malicious child attachment on multi-party transactions downgrading
the block inclusion efficiency.

Antoine

Le mer. 9 mars 2022 à 10:37, Gloria Zhao via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> a écrit :

> Hi RBF friends,
>
> Posting a summary of RBF discussions at coredev (mostly on transaction
> relay rate-limiting), user-elected descendant limit as a short term
> solution to unblock package RBF, and mining score, all open for feedback:
>
> One big concept discussed was baking DoS protection into the p2p level
> rather than policy level. TLDR: The fees are not paid to the node operator,
> but to the miner. While we can use fees to reason about the cost of an
> attack, if we're ultimately interested in preventing resource exhaustion,
> maybe we want to "stop the bleeding" when it happens and bound the amount
> of resources used in general. There were two main ideas:
>
> 1. Transaction relay rate limiting (i.e. the one you proposed above or
> some variation) with a feerate-based priority queue
> 2. Staggered broadcast of replacement transactions: within some time
> interval, maybe accept multiple replacements for the same prevout, but only
> relay the original transaction.
>
> Looking to solicit feedback on these ideas and the concept in general. Is
> it a good idea (separate from RBF) to add rate-limiting in transaction
> relay? And is it the right direction to think about RBF DoS protection this
> way?
>
> A lingering concern that I have about this idea is it would then be
> possible to impact the propagation of another person’s transaction, i.e.,
> an attacker can censor somebody’s transaction from ever being announced by
> a node if they send enough transactions to fill up the rate limit.
> Obviously this would be expensive since they're spending a lot on fees, but
> I imagine it could be profitable in some situations to spend a few thousand
> dollars to prevent anyone from hearing about a transaction for a few hours.
> This might be a non-issue in practice if the rate limit is generous and
> traffic isn’t horrendous, but is this a problem?
>
> And if we don't require an increase in (i.e. addition of "new") absolute
> fees, users are essentially allowed to “recycle” fees. In the scenario
> where we prioritize relay based on feerate, users could potentially be
> placed higher in the queue, ahead of other users’ transactions, multiple
> times, without ever adding more fees to the transaction. Again, maybe this
> isn’t a huge deal in practice if we set the parameters right, but it seems…
> not great, in principle.
>
> ---------
>
> It's probably also a good idea to point out that there's been some
> discussion happening on the gist containing my original post on this thread
> (https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).
>
> Suhas and Matt [proposed][0] adding a policy rule allowing users to
> specify descendant limits on their transactions. For example, some nth bit
> of nSequence with nVersion 3 means "this transaction won't have more than X
> vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
> solves the pinning problem with package RBF where the attacker's package
> contains a very large and high-fee descendant.
>
> We could add this policy and deploy it with package RBF/package relay so
> that LN can use it by setting the user-elected descendant limit flag on
> commitment transactions. (Otherwise package RBF is blocked until we find a
> more comprehensive solution to the pinning attack).
>
> It's simple to [implement][1] as a mempool policy, but adds some
> complexity for wallets that use it, since it limits their use of UTXOs from
> transactions with this bit set.
>
> ---------
>
> Also, coming back to the idea of "we can't just use {individual, ancestor}
> feerate," I'm interested in soliciting feedback on adding a “mining score”
> calculator. I've implemented one [here][2] which takes the transaction in
> question, grabs all of the connected mempool transactions (including
> siblings, coparents, etc., as they wouldn’t be in the ancestor nor
> descendant sets), and builds a “block template” using our current mining
> algorithm. The mining score of a transaction is the ancestor feerate at
> which it is included.
>
> This would be helpful for something like ancestor-aware funding and
> fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
> priority queue for transaction relay, we'd want to use something like this
> as the priority value. And for RBF, we probably want to require that a
> replacement have a higher mining score than the original transactions. This
> could be computationally expensive to do all the time; it could be good to
> cache it but that could make mempool bookkeeping more complicated. Also, if
> we end up trying to switch to a candidate set-based algorithm for mining,
> we'd of course need a new calculator.
>
> [0]:
> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4058140#gistcomment-4058140
> [1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
> [2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
> [3]: https://github.com/bitcoin/bitcoin/issues/9645
> [4]: https://github.com/bitcoin/bitcoin/issues/15553
>
> Best,
> Gloria
>
> On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:
>
>> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
>> > @aj:
>> > > I wonder sometimes if it could be sufficient to just have a relay rate
>> > > limit and prioritise by ancestor feerate though. Maybe something like:
>> > > - instead of adding txs to each peers setInventoryTxToSend
>> immediately,
>> > >   set a mempool flag "relayed=false"
>> > > - on a time delay, add the top N (by fee rate) "relayed=false" txs to
>> > >   each peer's setInventoryTxToSend and mark them as "relayed=true";
>> > >   calculate how much kB those txs were, and do this again after
>> > >   SIZE/RATELIMIT seconds
>>
>> > > - don't include "relayed=false" txs when building blocks?
>>
>> The "?" was me not being sure that point is a good suggestion...
>>
>> Miners might reasonably decide to have no rate limit, and always relay,
>> and never exclude txs -- but the question then becomes is whether they
>> hear about the tx at all, so rate limiting behaviour could still be a
>> potential problem for whoever made the tx.
>>
>> > Wow cool! I think outbound tx relay size-based rate-limiting and
>> > prioritizing tx relay by feerate are great ideas for preventing spammers
>> > from wasting bandwidth network-wide. I agree, this would slow the low
>> > feerate spam down, preventing a huge network-wide bandwidth spike. And
>> it
>> > would allow high feerate transactions to propagate as they should,
>> > regardless of how busy traffic is. Combined with inbound tx request
>> > rate-limiting, might this be sufficient to prevent DoS regardless of the
>> > fee-based replacement policies?
>>
>> I think you only want to do outbound rate limits, ie, how often you send
>> INV, GETDATA and TX messages? Once you receive any of those, I think
>> you have to immediately process / ignore it, you can't really sensibly
>> defer it (beyond the existing queues we have that just build up while
>> we're busy processing other things first)?
>>
>> > One point that I'm not 100% clear on: is it ok to prioritize the
>> > transactions by ancestor feerate in this scheme? As I described in the
>> > original post, this can be quite different from the actual feerate we
>> would
>> > consider a transaction in a block for. The transaction could have a high
>> > feerate sibling bumping its ancestor.
>> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB).
>> If
>> > we just received C, it would be incorrect to give it a priority equal to
>> > its ancestor feerate (3sat/vB) because if we constructed a block
>> template
>> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
>> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If we
>> > also exclude C when building blocks, we're missing out on good fees.
>>
>> I think you're right that this would be ugly. It's something of a
>> special case:
>>
>>  a) you really care about C getting into the next block; but
>>  b) you're trusting B not being replaced by a higher fee tx that
>>     doesn't have A as a parent; and
>>  c) there's a lot of txs bidding the floor of the next block up to a
>>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>>     rate of 5sat/vB
>>
>> Without (a), maybe you don't care about it getting to a miner quickly.
>> If your trust in (b) was misplaced, then your tx's effective fee rate
>> will drop and (because of (c)), you'll lose anyway. And if the spam ends
>> up outside of (c)'s range, either the rate limiting won't take effect
>> (spam's too cheap) and you'll be fine, or you'll miss out on the block
>> anyway (spam's paying more than your tx rate) and you never had any hope
>> of making it in.
>>
>> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
>> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
>> per 10 minutes for outbound connections. This would be a weight based
>> rate limit instead-of/in-addition-to that, I guess.
>>
>> As far as a non-ugly approach goes, I think you'd have to be smarter about
>> tracking the "effective fee rate" than the ancestor fee rate manages;
>> maybe that's something that could fall out of Murch and Clara's candidate
>> set blockbuilding ideas [0] ?
>>
>> Perhaps that same work would also make it possible to come up with
>> a better answer to "do I care that this replacement would invalidate
>> these descendents?"
>>
>> [0] https://github.com/Xekyo/blockbuilding
>>
>> > > - keep high-feerate evicted txs around for a while in case they get
>> > >   mined by someone else to improve compact block relay, a la the
>> > >   orphan pool?
>> > Replaced transactions are already added to vExtraTxnForCompact :D
>>
>> I guess I was thinking that it's just a 100 tx LRU cache, which might
>> not be good enough?
>>
>> Maybe it would be more on point to have a rate limit apply only to
>> replacement transactions?
>>
>> > For wallets, AJ's "All you need is for there to be *a* path that follows
>> > the new relay rules and gets from your node/wallet to perhaps 10% of
>> > hashpower" makes sense to me (which would be the former).
>>
>> Perhaps a corollarly of that is that it's *better* to have the mempool
>> acceptance rule only consider economic incentives, and have the spam
>> prevention only be about "shall I tell my peers about this?"
>>
>> If you don't have that split; then the anti-spam rules can prevent you
>> from getting the tx in the mempool at all; whereas if you do have the
>> split, then even if the bitcoind anti-spam rules are blocking you at
>> every turn, you can still send your tx to miners by some other route,
>> and then they can add it to their mempool directly without any hassle.
>>
>> Cheers,
>> aj
>>
>> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 22174 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Billy Tetrud]

[-- Attachment #1: Type: text/plain, Size: 26281 bytes --]

Hi Gloria,

It seems you're responding to what I wrote on github. Happy to respond, but
perhaps we should keep it there so the conversation is kept linearly
together?

I'm curious what you think of my thoughts on what you brought up most
recently in this thread about rate limiting / staggered window dos
protection stuff.

Cheers,
BT


On Mon, Mar 14, 2022 at 5:29 AM Gloria Zhao <gloriajzhao@gmail.com> wrote:

> Hi Billy,
>
> > We should expect miners will be using a more complex, more optimal way
> of determining what blocks they're working on [...] we should instead run
> with the assumption that miners keep all potentially relevant transactions
> in their mempools, including potentially many conflicting transctions, in
> order to create the most profitable blocks. And therefore we shouldn't put
> the constraint on normal non-mining full nodes to do that same more-complex
> mempool behavior or add any complexity for the purpose of denying
> transaction replacements.
>
> > I think a lot of the complexity in these ideas is because of the attempt
> to match relay rules with miner
> inclusion rules.
>
> I think the assumption that miners are using a completely different
> implementation of mempool and block template building is false. IIUC, most
> miners use Bitcoin Core and perhaps configure their node differently (e.g.
> larger mempool and different minfeerates), but also use `getblocktemplate`
> which means the same ancestor package-based mining algorithm.
>
> Of course, I'm not a miner, so if anybody is a miner or has seen miners'
> setups, please correct me if I'm wrong.
>
> In either case, we would want our mining algorithm to result in block
> templates that are as close as possible to perfectly incentive compatibile.
>
> Fundamentally, I believe default mempool policy (which perhaps naturally
> creates a network-wide transaction relay policy) should be as close to the
> mining code as possible. Imagine node A only keeps 1 block's worth of
> transactions, and node B keeps a (default) 300MB mempool. The contents of
> node A's mempool should be as close as possible to a block template
> generated from node B's mempool. Otherwise, node A's mempool is not very
> useful - their fee estimation is flawed and compact block relay won't do
> them much good if they need to re-request a lot of block transactions.
> Next, imagine that node B is a miner. It would be very suboptimal if the
> mining code was ancestor package-based (i.e. supports CPFP), but the
> mempool policy only cared about individual feerate, and evicted low-fee
> parents despite their high-fee children. It's easy to see why this would be
> suboptimal.
> Attempting to match mempool policy with the mining algorithm is also
> arguably the point of package relay. Our mining code uses ancestor packages
> which is good, but we only submit transactions one at a time to the
> mempool, so a transaction's high-fee children can't be considered until
> they are all already in the mempool. Package relay allows us to start
> thinking about ancestor packages immediately when evaluating transactions
> for submission to the mempool.
>
> The attempt to match policy with miner inclusion rules is deliberate and
> necessary.
>
> > I want to echo James O'Beirne's opinion on this that this may be the
> wrong path to go down (a path of more complexity without much gain). He
> said: "Special consideration for "what should be in the next block" and/or
> the caching of block templates seems like an imposing dependency, dragging
> in a bunch of state and infrastructure to a question that should be solely
> limited to mempool feerate aggregates and the feerate of the particular txn
> package a wallet is concerned with."
>
> It seems that I under-explained the purpose of building/caching block
> templates in my original post, since both you and James have the same
> misunderstanding. Since RBF's introduction, we have improved to an ancestor
> package-based mining algorithm. This supports CPFP (incentive compatible)
> and it is now common to see more complex "families" of transactions as
> users fee-bump transactions (market is working, yay). On the other hand, we
> no longer have an accurate way of determining a transaction's "mining
> score," i.e., the feerate of this transaction's ancestor package when it is
> included in a block template using our current mining algorithm.
>
> This limitation is a big blocker in proposing new fee/feerate RBF rules.
> For example, if we say "the transaction needs a better feerate," this is
> obviously flawed, since the original transactions may have very
> high-feerate children, and the replacement transaction may have low feerate
> parents. So what we really want is "the transaction needs to be more
> incentive compatible to mine based on our mining algorithm," but we have no
> way of getting that information right now.
>
> In my original post, I [described 4 heuristics to get transaction's
> "mining score"][1] using the current data cached in the mempool (e.g. max
> ancestor feerate of descendant set), as well as why they don't work. As
> such, the best way to calculate a transaction's mining score AFAICT is to
> grab all of the related transactions and build a mini "block template" with
> them. The [implementation][2] I sent last week also cuts out some of the
> fluff, so the pseudocode looks like this:
>
> // Get ALL connected entries (ancestors, descendants, siblings, cousins,
> coparents, etc.)
> vector<TxMempoolEntry> cluster = mempool.GetAllTransactionsRelatedTo(txid);
> sort(cluster, ancestorfeerate);
>
> // For deducting ancestors when they are included separately
> vector<ModifiedTxMempoolEntry> modified_entries;
>
> while (!cluster.empty() and !modified_entries.empty()) {
>     iter = BetterAncestorFeerateOf(cluster, modified_entries);
>     best_ancestor_package = GetAncestorSet(iter);
>     mining_score = Feerate(best_ancestor_package);
>     for (entry : best_ancestor_package) {
>        mining_scores[entry] = mining_score;
>        for (descendant : GetAllDescendants(entry) {
>
> modified_entries[descendant].DeductAncestorFromAccounting(entry);
>        }
>     }
> }
>
> Perhaps somebody will come up with a better way to do this, but this is my
> solution, and I hope I've now sufficiently described why I've made an
> effort to think about this. It's not because I want to make things more
> fancy or complicated, but because we currently have no way of accurately
> determining a transaction's mining score. The reason I proposed a
> persistent block template is so we can efficiently query the mining score
> of all transactions in the mempool.
>
> > However, I do think that these DOS concerns are quite overblown. I wrote
> up a comment on your rbf-improvements.md
> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#gistcomment-4093100> detailing
> my thought process on that. The summary is that as long as the fee-rate
> relay rule is maintained, any "spam" is actually paid for, either by the
> "first" transaction in the spam chain, or by the "spam" itself.
>
> The DoS concern is not overblown. I recommend you re-read the [current RBF
> policy][3]; Rule 3 and 4 *are* the feerate relay rules. Removing Rule 3
> means allowing recycled fees, so new transactions are not necessarily
> "paying" anything.
>
> [1]:
> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#mining-score-of-a-mempool-transaction
> [2]: https://github.com/glozow/bitcoin/tree/2022-02-mining-score
> [3]:
> https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
>
> Best,
> Gloria
>
> On Sat, Mar 12, 2022 at 8:18 AM Billy Tetrud <billy.tetrud@gmail.com>
> wrote:
>
>> In reading through more of the discussion, it seems the idea I presented
>> above might basically be a reformulation of t-bast's rate-limiting idea
>> presented in this comment
>> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4081349#gistcomment-4081349>.
>> Perhaps he could comment on whether that characterization is accurate.
>>
>> On Fri, Mar 11, 2022 at 10:22 AM Billy Tetrud <billy.tetrud@gmail.com>
>> wrote:
>>
>>> Hi Gloria,
>>>
>>> >  1. Transaction relay rate limiting
>>>
>>> I have a similar concern as yours, that this could prevent higher
>>> fee-rate transactions from being broadcast.
>>>
>>> > 2. Staggered broadcast of replacement transactions: within some time
>>> interval, maybe accept multiple replacements for the same prevout, but only
>>> relay the original transaction.
>>>
>>> By this do you mean basically having a batching window where, on
>>> receiving a replacement transaction, a node will wait for a period of time,
>>> potentially receiving many replacements for the same transaction (or many
>>> separate conflicting transactions), and only broadcasting the "best" one(s)
>>> at the end of that time window?
>>>
>>> Its an interesting idea, but it would produce a problem. Every hop that
>>> replacement transaction takes would be delayed by this staggered window. If
>>> the window were 3 minutes and transactions generally take 20 hops to get to
>>> the majority of miners, a "best-case average" delay might be 3.75 minutes
>>> (noting that among your 8 nodes, its quite likely one of them would have a
>>> window ending much sooner than 3 minutes). Some (maybe 3% of) nodes would
>>> experience delays of more than 20 minutes. That kind of delay isn't great.
>>>
>>> However it made me think of another idea: a transaction replacement
>>> broadcast cooldown. What if nodes kept track of the time they broadcasted
>>> the last replacement for a package and had a relay cooldown after the last
>>> replacement was broadcasted? A node receiving a replacement would relay the
>>> replacement immediately if the package its replacing was broadcasted more
>>> than X seconds ago, and otherwise it would wait until the time when that
>>> package was broadcasted at least X seconds ago to broadcast it. Any
>>> replacements it receives during that waiting period would replace as
>>> normal, meaning the unrebroadcasted replacement would never be
>>> broadcasted, and only the highest value replacement would be broadcasted at
>>> the end of the cooldown.
>>>
>>> This wouldn't prevent a higher-fee-rate transaction from being
>>> broadcasted (like rate limiting could), but would still be effective at
>>> limiting unnecessary data transmission. Another benefit is that in the
>>> non-adversarial case, replacement transactions wouldn't be subject to any
>>> delay at all (while in the staggered broadcast idea, most replacements
>>> would experience some delay). And in the adversarial case, where a
>>> malicious actor broadcasts a low-as-possible-value replacement just before
>>> yours, the worst case delay is just whatever the cooldown period is. I
>>> would imagine that maybe 1 minute would be a reasonable worst-case delay.
>>> This would limit spam for a transaction that makes it into a block to ~10x
>>> (9 to 1). I don't see much of a downside to doing this beyond just the
>>> slight additional complexity of relay rules (and considering it could save
>>> substantial additional code complexity, even that is a benefit).
>>>
>>> All a node would need to do is keep a timestamp on each transaction they
>>> receive for when it was broadcasted and check it when a replacement comes
>>> in. If now-broadcastDate < cooldown, set a timer for cooldown -
>>> (now-broadcastDate) to broadcast it. If another replacement comes in, clear
>>> that timer and repeat using the original broadcast date (since the
>>> unbroadcast transaction doesn't have a broadcast date yet).
>>>
>>> I think it might also be useful to note that eliminating "extra data"
>>> caused by careless or malicious actors (spam or whatever you want to call
>>> it) should not be the goal. It is impossible to prevent all spam. What we
>>> should be aiming for is more specific: we should attempt to design a system
>>> where spam is manageable. Eg if our goal is to ensure that a bitcoin node
>>> uses no more than 10% of the bandwidth of a "normal" user, if current
>>> non-spam traffic only requires 1% of a "normal" users's bandwidth, then the
>>> network can bear a 9 to 1 ratio of spam. When a node spins up, there is a
>>> lot more data to download and process. So we know that all full nodes can
>>> handle at least as much traffic as they handle during IBD. What's the
>>> difference between those amounts? I'm not sure, but I would guess that IBD
>>> is at least a couple times more demanding than a fully synced node. So I
>>> might suggest that as long as spam can be kept below a ratio of maybe 2 to
>>> 1, we should consider the design acceptable (and therefore more complexity
>>> unnecessary).
>>>
>>> The 1 minute broadcast cooldown I mentioned before wouldn't be quite
>>> sufficient to achieve that ratio. But a 3.33 minute cooldown would be.
>>> Whether this is "too much" is something that would have to be discussed, I
>>> suspect a worst-case adversarial 3.33 minute delay would not be "too much".
>>> Doing this could basically eliminate any risk of actual service denial via
>>> replacement transactions.
>>>
>>> However, I do think that these DOS concerns are quite overblown. I wrote
>>> up a comment on your rbf-improvements.md
>>> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#gistcomment-4093100> detailing
>>> my thought process on that. The summary is that as long as the fee-rate
>>> relay rule is maintained, any "spam" is actually paid for, either by the
>>> "first" transaction in the spam chain, or by the "spam" itself. Even
>>> without something like a minimum RBF relay delay limiting how much spam
>>> could be created, the economics of the fee-rate rule already sufficiently
>>> mitigate the issue of spam.
>>> On Wed, Mar 9, 2022 at 9:37 AM Gloria Zhao via bitcoin-dev <
>>> bitcoin-dev@lists.linuxfoundation.org> wrote:
>>>
>>>> Hi RBF friends,
>>>>
>>>> Posting a summary of RBF discussions at coredev (mostly on transaction
>>>> relay rate-limiting), user-elected descendant limit as a short term
>>>> solution to unblock package RBF, and mining score, all open for feedback:
>>>>
>>>> One big concept discussed was baking DoS protection into the p2p level
>>>> rather than policy level. TLDR: The fees are not paid to the node operator,
>>>> but to the miner. While we can use fees to reason about the cost of an
>>>> attack, if we're ultimately interested in preventing resource exhaustion,
>>>> maybe we want to "stop the bleeding" when it happens and bound the amount
>>>> of resources used in general. There were two main ideas:
>>>>
>>>> 1. Transaction relay rate limiting (i.e. the one you proposed above or
>>>> some variation) with a feerate-based priority queue
>>>> 2. Staggered broadcast of replacement transactions: within some time
>>>> interval, maybe accept multiple replacements for the same prevout, but only
>>>> relay the original transaction.
>>>>
>>>> Looking to solicit feedback on these ideas and the concept in general.
>>>> Is it a good idea (separate from RBF) to add rate-limiting in transaction
>>>> relay? And is it the right direction to think about RBF DoS protection this
>>>> way?
>>>>
>>>> A lingering concern that I have about this idea is it would then be
>>>> possible to impact the propagation of another person’s transaction, i.e.,
>>>> an attacker can censor somebody’s transaction from ever being announced by
>>>> a node if they send enough transactions to fill up the rate limit.
>>>> Obviously this would be expensive since they're spending a lot on fees, but
>>>> I imagine it could be profitable in some situations to spend a few thousand
>>>> dollars to prevent anyone from hearing about a transaction for a few hours.
>>>> This might be a non-issue in practice if the rate limit is generous and
>>>> traffic isn’t horrendous, but is this a problem?
>>>>
>>>> And if we don't require an increase in (i.e. addition of "new")
>>>> absolute fees, users are essentially allowed to “recycle” fees. In the
>>>> scenario where we prioritize relay based on feerate, users could
>>>> potentially be placed higher in the queue, ahead of other users’
>>>> transactions, multiple times, without ever adding more fees to the
>>>> transaction. Again, maybe this isn’t a huge deal in practice if we set the
>>>> parameters right, but it seems… not great, in principle.
>>>>
>>>> ---------
>>>>
>>>> It's probably also a good idea to point out that there's been some
>>>> discussion happening on the gist containing my original post on this thread
>>>> (https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).
>>>>
>>>> Suhas and Matt [proposed][0] adding a policy rule allowing users to
>>>> specify descendant limits on their transactions. For example, some nth bit
>>>> of nSequence with nVersion 3 means "this transaction won't have more than X
>>>> vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
>>>> solves the pinning problem with package RBF where the attacker's package
>>>> contains a very large and high-fee descendant.
>>>>
>>>> We could add this policy and deploy it with package RBF/package relay
>>>> so that LN can use it by setting the user-elected descendant limit flag on
>>>> commitment transactions. (Otherwise package RBF is blocked until we find a
>>>> more comprehensive solution to the pinning attack).
>>>>
>>>> It's simple to [implement][1] as a mempool policy, but adds some
>>>> complexity for wallets that use it, since it limits their use of UTXOs from
>>>> transactions with this bit set.
>>>>
>>>> ---------
>>>>
>>>> Also, coming back to the idea of "we can't just use {individual,
>>>> ancestor} feerate," I'm interested in soliciting feedback on adding a
>>>> “mining score” calculator. I've implemented one [here][2] which takes the
>>>> transaction in question, grabs all of the connected mempool transactions
>>>> (including siblings, coparents, etc., as they wouldn’t be in the ancestor
>>>> nor descendant sets), and builds a “block template” using our current
>>>> mining algorithm. The mining score of a transaction is the ancestor feerate
>>>> at which it is included.
>>>>
>>>> This would be helpful for something like ancestor-aware funding and
>>>> fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
>>>> priority queue for transaction relay, we'd want to use something like this
>>>> as the priority value. And for RBF, we probably want to require that a
>>>> replacement have a higher mining score than the original transactions. This
>>>> could be computationally expensive to do all the time; it could be good to
>>>> cache it but that could make mempool bookkeeping more complicated. Also, if
>>>> we end up trying to switch to a candidate set-based algorithm for mining,
>>>> we'd of course need a new calculator.
>>>>
>>>> [0]:
>>>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4058140#gistcomment-4058140
>>>> [1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
>>>> [2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
>>>> [3]: https://github.com/bitcoin/bitcoin/issues/9645
>>>> [4]: https://github.com/bitcoin/bitcoin/issues/15553
>>>>
>>>> Best,
>>>> Gloria
>>>>
>>>> On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:
>>>>
>>>>> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
>>>>> > @aj:
>>>>> > > I wonder sometimes if it could be sufficient to just have a relay
>>>>> rate
>>>>> > > limit and prioritise by ancestor feerate though. Maybe something
>>>>> like:
>>>>> > > - instead of adding txs to each peers setInventoryTxToSend
>>>>> immediately,
>>>>> > >   set a mempool flag "relayed=false"
>>>>> > > - on a time delay, add the top N (by fee rate) "relayed=false" txs
>>>>> to
>>>>> > >   each peer's setInventoryTxToSend and mark them as "relayed=true";
>>>>> > >   calculate how much kB those txs were, and do this again after
>>>>> > >   SIZE/RATELIMIT seconds
>>>>>
>>>>> > > - don't include "relayed=false" txs when building blocks?
>>>>>
>>>>> The "?" was me not being sure that point is a good suggestion...
>>>>>
>>>>> Miners might reasonably decide to have no rate limit, and always relay,
>>>>> and never exclude txs -- but the question then becomes is whether they
>>>>> hear about the tx at all, so rate limiting behaviour could still be a
>>>>> potential problem for whoever made the tx.
>>>>>
>>>>> > Wow cool! I think outbound tx relay size-based rate-limiting and
>>>>> > prioritizing tx relay by feerate are great ideas for preventing
>>>>> spammers
>>>>> > from wasting bandwidth network-wide. I agree, this would slow the low
>>>>> > feerate spam down, preventing a huge network-wide bandwidth spike.
>>>>> And it
>>>>> > would allow high feerate transactions to propagate as they should,
>>>>> > regardless of how busy traffic is. Combined with inbound tx request
>>>>> > rate-limiting, might this be sufficient to prevent DoS regardless of
>>>>> the
>>>>> > fee-based replacement policies?
>>>>>
>>>>> I think you only want to do outbound rate limits, ie, how often you
>>>>> send
>>>>> INV, GETDATA and TX messages? Once you receive any of those, I think
>>>>> you have to immediately process / ignore it, you can't really sensibly
>>>>> defer it (beyond the existing queues we have that just build up while
>>>>> we're busy processing other things first)?
>>>>>
>>>>> > One point that I'm not 100% clear on: is it ok to prioritize the
>>>>> > transactions by ancestor feerate in this scheme? As I described in
>>>>> the
>>>>> > original post, this can be quite different from the actual feerate
>>>>> we would
>>>>> > consider a transaction in a block for. The transaction could have a
>>>>> high
>>>>> > feerate sibling bumping its ancestor.
>>>>> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C
>>>>> (5sat/vB). If
>>>>> > we just received C, it would be incorrect to give it a priority
>>>>> equal to
>>>>> > its ancestor feerate (3sat/vB) because if we constructed a block
>>>>> template
>>>>> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
>>>>> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If
>>>>> we
>>>>> > also exclude C when building blocks, we're missing out on good fees.
>>>>>
>>>>> I think you're right that this would be ugly. It's something of a
>>>>> special case:
>>>>>
>>>>>  a) you really care about C getting into the next block; but
>>>>>  b) you're trusting B not being replaced by a higher fee tx that
>>>>>     doesn't have A as a parent; and
>>>>>  c) there's a lot of txs bidding the floor of the next block up to a
>>>>>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>>>>>     rate of 5sat/vB
>>>>>
>>>>> Without (a), maybe you don't care about it getting to a miner quickly.
>>>>> If your trust in (b) was misplaced, then your tx's effective fee rate
>>>>> will drop and (because of (c)), you'll lose anyway. And if the spam
>>>>> ends
>>>>> up outside of (c)'s range, either the rate limiting won't take effect
>>>>> (spam's too cheap) and you'll be fine, or you'll miss out on the block
>>>>> anyway (spam's paying more than your tx rate) and you never had any
>>>>> hope
>>>>> of making it in.
>>>>>
>>>>> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
>>>>> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
>>>>> per 10 minutes for outbound connections. This would be a weight based
>>>>> rate limit instead-of/in-addition-to that, I guess.
>>>>>
>>>>> As far as a non-ugly approach goes, I think you'd have to be smarter
>>>>> about
>>>>> tracking the "effective fee rate" than the ancestor fee rate manages;
>>>>> maybe that's something that could fall out of Murch and Clara's
>>>>> candidate
>>>>> set blockbuilding ideas [0] ?
>>>>>
>>>>> Perhaps that same work would also make it possible to come up with
>>>>> a better answer to "do I care that this replacement would invalidate
>>>>> these descendents?"
>>>>>
>>>>> [0] https://github.com/Xekyo/blockbuilding
>>>>>
>>>>> > > - keep high-feerate evicted txs around for a while in case they get
>>>>> > >   mined by someone else to improve compact block relay, a la the
>>>>> > >   orphan pool?
>>>>> > Replaced transactions are already added to vExtraTxnForCompact :D
>>>>>
>>>>> I guess I was thinking that it's just a 100 tx LRU cache, which might
>>>>> not be good enough?
>>>>>
>>>>> Maybe it would be more on point to have a rate limit apply only to
>>>>> replacement transactions?
>>>>>
>>>>> > For wallets, AJ's "All you need is for there to be *a* path that
>>>>> follows
>>>>> > the new relay rules and gets from your node/wallet to perhaps 10% of
>>>>> > hashpower" makes sense to me (which would be the former).
>>>>>
>>>>> Perhaps a corollarly of that is that it's *better* to have the mempool
>>>>> acceptance rule only consider economic incentives, and have the spam
>>>>> prevention only be about "shall I tell my peers about this?"
>>>>>
>>>>> If you don't have that split; then the anti-spam rules can prevent you
>>>>> from getting the tx in the mempool at all; whereas if you do have the
>>>>> split, then even if the bitcoind anti-spam rules are blocking you at
>>>>> every turn, you can still send your tx to miners by some other route,
>>>>> and then they can add it to their mempool directly without any hassle.
>>>>>
>>>>> Cheers,
>>>>> aj
>>>>>
>>>>> _______________________________________________
>>>> bitcoin-dev mailing list
>>>> bitcoin-dev@lists.linuxfoundation.org
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>>
>>>

[-- Attachment #2: Type: text/html, Size: 29295 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Gloria Zhao]

[-- Attachment #1: Type: text/plain, Size: 25249 bytes --]

Hi Billy,

> We should expect miners will be using a more complex, more optimal way of
determining what blocks they're working on [...] we should instead run with
the assumption that miners keep all potentially relevant transactions in
their mempools, including potentially many conflicting transctions, in
order to create the most profitable blocks. And therefore we shouldn't put
the constraint on normal non-mining full nodes to do that same more-complex
mempool behavior or add any complexity for the purpose of denying
transaction replacements.

> I think a lot of the complexity in these ideas is because of the attempt
to match relay rules with miner
inclusion rules.

I think the assumption that miners are using a completely different
implementation of mempool and block template building is false. IIUC, most
miners use Bitcoin Core and perhaps configure their node differently (e.g.
larger mempool and different minfeerates), but also use `getblocktemplate`
which means the same ancestor package-based mining algorithm.

Of course, I'm not a miner, so if anybody is a miner or has seen miners'
setups, please correct me if I'm wrong.

In either case, we would want our mining algorithm to result in block
templates that are as close as possible to perfectly incentive compatibile.

Fundamentally, I believe default mempool policy (which perhaps naturally
creates a network-wide transaction relay policy) should be as close to the
mining code as possible. Imagine node A only keeps 1 block's worth of
transactions, and node B keeps a (default) 300MB mempool. The contents of
node A's mempool should be as close as possible to a block template
generated from node B's mempool. Otherwise, node A's mempool is not very
useful - their fee estimation is flawed and compact block relay won't do
them much good if they need to re-request a lot of block transactions.
Next, imagine that node B is a miner. It would be very suboptimal if the
mining code was ancestor package-based (i.e. supports CPFP), but the
mempool policy only cared about individual feerate, and evicted low-fee
parents despite their high-fee children. It's easy to see why this would be
suboptimal.
Attempting to match mempool policy with the mining algorithm is also
arguably the point of package relay. Our mining code uses ancestor packages
which is good, but we only submit transactions one at a time to the
mempool, so a transaction's high-fee children can't be considered until
they are all already in the mempool. Package relay allows us to start
thinking about ancestor packages immediately when evaluating transactions
for submission to the mempool.

The attempt to match policy with miner inclusion rules is deliberate and
necessary.

> I want to echo James O'Beirne's opinion on this that this may be the
wrong path to go down (a path of more complexity without much gain). He
said: "Special consideration for "what should be in the next block" and/or
the caching of block templates seems like an imposing dependency, dragging
in a bunch of state and infrastructure to a question that should be solely
limited to mempool feerate aggregates and the feerate of the particular txn
package a wallet is concerned with."

It seems that I under-explained the purpose of building/caching block
templates in my original post, since both you and James have the same
misunderstanding. Since RBF's introduction, we have improved to an ancestor
package-based mining algorithm. This supports CPFP (incentive compatible)
and it is now common to see more complex "families" of transactions as
users fee-bump transactions (market is working, yay). On the other hand, we
no longer have an accurate way of determining a transaction's "mining
score," i.e., the feerate of this transaction's ancestor package when it is
included in a block template using our current mining algorithm.

This limitation is a big blocker in proposing new fee/feerate RBF rules.
For example, if we say "the transaction needs a better feerate," this is
obviously flawed, since the original transactions may have very
high-feerate children, and the replacement transaction may have low feerate
parents. So what we really want is "the transaction needs to be more
incentive compatible to mine based on our mining algorithm," but we have no
way of getting that information right now.

In my original post, I [described 4 heuristics to get transaction's "mining
score"][1] using the current data cached in the mempool (e.g. max ancestor
feerate of descendant set), as well as why they don't work. As such, the
best way to calculate a transaction's mining score AFAICT is to grab all of
the related transactions and build a mini "block template" with them. The
[implementation][2] I sent last week also cuts out some of the fluff, so
the pseudocode looks like this:

// Get ALL connected entries (ancestors, descendants, siblings, cousins,
coparents, etc.)
vector<TxMempoolEntry> cluster = mempool.GetAllTransactionsRelatedTo(txid);
sort(cluster, ancestorfeerate);

// For deducting ancestors when they are included separately
vector<ModifiedTxMempoolEntry> modified_entries;

while (!cluster.empty() and !modified_entries.empty()) {
    iter = BetterAncestorFeerateOf(cluster, modified_entries);
    best_ancestor_package = GetAncestorSet(iter);
    mining_score = Feerate(best_ancestor_package);
    for (entry : best_ancestor_package) {
       mining_scores[entry] = mining_score;
       for (descendant : GetAllDescendants(entry) {
           modified_entries[descendant].DeductAncestorFromAccounting(entry);
       }
    }
}

Perhaps somebody will come up with a better way to do this, but this is my
solution, and I hope I've now sufficiently described why I've made an
effort to think about this. It's not because I want to make things more
fancy or complicated, but because we currently have no way of accurately
determining a transaction's mining score. The reason I proposed a
persistent block template is so we can efficiently query the mining score
of all transactions in the mempool.

> However, I do think that these DOS concerns are quite overblown. I wrote
up a comment on your rbf-improvements.md
<https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#gistcomment-4093100>
detailing
my thought process on that. The summary is that as long as the fee-rate
relay rule is maintained, any "spam" is actually paid for, either by the
"first" transaction in the spam chain, or by the "spam" itself.

The DoS concern is not overblown. I recommend you re-read the [current RBF
policy][3]; Rule 3 and 4 *are* the feerate relay rules. Removing Rule 3
means allowing recycled fees, so new transactions are not necessarily
"paying" anything.

[1]:
https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#mining-score-of-a-mempool-transaction
[2]: https://github.com/glozow/bitcoin/tree/2022-02-mining-score
[3]:
https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md

Best,
Gloria

On Sat, Mar 12, 2022 at 8:18 AM Billy Tetrud <billy.tetrud@gmail.com> wrote:

> In reading through more of the discussion, it seems the idea I presented
> above might basically be a reformulation of t-bast's rate-limiting idea
> presented in this comment
> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4081349#gistcomment-4081349>.
> Perhaps he could comment on whether that characterization is accurate.
>
> On Fri, Mar 11, 2022 at 10:22 AM Billy Tetrud <billy.tetrud@gmail.com>
> wrote:
>
>> Hi Gloria,
>>
>> >  1. Transaction relay rate limiting
>>
>> I have a similar concern as yours, that this could prevent higher
>> fee-rate transactions from being broadcast.
>>
>> > 2. Staggered broadcast of replacement transactions: within some time
>> interval, maybe accept multiple replacements for the same prevout, but only
>> relay the original transaction.
>>
>> By this do you mean basically having a batching window where, on
>> receiving a replacement transaction, a node will wait for a period of time,
>> potentially receiving many replacements for the same transaction (or many
>> separate conflicting transactions), and only broadcasting the "best" one(s)
>> at the end of that time window?
>>
>> Its an interesting idea, but it would produce a problem. Every hop that
>> replacement transaction takes would be delayed by this staggered window. If
>> the window were 3 minutes and transactions generally take 20 hops to get to
>> the majority of miners, a "best-case average" delay might be 3.75 minutes
>> (noting that among your 8 nodes, its quite likely one of them would have a
>> window ending much sooner than 3 minutes). Some (maybe 3% of) nodes would
>> experience delays of more than 20 minutes. That kind of delay isn't great.
>>
>> However it made me think of another idea: a transaction replacement
>> broadcast cooldown. What if nodes kept track of the time they broadcasted
>> the last replacement for a package and had a relay cooldown after the last
>> replacement was broadcasted? A node receiving a replacement would relay the
>> replacement immediately if the package its replacing was broadcasted more
>> than X seconds ago, and otherwise it would wait until the time when that
>> package was broadcasted at least X seconds ago to broadcast it. Any
>> replacements it receives during that waiting period would replace as
>> normal, meaning the unrebroadcasted replacement would never be
>> broadcasted, and only the highest value replacement would be broadcasted at
>> the end of the cooldown.
>>
>> This wouldn't prevent a higher-fee-rate transaction from being
>> broadcasted (like rate limiting could), but would still be effective at
>> limiting unnecessary data transmission. Another benefit is that in the
>> non-adversarial case, replacement transactions wouldn't be subject to any
>> delay at all (while in the staggered broadcast idea, most replacements
>> would experience some delay). And in the adversarial case, where a
>> malicious actor broadcasts a low-as-possible-value replacement just before
>> yours, the worst case delay is just whatever the cooldown period is. I
>> would imagine that maybe 1 minute would be a reasonable worst-case delay.
>> This would limit spam for a transaction that makes it into a block to ~10x
>> (9 to 1). I don't see much of a downside to doing this beyond just the
>> slight additional complexity of relay rules (and considering it could save
>> substantial additional code complexity, even that is a benefit).
>>
>> All a node would need to do is keep a timestamp on each transaction they
>> receive for when it was broadcasted and check it when a replacement comes
>> in. If now-broadcastDate < cooldown, set a timer for cooldown -
>> (now-broadcastDate) to broadcast it. If another replacement comes in, clear
>> that timer and repeat using the original broadcast date (since the
>> unbroadcast transaction doesn't have a broadcast date yet).
>>
>> I think it might also be useful to note that eliminating "extra data"
>> caused by careless or malicious actors (spam or whatever you want to call
>> it) should not be the goal. It is impossible to prevent all spam. What we
>> should be aiming for is more specific: we should attempt to design a system
>> where spam is manageable. Eg if our goal is to ensure that a bitcoin node
>> uses no more than 10% of the bandwidth of a "normal" user, if current
>> non-spam traffic only requires 1% of a "normal" users's bandwidth, then the
>> network can bear a 9 to 1 ratio of spam. When a node spins up, there is a
>> lot more data to download and process. So we know that all full nodes can
>> handle at least as much traffic as they handle during IBD. What's the
>> difference between those amounts? I'm not sure, but I would guess that IBD
>> is at least a couple times more demanding than a fully synced node. So I
>> might suggest that as long as spam can be kept below a ratio of maybe 2 to
>> 1, we should consider the design acceptable (and therefore more complexity
>> unnecessary).
>>
>> The 1 minute broadcast cooldown I mentioned before wouldn't be quite
>> sufficient to achieve that ratio. But a 3.33 minute cooldown would be.
>> Whether this is "too much" is something that would have to be discussed, I
>> suspect a worst-case adversarial 3.33 minute delay would not be "too much".
>> Doing this could basically eliminate any risk of actual service denial via
>> replacement transactions.
>>
>> However, I do think that these DOS concerns are quite overblown. I wrote
>> up a comment on your rbf-improvements.md
>> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#gistcomment-4093100> detailing
>> my thought process on that. The summary is that as long as the fee-rate
>> relay rule is maintained, any "spam" is actually paid for, either by the
>> "first" transaction in the spam chain, or by the "spam" itself. Even
>> without something like a minimum RBF relay delay limiting how much spam
>> could be created, the economics of the fee-rate rule already sufficiently
>> mitigate the issue of spam.
>> On Wed, Mar 9, 2022 at 9:37 AM Gloria Zhao via bitcoin-dev <
>> bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>>> Hi RBF friends,
>>>
>>> Posting a summary of RBF discussions at coredev (mostly on transaction
>>> relay rate-limiting), user-elected descendant limit as a short term
>>> solution to unblock package RBF, and mining score, all open for feedback:
>>>
>>> One big concept discussed was baking DoS protection into the p2p level
>>> rather than policy level. TLDR: The fees are not paid to the node operator,
>>> but to the miner. While we can use fees to reason about the cost of an
>>> attack, if we're ultimately interested in preventing resource exhaustion,
>>> maybe we want to "stop the bleeding" when it happens and bound the amount
>>> of resources used in general. There were two main ideas:
>>>
>>> 1. Transaction relay rate limiting (i.e. the one you proposed above or
>>> some variation) with a feerate-based priority queue
>>> 2. Staggered broadcast of replacement transactions: within some time
>>> interval, maybe accept multiple replacements for the same prevout, but only
>>> relay the original transaction.
>>>
>>> Looking to solicit feedback on these ideas and the concept in general.
>>> Is it a good idea (separate from RBF) to add rate-limiting in transaction
>>> relay? And is it the right direction to think about RBF DoS protection this
>>> way?
>>>
>>> A lingering concern that I have about this idea is it would then be
>>> possible to impact the propagation of another person’s transaction, i.e.,
>>> an attacker can censor somebody’s transaction from ever being announced by
>>> a node if they send enough transactions to fill up the rate limit.
>>> Obviously this would be expensive since they're spending a lot on fees, but
>>> I imagine it could be profitable in some situations to spend a few thousand
>>> dollars to prevent anyone from hearing about a transaction for a few hours.
>>> This might be a non-issue in practice if the rate limit is generous and
>>> traffic isn’t horrendous, but is this a problem?
>>>
>>> And if we don't require an increase in (i.e. addition of "new") absolute
>>> fees, users are essentially allowed to “recycle” fees. In the scenario
>>> where we prioritize relay based on feerate, users could potentially be
>>> placed higher in the queue, ahead of other users’ transactions, multiple
>>> times, without ever adding more fees to the transaction. Again, maybe this
>>> isn’t a huge deal in practice if we set the parameters right, but it seems…
>>> not great, in principle.
>>>
>>> ---------
>>>
>>> It's probably also a good idea to point out that there's been some
>>> discussion happening on the gist containing my original post on this thread
>>> (https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).
>>>
>>> Suhas and Matt [proposed][0] adding a policy rule allowing users to
>>> specify descendant limits on their transactions. For example, some nth bit
>>> of nSequence with nVersion 3 means "this transaction won't have more than X
>>> vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
>>> solves the pinning problem with package RBF where the attacker's package
>>> contains a very large and high-fee descendant.
>>>
>>> We could add this policy and deploy it with package RBF/package relay so
>>> that LN can use it by setting the user-elected descendant limit flag on
>>> commitment transactions. (Otherwise package RBF is blocked until we find a
>>> more comprehensive solution to the pinning attack).
>>>
>>> It's simple to [implement][1] as a mempool policy, but adds some
>>> complexity for wallets that use it, since it limits their use of UTXOs from
>>> transactions with this bit set.
>>>
>>> ---------
>>>
>>> Also, coming back to the idea of "we can't just use {individual,
>>> ancestor} feerate," I'm interested in soliciting feedback on adding a
>>> “mining score” calculator. I've implemented one [here][2] which takes the
>>> transaction in question, grabs all of the connected mempool transactions
>>> (including siblings, coparents, etc., as they wouldn’t be in the ancestor
>>> nor descendant sets), and builds a “block template” using our current
>>> mining algorithm. The mining score of a transaction is the ancestor feerate
>>> at which it is included.
>>>
>>> This would be helpful for something like ancestor-aware funding and
>>> fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
>>> priority queue for transaction relay, we'd want to use something like this
>>> as the priority value. And for RBF, we probably want to require that a
>>> replacement have a higher mining score than the original transactions. This
>>> could be computationally expensive to do all the time; it could be good to
>>> cache it but that could make mempool bookkeeping more complicated. Also, if
>>> we end up trying to switch to a candidate set-based algorithm for mining,
>>> we'd of course need a new calculator.
>>>
>>> [0]:
>>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4058140#gistcomment-4058140
>>> [1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
>>> [2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
>>> [3]: https://github.com/bitcoin/bitcoin/issues/9645
>>> [4]: https://github.com/bitcoin/bitcoin/issues/15553
>>>
>>> Best,
>>> Gloria
>>>
>>> On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:
>>>
>>>> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
>>>> > @aj:
>>>> > > I wonder sometimes if it could be sufficient to just have a relay
>>>> rate
>>>> > > limit and prioritise by ancestor feerate though. Maybe something
>>>> like:
>>>> > > - instead of adding txs to each peers setInventoryTxToSend
>>>> immediately,
>>>> > >   set a mempool flag "relayed=false"
>>>> > > - on a time delay, add the top N (by fee rate) "relayed=false" txs
>>>> to
>>>> > >   each peer's setInventoryTxToSend and mark them as "relayed=true";
>>>> > >   calculate how much kB those txs were, and do this again after
>>>> > >   SIZE/RATELIMIT seconds
>>>>
>>>> > > - don't include "relayed=false" txs when building blocks?
>>>>
>>>> The "?" was me not being sure that point is a good suggestion...
>>>>
>>>> Miners might reasonably decide to have no rate limit, and always relay,
>>>> and never exclude txs -- but the question then becomes is whether they
>>>> hear about the tx at all, so rate limiting behaviour could still be a
>>>> potential problem for whoever made the tx.
>>>>
>>>> > Wow cool! I think outbound tx relay size-based rate-limiting and
>>>> > prioritizing tx relay by feerate are great ideas for preventing
>>>> spammers
>>>> > from wasting bandwidth network-wide. I agree, this would slow the low
>>>> > feerate spam down, preventing a huge network-wide bandwidth spike.
>>>> And it
>>>> > would allow high feerate transactions to propagate as they should,
>>>> > regardless of how busy traffic is. Combined with inbound tx request
>>>> > rate-limiting, might this be sufficient to prevent DoS regardless of
>>>> the
>>>> > fee-based replacement policies?
>>>>
>>>> I think you only want to do outbound rate limits, ie, how often you send
>>>> INV, GETDATA and TX messages? Once you receive any of those, I think
>>>> you have to immediately process / ignore it, you can't really sensibly
>>>> defer it (beyond the existing queues we have that just build up while
>>>> we're busy processing other things first)?
>>>>
>>>> > One point that I'm not 100% clear on: is it ok to prioritize the
>>>> > transactions by ancestor feerate in this scheme? As I described in the
>>>> > original post, this can be quite different from the actual feerate we
>>>> would
>>>> > consider a transaction in a block for. The transaction could have a
>>>> high
>>>> > feerate sibling bumping its ancestor.
>>>> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C
>>>> (5sat/vB). If
>>>> > we just received C, it would be incorrect to give it a priority equal
>>>> to
>>>> > its ancestor feerate (3sat/vB) because if we constructed a block
>>>> template
>>>> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
>>>> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If
>>>> we
>>>> > also exclude C when building blocks, we're missing out on good fees.
>>>>
>>>> I think you're right that this would be ugly. It's something of a
>>>> special case:
>>>>
>>>>  a) you really care about C getting into the next block; but
>>>>  b) you're trusting B not being replaced by a higher fee tx that
>>>>     doesn't have A as a parent; and
>>>>  c) there's a lot of txs bidding the floor of the next block up to a
>>>>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>>>>     rate of 5sat/vB
>>>>
>>>> Without (a), maybe you don't care about it getting to a miner quickly.
>>>> If your trust in (b) was misplaced, then your tx's effective fee rate
>>>> will drop and (because of (c)), you'll lose anyway. And if the spam ends
>>>> up outside of (c)'s range, either the rate limiting won't take effect
>>>> (spam's too cheap) and you'll be fine, or you'll miss out on the block
>>>> anyway (spam's paying more than your tx rate) and you never had any hope
>>>> of making it in.
>>>>
>>>> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
>>>> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
>>>> per 10 minutes for outbound connections. This would be a weight based
>>>> rate limit instead-of/in-addition-to that, I guess.
>>>>
>>>> As far as a non-ugly approach goes, I think you'd have to be smarter
>>>> about
>>>> tracking the "effective fee rate" than the ancestor fee rate manages;
>>>> maybe that's something that could fall out of Murch and Clara's
>>>> candidate
>>>> set blockbuilding ideas [0] ?
>>>>
>>>> Perhaps that same work would also make it possible to come up with
>>>> a better answer to "do I care that this replacement would invalidate
>>>> these descendents?"
>>>>
>>>> [0] https://github.com/Xekyo/blockbuilding
>>>>
>>>> > > - keep high-feerate evicted txs around for a while in case they get
>>>> > >   mined by someone else to improve compact block relay, a la the
>>>> > >   orphan pool?
>>>> > Replaced transactions are already added to vExtraTxnForCompact :D
>>>>
>>>> I guess I was thinking that it's just a 100 tx LRU cache, which might
>>>> not be good enough?
>>>>
>>>> Maybe it would be more on point to have a rate limit apply only to
>>>> replacement transactions?
>>>>
>>>> > For wallets, AJ's "All you need is for there to be *a* path that
>>>> follows
>>>> > the new relay rules and gets from your node/wallet to perhaps 10% of
>>>> > hashpower" makes sense to me (which would be the former).
>>>>
>>>> Perhaps a corollarly of that is that it's *better* to have the mempool
>>>> acceptance rule only consider economic incentives, and have the spam
>>>> prevention only be about "shall I tell my peers about this?"
>>>>
>>>> If you don't have that split; then the anti-spam rules can prevent you
>>>> from getting the tx in the mempool at all; whereas if you do have the
>>>> split, then even if the bitcoind anti-spam rules are blocking you at
>>>> every turn, you can still send your tx to miners by some other route,
>>>> and then they can add it to their mempool directly without any hassle.
>>>>
>>>> Cheers,
>>>> aj
>>>>
>>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>

[-- Attachment #2: Type: text/html, Size: 28407 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Billy Tetrud]

[-- Attachment #1: Type: text/plain, Size: 17656 bytes --]

In reading through more of the discussion, it seems the idea I presented
above might basically be a reformulation of t-bast's rate-limiting idea
presented in this comment
<https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4081349#gistcomment-4081349>.
Perhaps he could comment on whether that characterization is accurate.

On Fri, Mar 11, 2022 at 10:22 AM Billy Tetrud <billy.tetrud@gmail.com>
wrote:

> Hi Gloria,
>
> >  1. Transaction relay rate limiting
>
> I have a similar concern as yours, that this could prevent higher fee-rate
> transactions from being broadcast.
>
> > 2. Staggered broadcast of replacement transactions: within some time
> interval, maybe accept multiple replacements for the same prevout, but only
> relay the original transaction.
>
> By this do you mean basically having a batching window where, on receiving
> a replacement transaction, a node will wait for a period of time,
> potentially receiving many replacements for the same transaction (or many
> separate conflicting transactions), and only broadcasting the "best" one(s)
> at the end of that time window?
>
> Its an interesting idea, but it would produce a problem. Every hop that
> replacement transaction takes would be delayed by this staggered window. If
> the window were 3 minutes and transactions generally take 20 hops to get to
> the majority of miners, a "best-case average" delay might be 3.75 minutes
> (noting that among your 8 nodes, its quite likely one of them would have a
> window ending much sooner than 3 minutes). Some (maybe 3% of) nodes would
> experience delays of more than 20 minutes. That kind of delay isn't great.
>
> However it made me think of another idea: a transaction replacement
> broadcast cooldown. What if nodes kept track of the time they broadcasted
> the last replacement for a package and had a relay cooldown after the last
> replacement was broadcasted? A node receiving a replacement would relay the
> replacement immediately if the package its replacing was broadcasted more
> than X seconds ago, and otherwise it would wait until the time when that
> package was broadcasted at least X seconds ago to broadcast it. Any
> replacements it receives during that waiting period would replace as
> normal, meaning the unrebroadcasted replacement would never be
> broadcasted, and only the highest value replacement would be broadcasted at
> the end of the cooldown.
>
> This wouldn't prevent a higher-fee-rate transaction from being broadcasted
> (like rate limiting could), but would still be effective at limiting
> unnecessary data transmission. Another benefit is that in the
> non-adversarial case, replacement transactions wouldn't be subject to any
> delay at all (while in the staggered broadcast idea, most replacements
> would experience some delay). And in the adversarial case, where a
> malicious actor broadcasts a low-as-possible-value replacement just before
> yours, the worst case delay is just whatever the cooldown period is. I
> would imagine that maybe 1 minute would be a reasonable worst-case delay.
> This would limit spam for a transaction that makes it into a block to ~10x
> (9 to 1). I don't see much of a downside to doing this beyond just the
> slight additional complexity of relay rules (and considering it could save
> substantial additional code complexity, even that is a benefit).
>
> All a node would need to do is keep a timestamp on each transaction they
> receive for when it was broadcasted and check it when a replacement comes
> in. If now-broadcastDate < cooldown, set a timer for cooldown -
> (now-broadcastDate) to broadcast it. If another replacement comes in, clear
> that timer and repeat using the original broadcast date (since the
> unbroadcast transaction doesn't have a broadcast date yet).
>
> I think it might also be useful to note that eliminating "extra data"
> caused by careless or malicious actors (spam or whatever you want to call
> it) should not be the goal. It is impossible to prevent all spam. What we
> should be aiming for is more specific: we should attempt to design a system
> where spam is manageable. Eg if our goal is to ensure that a bitcoin node
> uses no more than 10% of the bandwidth of a "normal" user, if current
> non-spam traffic only requires 1% of a "normal" users's bandwidth, then the
> network can bear a 9 to 1 ratio of spam. When a node spins up, there is a
> lot more data to download and process. So we know that all full nodes can
> handle at least as much traffic as they handle during IBD. What's the
> difference between those amounts? I'm not sure, but I would guess that IBD
> is at least a couple times more demanding than a fully synced node. So I
> might suggest that as long as spam can be kept below a ratio of maybe 2 to
> 1, we should consider the design acceptable (and therefore more complexity
> unnecessary).
>
> The 1 minute broadcast cooldown I mentioned before wouldn't be quite
> sufficient to achieve that ratio. But a 3.33 minute cooldown would be.
> Whether this is "too much" is something that would have to be discussed, I
> suspect a worst-case adversarial 3.33 minute delay would not be "too much".
> Doing this could basically eliminate any risk of actual service denial via
> replacement transactions.
>
> However, I do think that these DOS concerns are quite overblown. I wrote
> up a comment on your rbf-improvements.md
> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#gistcomment-4093100> detailing
> my thought process on that. The summary is that as long as the fee-rate
> relay rule is maintained, any "spam" is actually paid for, either by the
> "first" transaction in the spam chain, or by the "spam" itself. Even
> without something like a minimum RBF relay delay limiting how much spam
> could be created, the economics of the fee-rate rule already sufficiently
> mitigate the issue of spam.
> On Wed, Mar 9, 2022 at 9:37 AM Gloria Zhao via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Hi RBF friends,
>>
>> Posting a summary of RBF discussions at coredev (mostly on transaction
>> relay rate-limiting), user-elected descendant limit as a short term
>> solution to unblock package RBF, and mining score, all open for feedback:
>>
>> One big concept discussed was baking DoS protection into the p2p level
>> rather than policy level. TLDR: The fees are not paid to the node operator,
>> but to the miner. While we can use fees to reason about the cost of an
>> attack, if we're ultimately interested in preventing resource exhaustion,
>> maybe we want to "stop the bleeding" when it happens and bound the amount
>> of resources used in general. There were two main ideas:
>>
>> 1. Transaction relay rate limiting (i.e. the one you proposed above or
>> some variation) with a feerate-based priority queue
>> 2. Staggered broadcast of replacement transactions: within some time
>> interval, maybe accept multiple replacements for the same prevout, but only
>> relay the original transaction.
>>
>> Looking to solicit feedback on these ideas and the concept in general. Is
>> it a good idea (separate from RBF) to add rate-limiting in transaction
>> relay? And is it the right direction to think about RBF DoS protection this
>> way?
>>
>> A lingering concern that I have about this idea is it would then be
>> possible to impact the propagation of another person’s transaction, i.e.,
>> an attacker can censor somebody’s transaction from ever being announced by
>> a node if they send enough transactions to fill up the rate limit.
>> Obviously this would be expensive since they're spending a lot on fees, but
>> I imagine it could be profitable in some situations to spend a few thousand
>> dollars to prevent anyone from hearing about a transaction for a few hours.
>> This might be a non-issue in practice if the rate limit is generous and
>> traffic isn’t horrendous, but is this a problem?
>>
>> And if we don't require an increase in (i.e. addition of "new") absolute
>> fees, users are essentially allowed to “recycle” fees. In the scenario
>> where we prioritize relay based on feerate, users could potentially be
>> placed higher in the queue, ahead of other users’ transactions, multiple
>> times, without ever adding more fees to the transaction. Again, maybe this
>> isn’t a huge deal in practice if we set the parameters right, but it seems…
>> not great, in principle.
>>
>> ---------
>>
>> It's probably also a good idea to point out that there's been some
>> discussion happening on the gist containing my original post on this thread
>> (https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).
>>
>> Suhas and Matt [proposed][0] adding a policy rule allowing users to
>> specify descendant limits on their transactions. For example, some nth bit
>> of nSequence with nVersion 3 means "this transaction won't have more than X
>> vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
>> solves the pinning problem with package RBF where the attacker's package
>> contains a very large and high-fee descendant.
>>
>> We could add this policy and deploy it with package RBF/package relay so
>> that LN can use it by setting the user-elected descendant limit flag on
>> commitment transactions. (Otherwise package RBF is blocked until we find a
>> more comprehensive solution to the pinning attack).
>>
>> It's simple to [implement][1] as a mempool policy, but adds some
>> complexity for wallets that use it, since it limits their use of UTXOs from
>> transactions with this bit set.
>>
>> ---------
>>
>> Also, coming back to the idea of "we can't just use {individual,
>> ancestor} feerate," I'm interested in soliciting feedback on adding a
>> “mining score” calculator. I've implemented one [here][2] which takes the
>> transaction in question, grabs all of the connected mempool transactions
>> (including siblings, coparents, etc., as they wouldn’t be in the ancestor
>> nor descendant sets), and builds a “block template” using our current
>> mining algorithm. The mining score of a transaction is the ancestor feerate
>> at which it is included.
>>
>> This would be helpful for something like ancestor-aware funding and
>> fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
>> priority queue for transaction relay, we'd want to use something like this
>> as the priority value. And for RBF, we probably want to require that a
>> replacement have a higher mining score than the original transactions. This
>> could be computationally expensive to do all the time; it could be good to
>> cache it but that could make mempool bookkeeping more complicated. Also, if
>> we end up trying to switch to a candidate set-based algorithm for mining,
>> we'd of course need a new calculator.
>>
>> [0]:
>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4058140#gistcomment-4058140
>> [1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
>> [2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
>> [3]: https://github.com/bitcoin/bitcoin/issues/9645
>> [4]: https://github.com/bitcoin/bitcoin/issues/15553
>>
>> Best,
>> Gloria
>>
>> On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:
>>
>>> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
>>> > @aj:
>>> > > I wonder sometimes if it could be sufficient to just have a relay
>>> rate
>>> > > limit and prioritise by ancestor feerate though. Maybe something
>>> like:
>>> > > - instead of adding txs to each peers setInventoryTxToSend
>>> immediately,
>>> > >   set a mempool flag "relayed=false"
>>> > > - on a time delay, add the top N (by fee rate) "relayed=false" txs to
>>> > >   each peer's setInventoryTxToSend and mark them as "relayed=true";
>>> > >   calculate how much kB those txs were, and do this again after
>>> > >   SIZE/RATELIMIT seconds
>>>
>>> > > - don't include "relayed=false" txs when building blocks?
>>>
>>> The "?" was me not being sure that point is a good suggestion...
>>>
>>> Miners might reasonably decide to have no rate limit, and always relay,
>>> and never exclude txs -- but the question then becomes is whether they
>>> hear about the tx at all, so rate limiting behaviour could still be a
>>> potential problem for whoever made the tx.
>>>
>>> > Wow cool! I think outbound tx relay size-based rate-limiting and
>>> > prioritizing tx relay by feerate are great ideas for preventing
>>> spammers
>>> > from wasting bandwidth network-wide. I agree, this would slow the low
>>> > feerate spam down, preventing a huge network-wide bandwidth spike. And
>>> it
>>> > would allow high feerate transactions to propagate as they should,
>>> > regardless of how busy traffic is. Combined with inbound tx request
>>> > rate-limiting, might this be sufficient to prevent DoS regardless of
>>> the
>>> > fee-based replacement policies?
>>>
>>> I think you only want to do outbound rate limits, ie, how often you send
>>> INV, GETDATA and TX messages? Once you receive any of those, I think
>>> you have to immediately process / ignore it, you can't really sensibly
>>> defer it (beyond the existing queues we have that just build up while
>>> we're busy processing other things first)?
>>>
>>> > One point that I'm not 100% clear on: is it ok to prioritize the
>>> > transactions by ancestor feerate in this scheme? As I described in the
>>> > original post, this can be quite different from the actual feerate we
>>> would
>>> > consider a transaction in a block for. The transaction could have a
>>> high
>>> > feerate sibling bumping its ancestor.
>>> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB).
>>> If
>>> > we just received C, it would be incorrect to give it a priority equal
>>> to
>>> > its ancestor feerate (3sat/vB) because if we constructed a block
>>> template
>>> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
>>> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If we
>>> > also exclude C when building blocks, we're missing out on good fees.
>>>
>>> I think you're right that this would be ugly. It's something of a
>>> special case:
>>>
>>>  a) you really care about C getting into the next block; but
>>>  b) you're trusting B not being replaced by a higher fee tx that
>>>     doesn't have A as a parent; and
>>>  c) there's a lot of txs bidding the floor of the next block up to a
>>>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>>>     rate of 5sat/vB
>>>
>>> Without (a), maybe you don't care about it getting to a miner quickly.
>>> If your trust in (b) was misplaced, then your tx's effective fee rate
>>> will drop and (because of (c)), you'll lose anyway. And if the spam ends
>>> up outside of (c)'s range, either the rate limiting won't take effect
>>> (spam's too cheap) and you'll be fine, or you'll miss out on the block
>>> anyway (spam's paying more than your tx rate) and you never had any hope
>>> of making it in.
>>>
>>> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
>>> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
>>> per 10 minutes for outbound connections. This would be a weight based
>>> rate limit instead-of/in-addition-to that, I guess.
>>>
>>> As far as a non-ugly approach goes, I think you'd have to be smarter
>>> about
>>> tracking the "effective fee rate" than the ancestor fee rate manages;
>>> maybe that's something that could fall out of Murch and Clara's candidate
>>> set blockbuilding ideas [0] ?
>>>
>>> Perhaps that same work would also make it possible to come up with
>>> a better answer to "do I care that this replacement would invalidate
>>> these descendents?"
>>>
>>> [0] https://github.com/Xekyo/blockbuilding
>>>
>>> > > - keep high-feerate evicted txs around for a while in case they get
>>> > >   mined by someone else to improve compact block relay, a la the
>>> > >   orphan pool?
>>> > Replaced transactions are already added to vExtraTxnForCompact :D
>>>
>>> I guess I was thinking that it's just a 100 tx LRU cache, which might
>>> not be good enough?
>>>
>>> Maybe it would be more on point to have a rate limit apply only to
>>> replacement transactions?
>>>
>>> > For wallets, AJ's "All you need is for there to be *a* path that
>>> follows
>>> > the new relay rules and gets from your node/wallet to perhaps 10% of
>>> > hashpower" makes sense to me (which would be the former).
>>>
>>> Perhaps a corollarly of that is that it's *better* to have the mempool
>>> acceptance rule only consider economic incentives, and have the spam
>>> prevention only be about "shall I tell my peers about this?"
>>>
>>> If you don't have that split; then the anti-spam rules can prevent you
>>> from getting the tx in the mempool at all; whereas if you do have the
>>> split, then even if the bitcoind anti-spam rules are blocking you at
>>> every turn, you can still send your tx to miners by some other route,
>>> and then they can add it to their mempool directly without any hassle.
>>>
>>> Cheers,
>>> aj
>>>
>>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

[-- Attachment #2: Type: text/html, Size: 20081 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Billy Tetrud]

[-- Attachment #1: Type: text/plain, Size: 16777 bytes --]

Hi Gloria,

>  1. Transaction relay rate limiting

I have a similar concern as yours, that this could prevent higher fee-rate
transactions from being broadcast.

> 2. Staggered broadcast of replacement transactions: within some time
interval, maybe accept multiple replacements for the same prevout, but only
relay the original transaction.

By this do you mean basically having a batching window where, on receiving
a replacement transaction, a node will wait for a period of time,
potentially receiving many replacements for the same transaction (or many
separate conflicting transactions), and only broadcasting the "best" one(s)
at the end of that time window?

Its an interesting idea, but it would produce a problem. Every hop that
replacement transaction takes would be delayed by this staggered window. If
the window were 3 minutes and transactions generally take 20 hops to get to
the majority of miners, a "best-case average" delay might be 3.75 minutes
(noting that among your 8 nodes, its quite likely one of them would have a
window ending much sooner than 3 minutes). Some (maybe 3% of) nodes would
experience delays of more than 20 minutes. That kind of delay isn't great.

However it made me think of another idea: a transaction replacement
broadcast cooldown. What if nodes kept track of the time they broadcasted
the last replacement for a package and had a relay cooldown after the last
replacement was broadcasted? A node receiving a replacement would relay the
replacement immediately if the package its replacing was broadcasted more
than X seconds ago, and otherwise it would wait until the time when that
package was broadcasted at least X seconds ago to broadcast it. Any
replacements it receives during that waiting period would replace as
normal, meaning the unrebroadcasted replacement would never be
broadcasted, and only the highest value replacement would be broadcasted at
the end of the cooldown.

This wouldn't prevent a higher-fee-rate transaction from being broadcasted
(like rate limiting could), but would still be effective at limiting
unnecessary data transmission. Another benefit is that in the
non-adversarial case, replacement transactions wouldn't be subject to any
delay at all (while in the staggered broadcast idea, most replacements
would experience some delay). And in the adversarial case, where a
malicious actor broadcasts a low-as-possible-value replacement just before
yours, the worst case delay is just whatever the cooldown period is. I
would imagine that maybe 1 minute would be a reasonable worst-case delay.
This would limit spam for a transaction that makes it into a block to ~10x
(9 to 1). I don't see much of a downside to doing this beyond just the
slight additional complexity of relay rules (and considering it could save
substantial additional code complexity, even that is a benefit).

All a node would need to do is keep a timestamp on each transaction they
receive for when it was broadcasted and check it when a replacement comes
in. If now-broadcastDate < cooldown, set a timer for cooldown -
(now-broadcastDate) to broadcast it. If another replacement comes in, clear
that timer and repeat using the original broadcast date (since the
unbroadcast transaction doesn't have a broadcast date yet).

I think it might also be useful to note that eliminating "extra data"
caused by careless or malicious actors (spam or whatever you want to call
it) should not be the goal. It is impossible to prevent all spam. What we
should be aiming for is more specific: we should attempt to design a system
where spam is manageable. Eg if our goal is to ensure that a bitcoin node
uses no more than 10% of the bandwidth of a "normal" user, if current
non-spam traffic only requires 1% of a "normal" users's bandwidth, then the
network can bear a 9 to 1 ratio of spam. When a node spins up, there is a
lot more data to download and process. So we know that all full nodes can
handle at least as much traffic as they handle during IBD. What's the
difference between those amounts? I'm not sure, but I would guess that IBD
is at least a couple times more demanding than a fully synced node. So I
might suggest that as long as spam can be kept below a ratio of maybe 2 to
1, we should consider the design acceptable (and therefore more complexity
unnecessary).

The 1 minute broadcast cooldown I mentioned before wouldn't be quite
sufficient to achieve that ratio. But a 3.33 minute cooldown would be.
Whether this is "too much" is something that would have to be discussed, I
suspect a worst-case adversarial 3.33 minute delay would not be "too much".
Doing this could basically eliminate any risk of actual service denial via
replacement transactions.

However, I do think that these DOS concerns are quite overblown. I wrote up a
comment on your rbf-improvements.md
<https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4093100#gistcomment-4093100>
detailing
my thought process on that. The summary is that as long as the fee-rate
relay rule is maintained, any "spam" is actually paid for, either by the
"first" transaction in the spam chain, or by the "spam" itself. Even
without something like a minimum RBF relay delay limiting how much spam
could be created, the economics of the fee-rate rule already sufficiently
mitigate the issue of spam.
On Wed, Mar 9, 2022 at 9:37 AM Gloria Zhao via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi RBF friends,
>
> Posting a summary of RBF discussions at coredev (mostly on transaction
> relay rate-limiting), user-elected descendant limit as a short term
> solution to unblock package RBF, and mining score, all open for feedback:
>
> One big concept discussed was baking DoS protection into the p2p level
> rather than policy level. TLDR: The fees are not paid to the node operator,
> but to the miner. While we can use fees to reason about the cost of an
> attack, if we're ultimately interested in preventing resource exhaustion,
> maybe we want to "stop the bleeding" when it happens and bound the amount
> of resources used in general. There were two main ideas:
>
> 1. Transaction relay rate limiting (i.e. the one you proposed above or
> some variation) with a feerate-based priority queue
> 2. Staggered broadcast of replacement transactions: within some time
> interval, maybe accept multiple replacements for the same prevout, but only
> relay the original transaction.
>
> Looking to solicit feedback on these ideas and the concept in general. Is
> it a good idea (separate from RBF) to add rate-limiting in transaction
> relay? And is it the right direction to think about RBF DoS protection this
> way?
>
> A lingering concern that I have about this idea is it would then be
> possible to impact the propagation of another person’s transaction, i.e.,
> an attacker can censor somebody’s transaction from ever being announced by
> a node if they send enough transactions to fill up the rate limit.
> Obviously this would be expensive since they're spending a lot on fees, but
> I imagine it could be profitable in some situations to spend a few thousand
> dollars to prevent anyone from hearing about a transaction for a few hours.
> This might be a non-issue in practice if the rate limit is generous and
> traffic isn’t horrendous, but is this a problem?
>
> And if we don't require an increase in (i.e. addition of "new") absolute
> fees, users are essentially allowed to “recycle” fees. In the scenario
> where we prioritize relay based on feerate, users could potentially be
> placed higher in the queue, ahead of other users’ transactions, multiple
> times, without ever adding more fees to the transaction. Again, maybe this
> isn’t a huge deal in practice if we set the parameters right, but it seems…
> not great, in principle.
>
> ---------
>
> It's probably also a good idea to point out that there's been some
> discussion happening on the gist containing my original post on this thread
> (https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).
>
> Suhas and Matt [proposed][0] adding a policy rule allowing users to
> specify descendant limits on their transactions. For example, some nth bit
> of nSequence with nVersion 3 means "this transaction won't have more than X
> vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
> solves the pinning problem with package RBF where the attacker's package
> contains a very large and high-fee descendant.
>
> We could add this policy and deploy it with package RBF/package relay so
> that LN can use it by setting the user-elected descendant limit flag on
> commitment transactions. (Otherwise package RBF is blocked until we find a
> more comprehensive solution to the pinning attack).
>
> It's simple to [implement][1] as a mempool policy, but adds some
> complexity for wallets that use it, since it limits their use of UTXOs from
> transactions with this bit set.
>
> ---------
>
> Also, coming back to the idea of "we can't just use {individual, ancestor}
> feerate," I'm interested in soliciting feedback on adding a “mining score”
> calculator. I've implemented one [here][2] which takes the transaction in
> question, grabs all of the connected mempool transactions (including
> siblings, coparents, etc., as they wouldn’t be in the ancestor nor
> descendant sets), and builds a “block template” using our current mining
> algorithm. The mining score of a transaction is the ancestor feerate at
> which it is included.
>
> This would be helpful for something like ancestor-aware funding and
> fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
> priority queue for transaction relay, we'd want to use something like this
> as the priority value. And for RBF, we probably want to require that a
> replacement have a higher mining score than the original transactions. This
> could be computationally expensive to do all the time; it could be good to
> cache it but that could make mempool bookkeeping more complicated. Also, if
> we end up trying to switch to a candidate set-based algorithm for mining,
> we'd of course need a new calculator.
>
> [0]:
> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4058140#gistcomment-4058140
> [1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
> [2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
> [3]: https://github.com/bitcoin/bitcoin/issues/9645
> [4]: https://github.com/bitcoin/bitcoin/issues/15553
>
> Best,
> Gloria
>
> On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:
>
>> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
>> > @aj:
>> > > I wonder sometimes if it could be sufficient to just have a relay rate
>> > > limit and prioritise by ancestor feerate though. Maybe something like:
>> > > - instead of adding txs to each peers setInventoryTxToSend
>> immediately,
>> > >   set a mempool flag "relayed=false"
>> > > - on a time delay, add the top N (by fee rate) "relayed=false" txs to
>> > >   each peer's setInventoryTxToSend and mark them as "relayed=true";
>> > >   calculate how much kB those txs were, and do this again after
>> > >   SIZE/RATELIMIT seconds
>>
>> > > - don't include "relayed=false" txs when building blocks?
>>
>> The "?" was me not being sure that point is a good suggestion...
>>
>> Miners might reasonably decide to have no rate limit, and always relay,
>> and never exclude txs -- but the question then becomes is whether they
>> hear about the tx at all, so rate limiting behaviour could still be a
>> potential problem for whoever made the tx.
>>
>> > Wow cool! I think outbound tx relay size-based rate-limiting and
>> > prioritizing tx relay by feerate are great ideas for preventing spammers
>> > from wasting bandwidth network-wide. I agree, this would slow the low
>> > feerate spam down, preventing a huge network-wide bandwidth spike. And
>> it
>> > would allow high feerate transactions to propagate as they should,
>> > regardless of how busy traffic is. Combined with inbound tx request
>> > rate-limiting, might this be sufficient to prevent DoS regardless of the
>> > fee-based replacement policies?
>>
>> I think you only want to do outbound rate limits, ie, how often you send
>> INV, GETDATA and TX messages? Once you receive any of those, I think
>> you have to immediately process / ignore it, you can't really sensibly
>> defer it (beyond the existing queues we have that just build up while
>> we're busy processing other things first)?
>>
>> > One point that I'm not 100% clear on: is it ok to prioritize the
>> > transactions by ancestor feerate in this scheme? As I described in the
>> > original post, this can be quite different from the actual feerate we
>> would
>> > consider a transaction in a block for. The transaction could have a high
>> > feerate sibling bumping its ancestor.
>> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB).
>> If
>> > we just received C, it would be incorrect to give it a priority equal to
>> > its ancestor feerate (3sat/vB) because if we constructed a block
>> template
>> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
>> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If we
>> > also exclude C when building blocks, we're missing out on good fees.
>>
>> I think you're right that this would be ugly. It's something of a
>> special case:
>>
>>  a) you really care about C getting into the next block; but
>>  b) you're trusting B not being replaced by a higher fee tx that
>>     doesn't have A as a parent; and
>>  c) there's a lot of txs bidding the floor of the next block up to a
>>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>>     rate of 5sat/vB
>>
>> Without (a), maybe you don't care about it getting to a miner quickly.
>> If your trust in (b) was misplaced, then your tx's effective fee rate
>> will drop and (because of (c)), you'll lose anyway. And if the spam ends
>> up outside of (c)'s range, either the rate limiting won't take effect
>> (spam's too cheap) and you'll be fine, or you'll miss out on the block
>> anyway (spam's paying more than your tx rate) and you never had any hope
>> of making it in.
>>
>> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
>> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
>> per 10 minutes for outbound connections. This would be a weight based
>> rate limit instead-of/in-addition-to that, I guess.
>>
>> As far as a non-ugly approach goes, I think you'd have to be smarter about
>> tracking the "effective fee rate" than the ancestor fee rate manages;
>> maybe that's something that could fall out of Murch and Clara's candidate
>> set blockbuilding ideas [0] ?
>>
>> Perhaps that same work would also make it possible to come up with
>> a better answer to "do I care that this replacement would invalidate
>> these descendents?"
>>
>> [0] https://github.com/Xekyo/blockbuilding
>>
>> > > - keep high-feerate evicted txs around for a while in case they get
>> > >   mined by someone else to improve compact block relay, a la the
>> > >   orphan pool?
>> > Replaced transactions are already added to vExtraTxnForCompact :D
>>
>> I guess I was thinking that it's just a 100 tx LRU cache, which might
>> not be good enough?
>>
>> Maybe it would be more on point to have a rate limit apply only to
>> replacement transactions?
>>
>> > For wallets, AJ's "All you need is for there to be *a* path that follows
>> > the new relay rules and gets from your node/wallet to perhaps 10% of
>> > hashpower" makes sense to me (which would be the former).
>>
>> Perhaps a corollarly of that is that it's *better* to have the mempool
>> acceptance rule only consider economic incentives, and have the spam
>> prevention only be about "shall I tell my peers about this?"
>>
>> If you don't have that split; then the anti-spam rules can prevent you
>> from getting the tx in the mempool at all; whereas if you do have the
>> split, then even if the bitcoind anti-spam rules are blocking you at
>> every turn, you can still send your tx to miners by some other route,
>> and then they can add it to their mempool directly without any hassle.
>>
>> Cheers,
>> aj
>>
>> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 19343 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Gloria Zhao]

[-- Attachment #1: Type: text/plain, Size: 10750 bytes --]

Hi RBF friends,

Posting a summary of RBF discussions at coredev (mostly on transaction
relay rate-limiting), user-elected descendant limit as a short term
solution to unblock package RBF, and mining score, all open for feedback:

One big concept discussed was baking DoS protection into the p2p level
rather than policy level. TLDR: The fees are not paid to the node operator,
but to the miner. While we can use fees to reason about the cost of an
attack, if we're ultimately interested in preventing resource exhaustion,
maybe we want to "stop the bleeding" when it happens and bound the amount
of resources used in general. There were two main ideas:

1. Transaction relay rate limiting (i.e. the one you proposed above or some
variation) with a feerate-based priority queue
2. Staggered broadcast of replacement transactions: within some time
interval, maybe accept multiple replacements for the same prevout, but only
relay the original transaction.

Looking to solicit feedback on these ideas and the concept in general. Is
it a good idea (separate from RBF) to add rate-limiting in transaction
relay? And is it the right direction to think about RBF DoS protection this
way?

A lingering concern that I have about this idea is it would then be
possible to impact the propagation of another person’s transaction, i.e.,
an attacker can censor somebody’s transaction from ever being announced by
a node if they send enough transactions to fill up the rate limit.
Obviously this would be expensive since they're spending a lot on fees, but
I imagine it could be profitable in some situations to spend a few thousand
dollars to prevent anyone from hearing about a transaction for a few hours.
This might be a non-issue in practice if the rate limit is generous and
traffic isn’t horrendous, but is this a problem?

And if we don't require an increase in (i.e. addition of "new") absolute
fees, users are essentially allowed to “recycle” fees. In the scenario
where we prioritize relay based on feerate, users could potentially be
placed higher in the queue, ahead of other users’ transactions, multiple
times, without ever adding more fees to the transaction. Again, maybe this
isn’t a huge deal in practice if we set the parameters right, but it seems…
not great, in principle.

---------

It's probably also a good idea to point out that there's been some
discussion happening on the gist containing my original post on this thread
(https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).

Suhas and Matt [proposed][0] adding a policy rule allowing users to specify
descendant limits on their transactions. For example, some nth bit of
nSequence with nVersion 3 means "this transaction won't have more than X
vbytes of descendants" where X = max(1000, vsizeof(tx)) or something. It
solves the pinning problem with package RBF where the attacker's package
contains a very large and high-fee descendant.

We could add this policy and deploy it with package RBF/package relay so
that LN can use it by setting the user-elected descendant limit flag on
commitment transactions. (Otherwise package RBF is blocked until we find a
more comprehensive solution to the pinning attack).

It's simple to [implement][1] as a mempool policy, but adds some complexity
for wallets that use it, since it limits their use of UTXOs from
transactions with this bit set.

---------

Also, coming back to the idea of "we can't just use {individual, ancestor}
feerate," I'm interested in soliciting feedback on adding a “mining score”
calculator. I've implemented one [here][2] which takes the transaction in
question, grabs all of the connected mempool transactions (including
siblings, coparents, etc., as they wouldn’t be in the ancestor nor
descendant sets), and builds a “block template” using our current mining
algorithm. The mining score of a transaction is the ancestor feerate at
which it is included.

This would be helpful for something like ancestor-aware funding and
fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
priority queue for transaction relay, we'd want to use something like this
as the priority value. And for RBF, we probably want to require that a
replacement have a higher mining score than the original transactions. This
could be computationally expensive to do all the time; it could be good to
cache it but that could make mempool bookkeeping more complicated. Also, if
we end up trying to switch to a candidate set-based algorithm for mining,
we'd of course need a new calculator.

[0]:
https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=4058140#gistcomment-4058140
[1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
[2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
[3]: https://github.com/bitcoin/bitcoin/issues/9645
[4]: https://github.com/bitcoin/bitcoin/issues/15553

Best,
Gloria

On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:

> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
> > @aj:
> > > I wonder sometimes if it could be sufficient to just have a relay rate
> > > limit and prioritise by ancestor feerate though. Maybe something like:
> > > - instead of adding txs to each peers setInventoryTxToSend immediately,
> > >   set a mempool flag "relayed=false"
> > > - on a time delay, add the top N (by fee rate) "relayed=false" txs to
> > >   each peer's setInventoryTxToSend and mark them as "relayed=true";
> > >   calculate how much kB those txs were, and do this again after
> > >   SIZE/RATELIMIT seconds
>
> > > - don't include "relayed=false" txs when building blocks?
>
> The "?" was me not being sure that point is a good suggestion...
>
> Miners might reasonably decide to have no rate limit, and always relay,
> and never exclude txs -- but the question then becomes is whether they
> hear about the tx at all, so rate limiting behaviour could still be a
> potential problem for whoever made the tx.
>
> > Wow cool! I think outbound tx relay size-based rate-limiting and
> > prioritizing tx relay by feerate are great ideas for preventing spammers
> > from wasting bandwidth network-wide. I agree, this would slow the low
> > feerate spam down, preventing a huge network-wide bandwidth spike. And it
> > would allow high feerate transactions to propagate as they should,
> > regardless of how busy traffic is. Combined with inbound tx request
> > rate-limiting, might this be sufficient to prevent DoS regardless of the
> > fee-based replacement policies?
>
> I think you only want to do outbound rate limits, ie, how often you send
> INV, GETDATA and TX messages? Once you receive any of those, I think
> you have to immediately process / ignore it, you can't really sensibly
> defer it (beyond the existing queues we have that just build up while
> we're busy processing other things first)?
>
> > One point that I'm not 100% clear on: is it ok to prioritize the
> > transactions by ancestor feerate in this scheme? As I described in the
> > original post, this can be quite different from the actual feerate we
> would
> > consider a transaction in a block for. The transaction could have a high
> > feerate sibling bumping its ancestor.
> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB). If
> > we just received C, it would be incorrect to give it a priority equal to
> > its ancestor feerate (3sat/vB) because if we constructed a block template
> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If we
> > also exclude C when building blocks, we're missing out on good fees.
>
> I think you're right that this would be ugly. It's something of a
> special case:
>
>  a) you really care about C getting into the next block; but
>  b) you're trusting B not being replaced by a higher fee tx that
>     doesn't have A as a parent; and
>  c) there's a lot of txs bidding the floor of the next block up to a
>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>     rate of 5sat/vB
>
> Without (a), maybe you don't care about it getting to a miner quickly.
> If your trust in (b) was misplaced, then your tx's effective fee rate
> will drop and (because of (c)), you'll lose anyway. And if the spam ends
> up outside of (c)'s range, either the rate limiting won't take effect
> (spam's too cheap) and you'll be fine, or you'll miss out on the block
> anyway (spam's paying more than your tx rate) and you never had any hope
> of making it in.
>
> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
> per 10 minutes for outbound connections. This would be a weight based
> rate limit instead-of/in-addition-to that, I guess.
>
> As far as a non-ugly approach goes, I think you'd have to be smarter about
> tracking the "effective fee rate" than the ancestor fee rate manages;
> maybe that's something that could fall out of Murch and Clara's candidate
> set blockbuilding ideas [0] ?
>
> Perhaps that same work would also make it possible to come up with
> a better answer to "do I care that this replacement would invalidate
> these descendents?"
>
> [0] https://github.com/Xekyo/blockbuilding
>
> > > - keep high-feerate evicted txs around for a while in case they get
> > >   mined by someone else to improve compact block relay, a la the
> > >   orphan pool?
> > Replaced transactions are already added to vExtraTxnForCompact :D
>
> I guess I was thinking that it's just a 100 tx LRU cache, which might
> not be good enough?
>
> Maybe it would be more on point to have a rate limit apply only to
> replacement transactions?
>
> > For wallets, AJ's "All you need is for there to be *a* path that follows
> > the new relay rules and gets from your node/wallet to perhaps 10% of
> > hashpower" makes sense to me (which would be the former).
>
> Perhaps a corollarly of that is that it's *better* to have the mempool
> acceptance rule only consider economic incentives, and have the spam
> prevention only be about "shall I tell my peers about this?"
>
> If you don't have that split; then the anti-spam rules can prevent you
> from getting the tx in the mempool at all; whereas if you do have the
> split, then even if the bitcoind anti-spam rules are blocking you at
> every turn, you can still send your tx to miners by some other route,
> and then they can add it to their mempool directly without any hassle.
>
> Cheers,
> aj
>
>

[-- Attachment #2: Type: text/html, Size: 12551 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[lisa neigut]

[-- Attachment #1: Type: text/plain, Size: 7417 bytes --]

Changing the way that RBF works is an excellent idea. Bitcoin is overdue
for revisiting these rules. Thanks to @glowzo et al for kicking off the
discussion.

I've been thinking about RBF for a very long time[1], and it's been fun to
see other people's thoughts on the topics. Here's my current thinking about
this, and proposal for how we should update the rules.

### Changing the Rules? How to Change Them
Regarding how to change them, Bram and aj are right -- we should move to a
model where transaction relay candidates are evaluated on their net
increase in fees per byte paid, and remove the requirement that the gross
fee of the preceding transaction is met or exceeded.

Our current ruleset is over complicated because it attempts to solve two
problems at once: the miner's best interests (highest possible fee take)
and relay policy.

I believe this is a mistake and the mempool should change its goals.
Instead, the mempool relay design for RBFs should be built around 1)
increasing the the per-byte fees paid of a transaction and 2) providing a
simple policy for applications building on top of bitcoin, such that
knowledge of the mempool is not required for successfully issuing
relay-able RBF transactions.

(A simple "must increase the per-byte feerate" heuristic for RBF relay
candidates has the nice benefit of being easy to program to on the
application side, and only requires knowledge of the previous candidate
transaction, not the entire mempool or any previous tx's relative position
within it.)

Finally, when blockspace is competitive , this simple policy ensures that
the per-byte value of every subsequent relayed transaction increases the
per-byte value of pending bytes for the next block. This provides a measure
of DoS protection and ensures that every relayed byte is more valuable (to
the miner/network) than the last.

*The only time that RBF is critical for relay is during full block periods
-- if there's not enough transactions to fill a block, using RBF to ensure
that a transaction is mined in a timely manner is moot. As such, RBF rules
should not be concerned with low-block environments.

### Mempools and Relay
The mempool currently serves two masters: the profit motive of the miner
and the relay motive of a utxo holder. It is in the interest of a user to
send the miner a high per-byte tx, such that it might end up in the next
block. It is in the miner's best interest to include the highest per-byte
set of transactions in their block template.

There is some conflation here in the current RBF policies between what is
in the mempool and what is considered a candidate for the block template.
If a miner has already included a more profitable package of txs into their
block template than a more valuable per-byte tx that the network has
relayed to them, it should be the responsibility of the block template
constructor to reject the new proposed tx, not the nodes relaying the
transaction to said miner.

This is a policy that the miner can (and should) implement at the level of
the template construction, however.

Is it the responsibility of the mempool to provide the best "historical"
block opportunity for a miner (e.g. the highest paying block given all txs
it's ever seen)? I would say no, that the ability of a utxo owner to
re-state the spend condition of a pending transaction is more important,
from a use-case perspective, and that the mempool should concern itself
solely with relaying increasingly more profitable bytes to miners. Let the
miners concern themselves with deciding what the best policy for their own
block construction is, and the mempool with relaying the highest value
bytes for the network. Net-net, this will benefit everyone as it becomes
easier for users to re-submit txs with increasingly greater feerates,
creating more active competition for available blockspace as more
applications are able to include it as a feature (and it works, reliable,
as a relay mechanism).

### Packages and RBF
Packages make the increasing per-byte rule less of a guarantee that
increasing the per-byte value of a single transaction will net a given
miner more fees than including the entire package.

Let's decompose this a bit. It's helpful to think of tx packages as
'composable txs'. Basically when you consider a 'package' it is actually a
large tx with sub-components, the individual txs. As a 'composed tx', you
can calculate the per-byte feerate of the entire set. This is the number
that you, as someone issuing an RBF, would need to beat in order to move
your tx up in the pending block queue.

RBF, however, is a transaction level policy: it allows you to replace any
*one* component of a package, or tree, with the side effect of possibly
invalidating other candidate txs. If the 'composed tx' (aka package) had a
net per-byte value that was higher than the new replacement transaction
because of a leaf tx that had an outsized per-byte feerate, then it would
be more profitable for the miner to have mined the entire package rather
than the replacement.

This edge case complicates the picture for the miner. Considered from the
viewpoint of the user issuing the RBF, however, it is far simpler. In the
ideal case, a person is issuing an RBF because the previous tx tree, even
with its high fee sub-component, was not a candidate for the next block.
And, in some cases, increasing the sub-component's per-byte feerate will
not achieve the goal of moving the tx any closer to being mined. It's only
by increasing the feerate above the present feerate of the candidate plus
desendents (tx package) that the transaction will advance in the queue.

While not uncomplicated, this is a simple enough metric for a wallet to
track, and does not require any knowledge of the current mempool to
effectively surpass. It's the wallet's responsibility to track this though;
failure to take descendants into account when deciding on the next per-byte
feerate for an RBF *will* mean that your RBF will be ineffective at
achieving the goal of getting your UTXO spent. Any wallet is incentivized
to always provide a higher per-byte feerate than the 'composed tx' (tx and
its descendants), so as to ensure an actual improvement in the unconfirmed
transaction's position in the block queue, so to speak.

Note that the v2 protocol for channel opens in lightning uses an RBF
negotiation that adheres basically to these rules (ea transaction must have
a strictly greater per-byte feerate).

We enforce a rate of 65/64th as the required increase in feerate for each
subsequent channel open transaction.

https://github.com/lightning/bolts/pull/851/files#diff-ed04ca2c673fd6aabde69389511fa9ee60cb44d6b2ef6c88b549ffaa753d6afeR1154

### RBF and DoS

Admittedly, changing these rules will increase the number of times that any
UTXO is eligible to be retransmitted (relayed) via the bitcoin node
network. Strictly increasing the per-byte feerate however ensures that this
re-relay is increasingly more expensive to the UTXO owner, however.


### in exitus
These are the things I've been thinking about with regards to RBF. I hope
they can help to highlight the challenges in the RBF design space a bit
more clearly, as well as spelling out the case for using a simple heuristic
such as "solely increasing per-byte feerate" as a good candidate for the
revised RBF policy.

~niftynei

[1]
https://basicbitch.software/posts/2018-12-27-Explaining-Replace-By-Fee.html

[-- Attachment #2: Type: text/html, Size: 15920 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Anthony Towns]

On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
> @aj:
> > I wonder sometimes if it could be sufficient to just have a relay rate
> > limit and prioritise by ancestor feerate though. Maybe something like:
> > - instead of adding txs to each peers setInventoryTxToSend immediately,
> >   set a mempool flag "relayed=false"
> > - on a time delay, add the top N (by fee rate) "relayed=false" txs to
> >   each peer's setInventoryTxToSend and mark them as "relayed=true";
> >   calculate how much kB those txs were, and do this again after
> >   SIZE/RATELIMIT seconds

> > - don't include "relayed=false" txs when building blocks?

The "?" was me not being sure that point is a good suggestion...

Miners might reasonably decide to have no rate limit, and always relay,
and never exclude txs -- but the question then becomes is whether they
hear about the tx at all, so rate limiting behaviour could still be a
potential problem for whoever made the tx.

> Wow cool! I think outbound tx relay size-based rate-limiting and
> prioritizing tx relay by feerate are great ideas for preventing spammers
> from wasting bandwidth network-wide. I agree, this would slow the low
> feerate spam down, preventing a huge network-wide bandwidth spike. And it
> would allow high feerate transactions to propagate as they should,
> regardless of how busy traffic is. Combined with inbound tx request
> rate-limiting, might this be sufficient to prevent DoS regardless of the
> fee-based replacement policies?

I think you only want to do outbound rate limits, ie, how often you send
INV, GETDATA and TX messages? Once you receive any of those, I think
you have to immediately process / ignore it, you can't really sensibly
defer it (beyond the existing queues we have that just build up while
we're busy processing other things first)?

> One point that I'm not 100% clear on: is it ok to prioritize the
> transactions by ancestor feerate in this scheme? As I described in the
> original post, this can be quite different from the actual feerate we would
> consider a transaction in a block for. The transaction could have a high
> feerate sibling bumping its ancestor.
> For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB). If
> we just received C, it would be incorrect to give it a priority equal to
> its ancestor feerate (3sat/vB) because if we constructed a block template
> now, B would bump A, and C's new ancestor feerate is 5sat/vB.
> Then, if we imagine that top N is >5sat/vB, we're not relaying C. If we
> also exclude C when building blocks, we're missing out on good fees.

I think you're right that this would be ugly. It's something of a
special case:

 a) you really care about C getting into the next block; but
 b) you're trusting B not being replaced by a higher fee tx that
    doesn't have A as a parent; and
 c) there's a lot of txs bidding the floor of the next block up to a
    level in-between the ancestor fee rate of 3sat/vB and the tx fee
    rate of 5sat/vB

Without (a), maybe you don't care about it getting to a miner quickly.
If your trust in (b) was misplaced, then your tx's effective fee rate
will drop and (because of (c)), you'll lose anyway. And if the spam ends
up outside of (c)'s range, either the rate limiting won't take effect
(spam's too cheap) and you'll be fine, or you'll miss out on the block
anyway (spam's paying more than your tx rate) and you never had any hope
of making it in.

Note that we already rate limit via INVENTORY_BROADCAST_MAX /
*_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs
per 10 minutes for outbound connections. This would be a weight based
rate limit instead-of/in-addition-to that, I guess.

As far as a non-ugly approach goes, I think you'd have to be smarter about
tracking the "effective fee rate" than the ancestor fee rate manages;
maybe that's something that could fall out of Murch and Clara's candidate
set blockbuilding ideas [0] ?

Perhaps that same work would also make it possible to come up with
a better answer to "do I care that this replacement would invalidate
these descendents?"

[0] https://github.com/Xekyo/blockbuilding

> > - keep high-feerate evicted txs around for a while in case they get
> >   mined by someone else to improve compact block relay, a la the
> >   orphan pool?
> Replaced transactions are already added to vExtraTxnForCompact :D

I guess I was thinking that it's just a 100 tx LRU cache, which might
not be good enough?

Maybe it would be more on point to have a rate limit apply only to
replacement transactions?

> For wallets, AJ's "All you need is for there to be *a* path that follows
> the new relay rules and gets from your node/wallet to perhaps 10% of
> hashpower" makes sense to me (which would be the former).

Perhaps a corollarly of that is that it's *better* to have the mempool
acceptance rule only consider economic incentives, and have the spam
prevention only be about "shall I tell my peers about this?"

If you don't have that split; then the anti-spam rules can prevent you
from getting the tx in the mempool at all; whereas if you do have the
split, then even if the bitcoind anti-spam rules are blocking you at
every turn, you can still send your tx to miners by some other route,
and then they can add it to their mempool directly without any hassle.

Cheers,
aj

Re: [bitcoin-dev] Improving RBF Policy

[Gloria Zhao]

[-- Attachment #1: Type: text/plain, Size: 49589 bytes --]

Hi everyone,

Thanks for giving your attention to the post! I haven't had time to write
responses to everything, but sending my thoughts about what has been most
noteworthy to me:

@jeremy:
> A final point is that a verifiable delay function could be used over,
e.g., each of the N COutpoints individually to rate-limit transaction
replacement. The VDF period can be made shorter / eliminated depending on
the feerate increase.

Thanks for the suggestion! In general, I don't think rate limiting by
outpoint/prevout is a safe option, as it is particularly dangerous for L2
applications with shared prevouts. For example, the prevout that LN channel
counterparties conflict on is the output from their shared funding tx. Any
kind of limit on spending this prevout can be monopolized by a spammy
attacker. For example, if you only allow 1 per minute, the attacker will
just race to take up that slot every minute to prevent the honest party's
transaction from being accepted.
This is similar to the pinning attack based on monopolizing the
transaction's descendant limit, except we can't carve out an exemption
because we wouldn't know whose replacement we're looking at.

@tbast:
> The way I understand it, limiting the impact on descendant transactions
is only important for DoS protection, not for incentive compatibility.

> I believe it's completely ok to require increasing both the fees and
feerate if we don't take descendants into account, because you control your
ancestor set - whereas the descendant set may be completely out of your
control.

Ignoring descendants of direct conflicts would certainly make our lives
much easier! Unfortunately, I don't think we can do this since they can be
fee bumps, i.e., in AJ's example. Considering descendants is important for
both incentive compatibility and DoS.
If the replacement transaction has a higher feerate than its direct
conflict, but the direct conflict also has high feerate descendants, we
might end up with lower fees and/or feerates by accepting the replacement.

@aj:
> I wonder sometimes if it could be sufficient to just have a relay rate
limit and prioritise by ancestor feerate though. Maybe something like:
>
> - instead of adding txs to each peers setInventoryTxToSend immediately,
>   set a mempool flag "relayed=false"
>
> - on a time delay, add the top N (by fee rate) "relayed=false" txs to
>   each peer's setInventoryTxToSend and mark them as "relayed=true";
>   calculate how much kB those txs were, and do this again after
>   SIZE/RATELIMIT seconds
>
> - don't include "relayed=false" txs when building blocks?

Wow cool! I think outbound tx relay size-based rate-limiting and
prioritizing tx relay by feerate are great ideas for preventing spammers
from wasting bandwidth network-wide. I agree, this would slow the low
feerate spam down, preventing a huge network-wide bandwidth spike. And it
would allow high feerate transactions to propagate as they should,
regardless of how busy traffic is. Combined with inbound tx request
rate-limiting, might this be sufficient to prevent DoS regardless of the
fee-based replacement policies?

One point that I'm not 100% clear on: is it ok to prioritize the
transactions by ancestor feerate in this scheme? As I described in the
original post, this can be quite different from the actual feerate we would
consider a transaction in a block for. The transaction could have a high
feerate sibling bumping its ancestor.
For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB). If
we just received C, it would be incorrect to give it a priority equal to
its ancestor feerate (3sat/vB) because if we constructed a block template
now, B would bump A, and C's new ancestor feerate is 5sat/vB.
Then, if we imagine that top N is >5sat/vB, we're not relaying C. If we
also exclude C when building blocks, we're missing out on good fees.

> - keep high-feerate evicted txs around for a while in case they get
>   mined by someone else to improve compact block relay, a la the
>   orphan pool?

Replaced transactions are already added to vExtraTxnForCompact :D

@ariard
> Deployment of Taproot opens interesting possibilities in the
vaults/payment channels design space, where the tapscripts can commit to
different set of timelocks/quorum of keys. Even if the pre-signed states
stay symmetric, whoever is the publisher, the feerate cost to spend can
fluctuate.

Indeed, perhaps with taproot we may legitimately have
same-txid-different-witness transactions as a normal thing rather than rare
edge case. But as with everything enabled by taproot, I wouldn't count our
tapscript eggs until a concrete use case hatches and/or an application
actually implements it.

> How this new replacement rule would behave if you have a parent in the
"replace-by-feerate" half but the child is in the "replace-by-fee" one ?

Thanks for considering my suggestion! This particular scenario is not
possible, since a child cannot be considered for the next block without its
parent. But if the original transactions are found both in and outside the
next block, I think it would be fine to just require both are met.

> Overall, I think there is the deployment issue to warn of. Moving to a
new set of RBF rules implies for a lot of Bitcoin applications to rewrite
their RBF logics.

I agree that transitioning as painlessly as possible would be a huge
priority in any kind of upgrade to mempool policy. I'm very interested in
hearing wallet devs' feedback on this.
I'm also not actually clear on what backwards compatibility in this
scenario would look like. I imagine it to mean we run both sets of RBF
rules and accept the replacement if it passes either one. Or do we only
accept the replacement if it passes both?
For wallets, AJ's "All you need is for there to be *a* path that follows
the new relay rules and gets from your node/wallet to perhaps 10% of
hashpower" makes sense to me (which would be the former). For merchants who
care more about making sure the original transaction isn't replaceable,
would they prefer that either policy is sufficient to prevent a replacement
(more in line with the latter)? Or is that covered by signaling / am I
overthinking this?

Thanks,
Gloria

On Mon, Feb 7, 2022 at 10:24 AM Bastien TEINTURIER via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Good morning,
>
> > The tricky question is what happens when X arrives on its own and it
> > might be that no one ever sends a replacement for B,C,D)
>
> It feels ok to me, but this is definitely arguable.
>
> It covers the fact that B,C,D could have been fake transactions whose
> sole purpose was to do a pinning attack: in that case the attacker would
> have found a way to ensure these transactions don't confirm anyway (or
> pay minimal/negligible fees).
>
> If these transactions were legitimate, I believe that their owners would
> remake them at some point (because these transactions reflect a business
> relationship that needed to happen, so it should very likely still
> happen). It's probably hard to verify because the new corresponding
> transactions may have nothing in common with the first, but I think the
> simplifications it offers for wallets is worth it (which is just my
> opinion and needs more scrutiny/feedback).
>
> > But if your backlog's feerate does drop off, *and* that matters, then
> > I don't think you can ignore the impact of the descendent transactions
> > that you might not get a replacement for.
>
> That is because you're only taking into account the current backlog, and
> not taking into account the fact that new items will be added to it soon
> to replace the evicted descendants. But I agree that this is a bet: we
> can't predict the future and guarantee these replacements will come.
>
> It is really a trade-off, ignoring descendents provides a much simpler
> contract that doesn't vary from one mempool to another, but when your
> backlog isn't full enough, you may lose some future profits if
> transactions don't come in later.
>
> > I think "Y% higher" rather than just "higher" is only useful for
> > rate-limiting, not incentive compatibility. (Though maybe it helps
> > stabilise a greedy algorithm in some cases?)
>
> That's true. I claimed these policies only address incentives, but using
> a percentage increase addresses rate-limiting a bit as well (I couldn't
> resist trying to do at least something for it!). I find it a very easy
> mechanism to implement, while choosing an absolute value is hard (it's
> always easier to think in relatives than absolutes).
>
> > This is why I think it is important to understand the rationales for
> introducing the rules in the first place
>
> I completely agree. As you mentioned, we are still in brainstorming
> phase, once (if?) we start to converge on what could be better policies,
> we do need to clearly explain each policy's expected goal. That will let
> future Bastien writing code in 2030 clearly highlight why the 2022 rules
> don't make sense anymore!
>
> Cheers,
> Bastien
>
> Le sam. 5 févr. 2022 à 14:22, Michael Folkson <
> michaelfolkson@protonmail.com> a écrit :
>
>> Thanks for this Bastien (and Gloria for initially posting about this).
>>
>> I sympathetically skimmed the eclair PR (
>> https://github.com/ACINQ/eclair/pull/2113) dealing with replaceable
>> transactions fee bumping.
>>
>> There will continue to be a (hopefully) friendly tug of war on this
>> probably for the rest of Bitcoin's existence. I am sure people like Luke,
>> Prayank etc will (rightfully) continue to raise that Lightning and other
>> second layer protocols shouldn't demand that policy rules be changed if
>> there is a reason (e.g. DoS vector) for those rules on the base network.
>> But if there are rules that have no upside, introduce unnecessary
>> complexity for no reason and make Lightning implementers like Bastien's
>> life miserable attempting to deal with them I really hope we can make
>> progress on removing or simplifying them.
>>
>> This is why I think it is important to understand the rationales for
>> introducing the rules in the first place (and why it is safe to remove them
>> if indeed it is) and being as rigorous as possible on the rationales for
>> introducing additional rules. It sounds like from Gloria's initial post we
>> are still at a brainstorming phase (which is fine) but knowing what we know
>> today I really hope we can learn from the mistakes of the original BIP 125,
>> namely the Core implementation not matching the BIP and the sparse
>> rationales for the rules. As Bastien says this is not criticizing the
>> original BIP 125 authors, 7 years is a long time especially in Bitcoin
>> world and they probably weren't thinking about Bastien sitting down to
>> write an eclair PR in late 2021 (and reviewers of that PR) when they wrote
>> the BIP in 2015.
>>
>> --
>> Michael Folkson
>> Email: michaelfolkson at protonmail.com
>> Keybase: michaelfolkson
>> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>>
>>
>>
>> ------- Original Message -------
>> On Monday, January 31st, 2022 at 3:57 PM, Bastien TEINTURIER via
>> bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> Hi Gloria,
>>
>> Many thanks for raising awareness on these issues and constantly pushing
>> towards finding a better model. This work will highly improve the
>> security of any multi-party contract trying to build on top of bitcoin
>> (because most multi-party contracts will need to have timeout conditions
>> and participants will need to make some transactions confirm before a
>> timeout happens - otherwise they may lose funds).
>>
>> For starters, let me quickly explain why the current rules are hard to
>> work with in the context of lightning (but I believe most L2 protocols
>> will have the same issues). Feel free to skip this part if you are
>> already convinced.
>>
>> ## Motivation
>>
>> The biggest pain point is BIP 125 rule 2.
>> If I need to increase the fees of a time-sensitive transaction because
>> the feerate has been rising since I broadcast it, I may need to also pay
>> high fees just to produce a confirmed utxo that I can use. I'm actually
>> paying a high fee twice instead of once (and needlessly using on-chain
>> space, our scarcest asset, because we could have avoided that additional
>> transaction!).
>>
>> It also has some annoying "non-determinism".
>> Imagine that my transaction has been evicted from my mempool because its
>> feerate was too low. I could think "Great, that means I don't have to
>> apply BIP 125 restrictions, I can just fund this transaction as if it
>> were a new one!". But actually I do, because my transaction could still
>> be in miner's mempools and I have no way of knowing it...this means that
>> whenever I have broadcast a transaction, I must assume that I will
>> always need to abide by whatever replacement rules the network applies.
>>
>> Fortunately, as far as I understand it, this rule only exists because of
>> a previous implementation detail of bitcoin core, so there's simply no
>> good reason to keep it.
>>
>> The second biggest pain point is rule 3. It prevents me from efficiently
>> using my capital while it's unconfirmed. Whenever I'm using a big utxo
>> to fund a transaction, I will get a big change output, and it would
>> really be a waste to be unable to use that change output to fund other
>> transactions. In order to be capital-efficient, I will end up creating
>> descendant trees for my time-sensitive transactions. But as Gloria
>> explained, replacing all my children will cost me an absurdly large
>> amount of fees. So what I'm actually planning to do instead is to RBF
>> one of the descendants high enough to get the whole tree confirmed.
>> But if those descendants' timeouts were far in the future, that's a
>> waste, I paid a lot more fees for them than I should have. I'd like to
>> just replace my transaction and republish the invalidated children
>> independently.
>>
>> Rule 4 doesn't hurt as much as the two previous ones, I don't have too
>> much to say about it.
>>
>> To be fair to the BIP 125 authors, all of these scenarios were very hard
>> to forecast at the time this BIP was created. We needed years to build
>> on those rules to get a better understanding of their limitations and if
>> the rationale behind them made sense in the long term.
>>
>> ## Proposals
>>
>> I believe that now is a good time to re-think those, and I really like
>> Gloria's categorization of the design constraints.
>>
>> I'd like to propose a different way of looking at descendants that makes
>> it easier to design the new rules. The way I understand it, limiting the
>> impact on descendant transactions is only important for DoS protection,
>> not for incentive compatibility. I would argue that after evictions,
>> descendant transactions will be submitted again (because they represent
>> transactions that people actually want to make), so evicting them does
>> not have a negative impact on mining incentives (in a world where blocks
>> are full most of the time).
>>
>> I'm curious to hear other people's thoughts on that. If it makes sense,
>> I would propose the following very simple rules:
>>
>> 1. The transaction's ancestor absolute fees must be X% higher than the
>> previous transaction's ancestor fees
>> 2. The transaction's ancestor feerate must be Y% higher than the
>> previous transaction's ancestor feerate
>>
>> I believe it's completely ok to require increasing both the fees and
>> feerate if we don't take descendants into account, because you control
>> your ancestor set - whereas the descendant set may be completely out of
>> your control.
>>
>> This is very easy to use by wallets, because the ancestor set is easy to
>> obtain. And an important point is that the ancestor set is the same in
>> every mempool, whereas the descendant set is not (your mempool may have
>> rejected the last descendants, while other people's mempools may still
>> contain them).
>>
>> Because of that reason, I'd like to avoid having a rule that relies on
>> some size of the replaced descendant set: it may be valid in your
>> mempool but invalid in someone else's, which makes it exploitable for
>> pinning attacks.
>>
>> I believe these rules are incentive compatible (again, if you accept
>> the fact that the descendants will be re-submitted and mined as well,
>> so their fees aren't lost).
>>
>> Can we choose X and Y so that these two rules are also DoS-resistant?
>> Unfortunately I'm not sure, so maybe we'll need to add a third rule to
>> address that. But before we do, can someone detail what it costs for a
>> node to evict a descendant tree? Given that bitcoin core doesn't allow
>> chains of more than 25 transactions, the maximum number of transactions
>> being replaced will be bounded by 25 * N (where N is the number of
>> outputs of the transaction being replaced). If it's just O(n) pruning of
>> a graph, maybe that's ok? Or maybe we make X or Y depend on the number
>> of outputs of the transaction being replaced (this would need very
>> careful thoughts)?
>>
>> If you made it this far, thanks for reading!
>> A couple of comments on the previous messages:
>>
>> > Currently, if we see a transaction
>> > that has the same txid as one in the mempool, we reject it as a
>> > duplicate, even if the feerate is much higher. It's unclear to me if
>> > we have a very strong reason to change this, but noting it as a
>> > limitation of our current replacement policy.
>>
>> I don't see a strong reason from an L2 protocol's point of view yet, but
>> there are many unkown unknowns. But from a miner incentive's point of
>> view, we should keep the transaction with the higher feerate, shouldn't
>> we? In that case it's also a more efficient use of on-chain space, which
>> is a win, right?
>>
>> > We might have a more-or-less long transition period during which we
>> support both...
>>
>> Yes, this is a long term thing.
>> Even if bitcoin core releases a new version with updated RBF rules, as a
>> wallet you'll need to keep using the old rules for a long time if you
>> want to be safe.
>>
>> But it's all the more reason to try to ship this as soon as possible,
>> this way maybe our grand-children will be able to benefit from it ;)
>> (just kidding on the timespan obviously).
>>
>> Cheers,
>> Bastien
>>
>> Le lun. 31 janv. 2022 à 00:11, Antoine Riard via bitcoin-dev <
>> bitcoin-dev@lists.linuxfoundation.org> a écrit :
>>
>>> Hi Gloria,
>>>
>>> Thanks for this RBF sum up. Few thoughts and more context comments if it
>>> can help other readers.
>>>
>>> > For starters, the absolute fee pinning attack is especially
>>> > problematic if we apply the same rules (i.e. Rule #3 and #4) in
>>> > Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>>> > LN channel. The mempool is rather full, so their pre-negotiated
>>> > commitment transactions' feerates would not be considered high
>>> > priority by miners. Bob broadcasts his commitment transaction and
>>> > attaches a very large child (100KvB with 100,000sat in fees) to his
>>> > anchor output. Alice broadcasts her commitment transaction with a
>>> > fee-bumping child (200vB with 50,000sat fees which is a generous
>>> > 250sat/vB), but this does not meet the absolute fee requirement. She
>>> > would need to add another 50,000sat to replace Bob's commitment
>>> > transaction.
>>>
>>> Solving LN pinning attacks, what we're aiming for is enabling a fair
>>> feerate bid between the counterparties, thus either forcing the adversary
>>> to overbid or to disengage from the confirmation competition. If the
>>> replace-by-feerate rule is adopted, there shouldn't be an incentive for Bob
>>> to
>>> pick up the first option. Though if he does, that's a winning outcome
>>> for Alice, as one of the commitment transactions confirms and her
>>> time-sensitive second-stage HTLC can be subsequently confirmed.
>>>
>>> > It's unclear to me if
>>> > we have a very strong reason to change this, but noting it as a
>>> > limitation of our current replacement policy. See [#24007][12].
>>>
>>> Deployment of Taproot opens interesting possibilities in the
>>> vaults/payment channels design space, where the tapscripts can commit to
>>> different set of timelocks/quorum of keys. Even if the pre-signed states
>>> stay symmetric, whoever is the publisher, the feerate cost to spend can
>>> fluctuate.
>>>
>>> > While this isn't completely broken, and the user interface is
>>> > secondary to the safety of the mempool policy
>>>
>>> I think with L2s transaction broadcast backend, the stability and
>>> clarity of the RBF user interface is primary. What we could be worried
>>> about is a too-much complex interface easing the way for an attacker to
>>> trigger your L2 node to issue policy-invalid chain of transactions.
>>> Especially, when we consider that an attacker might have leverage on chain
>>> of transactions composition ("force broadcast of commitment A then
>>> commitment B, knowing they will share a CPFP") or even transactions size
>>> ("overload commitment A with HTLCs").
>>>
>>> > * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>> > mempool, apply the current rules (absolute fees must increase and
>>> > pay for the replacement transaction's new bandwidth). Otherwise, use a
>>> > feerate-only rule.
>>>
>>> How this new replacement rule would behave if you have a parent in the
>>> "replace-by-feerate" half but the child is in the "replace-by-fee" one ?
>>>
>>> If we allow the replacement of the parent based on the feerate, we might
>>> decrease the top block absolute fees.
>>>
>>> If we block the replacement of the parent based on the feerate because
>>> the replacement absolute fees aren't above the replaced package, we still
>>> preclude a pinning vector. The child might be low-feerate junk and even
>>> attached to a low ancestor-score branch.
>>>
>>> If I'm correct on this limitation, maybe we could turn off the
>>> "replace-by-fee" behavior as soon as the mempool is fulfilled with a few
>>> blocks ?
>>>
>>> > * Rate-limit how many replacements we allow per prevout.
>>>
>>> Depending on how it is implemented, though I would be concerned it
>>> introduces a new pinning vector in the context of shared-utxo. If it's a
>>> hardcoded constant, it could be exhausted by an adversary starting at the
>>> lowest acceptable feerate then slowly increasing while still not reaching
>>> the top of the mempool. Same if it's time-based or block-based, no
>>> guarantee the replacement slot is honestly used by your counterparty.
>>>
>>> Further, an above-the-average replacement frequency might just be the
>>> reflection of your confirmation strategy reacting to block schedule or
>>> mempools historical data. As long as the feerate penalty is paid, I lean to
>>> allow replacement.
>>>
>>> (One solution could be to associate per-user "tag" to the LN
>>> transactions, where each "tag" would have its own replacement slots, but
>>> privacy?)
>>>
>>> > * Rate-limit transaction validation in general, per peer.
>>>
>>> I think we could improve on the Core's new transaction requester logic.
>>> Maybe we could bind the peer announced flow based on the feerate score
>>> (modulo validation time) of the previously validated transactions from that
>>> peer ? That said, while related to RBF, it sounds to me that enhancing
>>> Core's rate-limiting transaction strategy is a whole discussion in itself
>>> [0]. Especially ensuring it's tolerant to the specific requirements of LN &
>>> consorts.
>>>
>>> > What should they be? We can do some arithmetic to see what happens if
>>> > you start with the biggest/lowest feerate transaction and do a bunch
>>> > of replacements. Maybe we end up with values that are high enough to
>>> > prevent abuse and make sense for applications/users that do RBF.
>>>
>>> That's a good question.
>>>
>>> One observation is that the attacker can always renew the set of DoSy
>>> utxos to pursue the attack. So maybe we could pick up constants scaled on
>>> the block size ? That way an attacker would have to burn fees, thus
>>> deterring them from launching an attack. Even if the attackers are miners,
>>> they have to renounce their income to acquire new DoSy utxos. If a low-fee
>>> period, we could scale up the constants ?
>>>
>>>
>>> Overall, I think there is the deployment issue to warn of. Moving to a
>>> new set of RBF rules implies for a lot of Bitcoin applications to rewrite
>>> their RBF logics. We might have a more-or-less long transition period
>>> during which we support both...
>>>
>>> Cheers,
>>> Antoine
>>>
>>> [0] https://github.com/bitcoin/bitcoin/pull/21224
>>>
>>> Le jeu. 27 janv. 2022 à 09:10, Gloria Zhao via bitcoin-dev <
>>> bitcoin-dev@lists.linuxfoundation.org> a écrit :
>>>
>>>> Hi everyone,
>>>>
>>>> This post discusses limitations of current Bitcoin Core RBF policy and
>>>> attempts to start a conversation about how we can improve it,
>>>> summarizing some ideas that have been discussed. Please reply if you
>>>> have any new input on issues to be solved and ideas for improvement!
>>>>
>>>> Just in case I've screwed up the text wrapping again, another copy can
>>>> be
>>>> found here:
>>>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff
>>>>
>>>> ## Background
>>>>
>>>> Please feel free to skip this section if you are already familiar
>>>> with RBF.
>>>>
>>>> Nodes may receive *conflicting* unconfirmed transactions, aka
>>>> "double spends" of the same inputs. Instead of always keeping the
>>>> first transaction, since v0.12, Bitcoin Core mempool policy has
>>>> included a set of Replace-by-Fee (RBF) criteria that allows the second
>>>> transaction to replace the first one and any descendants it may have.
>>>>
>>>> Bitcoin Core RBF policy was previously documented as BIP 125.
>>>> The current RBF policy is documented [here][1]. In summary:
>>>>
>>>> 1. The directly conflicting transactions all signal replaceability
>>>> explicitly.
>>>>
>>>> 2. The replacement transaction only includes an unconfirmed input if
>>>> that input was included in one of the directly conflicting
>>>> transactions.
>>>>
>>>> 3. The replacement transaction pays an absolute fee of at least the
>>>> sum paid by the original transactions.
>>>>
>>>> 4. The additional fees pays for the replacement transaction's
>>>> bandwidth at or above the rate set by the node's *incremental relay
>>>> feerate*.
>>>>
>>>> 5. The sum of all directly conflicting transactions' descendant counts
>>>> (number of transactions inclusive of itself and its descendants)
>>>> does not exceed 100.
>>>>
>>>> We can split these rules into 3 categories/goals:
>>>>
>>>> - **Allow Opting Out**: Some applications/businesses are unable to
>>>> handle transactions that are replaceable (e.g. merchants that use
>>>> zero-confirmation transactions). We (try to) help these businesses by
>>>> honoring BIP125 signaling; we won't replace transactions that have not
>>>> opted in.
>>>>
>>>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>>> accept replacement transactions which would decrease fee profits
>>>> of a miner. In general, if our mempool policy deviates from what is
>>>> economically rational, it's likely that the transactions in our
>>>> mempool will not match the ones in miners' mempools, making our
>>>> fee estimation, compact block relay, and other mempool-dependent
>>>> functions unreliable. Incentive-incompatible policy may also
>>>> encourage transaction submission through routes other than the p2p
>>>> network, harming censorship-resistance and privacy of Bitcoin payments.
>>>>
>>>> - **DoS Protection**: Limit two types of DoS attacks on the node's
>>>> mempool: (1) the number of times a transaction can be replaced and
>>>> (2) the volume of transactions that can be evicted during a
>>>> replacement.
>>>>
>>>> Even more abstract: our goal is to make a replacement policy that
>>>> results in a useful interface for users and safe policy for
>>>> node operators.
>>>>
>>>> ## Motivation
>>>>
>>>> There are a number of known problems with the current RBF policy.
>>>> Many of these shortcomings exist due to mempool limitations at the
>>>> time RBF was implemented or result from new types of Bitcoin usage;
>>>> they are not criticisms of the original design.
>>>>
>>>> ### Pinning Attacks
>>>>
>>>> The most pressing concern is that attackers may take advantage of
>>>> limitations in RBF policy to prevent other users' transactions from
>>>> being mined or getting accepted as a replacement.
>>>>
>>>> #### SIGHASH_ANYONECANPAY Pinning
>>>>
>>>> BIP125#2 can be bypassed by creating intermediary transactions to be
>>>> replaced together. Anyone can simply split a 1-input 1-output
>>>> transaction off from the replacement transaction, then broadcast the
>>>> transaction as is. This can always be done, and quite cheaply. More
>>>> details in [this comment][2].
>>>>
>>>> In general, if a transaction is signed with SIGHASH\_ANYONECANPAY,
>>>> anybody can just attach a low feerate parent to this transaction and
>>>> lower its ancestor feerate. Even if you require SIGHASH\_ALL which
>>>> prevents an attacker from changing any outputs, the input can be a
>>>> very low amount (e.g. just above the dust limit) from a low-fee
>>>> ancestor and still bring down the ancestor feerate of the transaction.
>>>>
>>>> TLDR: if your transaction is signed with SIGHASH\_ANYONECANPAY and
>>>> signals replaceability, regardless of the feerate you broadcast at, an
>>>> attacker can lower its mining priority by adding an ancestor.
>>>>
>>>> #### Absolute Fee
>>>>
>>>> The restriction of requiring replacement transactions to increase the
>>>> absolute fee of the mempool has been described as "bonkers." If the
>>>> original transaction has a very large descendant that pays a large
>>>> amount of fees, even if it has a low feerate, the replacement
>>>> transaction must now pay those fees in order to meet Rule #3.
>>>>
>>>> #### Package RBF
>>>>
>>>> There are a number of reasons why, in order to enable Package RBF, we
>>>> cannot use the same criteria.
>>>>
>>>> For starters, the absolute fee pinning attack is especially
>>>> problematic if we apply the same rules (i.e. Rule #3 and #4) in
>>>> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>>>> LN channel. The mempool is rather full, so their pre-negotiated
>>>> commitment transactions' feerates would not be considered high
>>>> priority by miners. Bob broadcasts his commitment transaction and
>>>> attaches a very large child (100KvB with 100,000sat in fees) to his
>>>> anchor output. Alice broadcasts her commitment transaction with a
>>>> fee-bumping child (200vB with 50,000sat fees which is a generous
>>>> 250sat/vB), but this does not meet the absolute fee requirement. She
>>>> would need to add another 50,000sat to replace Bob's commitment
>>>> transaction.
>>>>
>>>> Disallowing new unconfirmed inputs (Rule #2) in Package RBF would be
>>>> broken for packages containing transactions already in the mempool,
>>>> explained [here][7].
>>>>
>>>> Note: I originally [proposed][6] Package RBF using the same Rule #3
>>>> and #4 before I realized how significant this pinning attack is. I'm
>>>> retracting that proposal, and a new set of Package RBF rules would
>>>> follow from whatever the new individual RBF rules end up being.
>>>>
>>>> #### Same Txid Different Witness
>>>>
>>>> Two transactions with the same non-witness data but different
>>>> witnesses have the same txid but different wtxid, and the same fee but
>>>> not necessarily the same feerate. Currently, if we see a transaction
>>>> that has the same txid as one in the mempool, we reject it as a
>>>> duplicate, even if the feerate is much higher. It's unclear to me if
>>>> we have a very strong reason to change this, but noting it as a
>>>> limitation of our current replacement policy. See [#24007][12].
>>>>
>>>> ### User Interface
>>>>
>>>> #### Using Unconfirmed UTXOs to Fund Replacements
>>>>
>>>> The restriction of only allowing confirmed UTXOs for funding a
>>>> fee-bump (Rule #2) can hurt users trying to fee-bump their
>>>> transactions and complicate wallet implementations. If the original
>>>> transaction's output value isn't sufficient to fund a fee-bump and/or
>>>> all of the user's other UTXOs are unconfirmed, they might not be able
>>>> to fund a replacement transaction. Wallet developers also need to
>>>> treat self-owned unconfirmed UTXOs as unusable for fee-bumping, which
>>>> adds complexity to wallet logic. For example, see BDK issues [#144][4]
>>>> and [#414][5].
>>>>
>>>> #### Interface Not Suitable for Coin Selection
>>>>
>>>> Currently, a user cannot simply create a replacement transaction
>>>> targeting a specific feerate or meeting a minimum fee amount and
>>>> expect to meet the RBF criteria. The fee amount depends on the size of
>>>> the replacement transaction, and feerate is almost irrelevant.
>>>>
>>>> Bitcoin Core's `bumpfee` doesn't use the RBF rules when funding the
>>>> replacement. It [estimates][13] a feerate which is "wallet incremental
>>>> relay fee" (a conservative overestimation of the node's incremental
>>>> relay fee) higher than the original transaction, selects coins for
>>>> that feerate, and hopes that it meets the RBF rules. It never fails
>>>> Rule #3 and #4 because it uses all original inputs and refuses to
>>>> bump a transaction with mempool descendants.
>>>>
>>>> This is suboptimal, but is designed to work with the coin selection
>>>> engine: select a feerate first, and then add fees to cover it.
>>>> Following the exact RBF rules would require working the other way
>>>> around: based on how much fees we've added to the transaction and its
>>>> current size, calculate the feerate to see if we meet Rule #4.
>>>>
>>>> While this isn't completely broken, and the user interface is
>>>> secondary to the safety of the mempool policy, we can do much better.
>>>> A much more user-friendly interface would depend *only* on the
>>>> fee and size of the original transactions.
>>>>
>>>> ### Updates to Mempool and Mining
>>>>
>>>> Since RBF was first implemented, a number of improvements have been
>>>> made to mempool and mining logic. For example, we now use ancestor
>>>> feerates in mining (allowing CPFP), and keep track of ancestor
>>>> packages in the mempool.
>>>>
>>>> ## Ideas for Improvements
>>>>
>>>> ### Goals
>>>>
>>>> To summarize, these seem to be desired changes, in order of priority:
>>>>
>>>> 1. Remove Rule #3. The replacement should not be *required* to pay
>>>> higher absolute fees.
>>>>
>>>> 2. Make it impossible for a replacement transaction to have a lower
>>>> mining score than the original transaction(s). This would eliminate
>>>> the `SIGHASH\_ANYONECANPAY` pinning attack.
>>>>
>>>> 3. Remove Rule #2. Adding new unconfirmed inputs should be allowed.
>>>>
>>>> 4. Create a more helpful interface that helps wallet fund replacement
>>>> transactions that aim for a feerate and fee.
>>>>
>>>> ### A Different Model for Fees
>>>>
>>>> For incentive compatibility, I believe there are different
>>>> formulations we should consider. Most importantly, if we want to get
>>>> rid of the absolute fee rule, we can no longer think of it as "the
>>>> transaction needs to pay for its own bandwidth," since we won't always
>>>> be getting additional fees. That means we need a new method of
>>>> rate-limiting replacements that doesn't require additional fees every
>>>> time.
>>>>
>>>> While it makes sense to think about monetary costs when launching a
>>>> specific type of attack, given that the fees are paid to the miner and
>>>> not to the mempool operators, maybe it doesn't make much sense to
>>>> think about "paying for bandwidth". Maybe we should implement
>>>> transaction validation rate-limiting differently, e.g. building it
>>>> into the P2P layer instead of the mempool policy layer.
>>>>
>>>> Recently, Suhas gave a [formulation][8] for incentive compatibility
>>>> that made sense to me: "are the fees expected to be paid in the next
>>>> (N?) blocks higher or lower if we process this transaction?"
>>>>
>>>> I started by thinking about this where N=1 or `1 + p`.
>>>> Here, a rational miner is looking at what fees they would
>>>> collect in the next block, and then some proportion `p` of the rest of
>>>> the blocks based on their hashrate. We're assuming `p` isn't *so high*
>>>> that they would be okay with lower absolute fees in the next 1 block.
>>>> We're also assuming `p` isn't *so low* that the miner doesn't care
>>>> about what's left of the mempool after this block.
>>>>
>>>> A tweak to this formulation is "if we process this transaction, would
>>>> the fees in the next 1 block higher or lower, and is the feerate
>>>> density of the rest of the mempool higher or lower?" This is pretty
>>>> similar, where N=1, but we consider the rest of the mempool by feerate
>>>> rather than fees.
>>>>
>>>> ### Mining Score of a Mempool Transaction
>>>>
>>>> We are often interested in finding out what
>>>> the "mining score" of a transaction in the mempool is. That is, when
>>>> the transaction is considered in block template building, what is the
>>>> feerate it is considered at?
>>>>
>>>> Obviously, it's not the transaction's individual feerate. Bitcoin Core
>>>> [mining code sorts][14] transactions by their ancestor feerate and
>>>> includes them packages at a time, keeping track of how this affects the
>>>> package feerates of remaining transactions in the mempool.
>>>>
>>>> *ancestor feerate*: Ancestor feerate is easily accessible information,
>>>> but it's not accurate either, because it doesn't take into account the
>>>> fact that subsets of a transaction's ancestor set can be included
>>>> without it. For example, ancestors may have high feerates on their own
>>>> or we may have [high feerate siblings][8].
>>>>
>>>> TLDR: *Looking at the current ancestor feerate of a transaction is
>>>> insufficient to tell us what feerate it will be considered at when
>>>> building a block template in the future.*
>>>>
>>>> *min(individual feerate, ancestor feerate)*: Another
>>>> heuristic that is simple to calculate based on current mempool tooling
>>>> is to use the [minimum of a transaction's individual score and its
>>>> ancestor score][10] as a conservative measure. But this can
>>>> overestimate as well (see the example below).
>>>>
>>>> *min ancestor feerate(tx + possible ancestor subsets)* We can also
>>>> take the minimum of every possible ancestor subset, but this can be
>>>> computationally expensive since there can be lots and lots of ancestor
>>>> subsets.
>>>>
>>>> *max ancestor feerate(tx + possible descendant subsets)*: Another idea
>>>> is to use the [maximum ancestor score of the transaction + each of its
>>>> descendants][9]. This doesn't work either; it has the same blindspot
>>>> of ancestor subsets being mined on their own.
>>>>
>>>> #### Mining Score Example
>>>>
>>>> Here's an example illustrating why mining score is tricky to
>>>> efficiently calculate for mempool transactions:
>>>>
>>>> Let's say you have same-size transactions A (21sat/vB), B (1sat/vB),
>>>> C(9sat/vB), D(5sat/vB).
>>>> The layout is: grandparent A, parent B, and two children C and D.
>>>>
>>>> ```
>>>> A
>>>> ^
>>>> B
>>>> ^ ^
>>>> C D
>>>> ```
>>>>
>>>> A miner using ancestor packages to build block templates will first
>>>> include A with a mining score of 21. Next, the miner will include B and
>>>> C with a mining score of 6. This leaves D, with a mining score of 5.
>>>>
>>>> Note: in this case, mining by ancestor feerate results in the most
>>>> rational decisions, but [a candidate set-based approach][10] which
>>>> makes ancestor feerate much less relevant could
>>>> be more advantageous in other situations.
>>>>
>>>> Here is a chart showing the "true" mining score alongside the values
>>>> calculating using imperfect heuristics described above. All of them
>>>> can overestimate or underestimate.
>>>>
>>>> ```
>>>> A B C D
>>>> mining score | 21 | 6 | 6 | 5 |
>>>> ancestor feerate | 21 | 11 | 10.3 | 9 |
>>>> min(individual, ancestor) | 21 | 1 | 9 | 5 |
>>>> min(tx + ancestor subsets) | 21 | 1 | 5 | 3 |
>>>> max(tx + descendants subsets) | 21 | 9 | 9 | 5 |
>>>>
>>>> ```
>>>>
>>>> Possibly the best solution for finding the "mining score" of a
>>>> transaction is to build a block template, see what feerate each
>>>> package is included at. Perhaps at some cutoff, remaining mempool
>>>> transactions can be estimated using some heuristic that leans
>>>> {overestimating, underestimating} depending on the situation.
>>>>
>>>> Mining score seems to be relevant in multiple places: Murch and I
>>>> recently [found][3] that it would be very important in
>>>> "ancestor-aware" funding of transactions (the wallet doesn't
>>>> incorporate ancestor fees when using unconfirmed transactions in coin
>>>> selection, which is a bug we want to fix).
>>>>
>>>> In general, it would be nice to know the exact mining priority of
>>>> one's unconfirmed transaction is. I can think of a few block/mempool
>>>> explorers who might want to display this information for users.
>>>>
>>>> ### RBF Improvement Proposals
>>>>
>>>> After speaking to quite a few people, here are some suggestions
>>>> for improvements that I have heard:
>>>>
>>>> * The ancestor score of the replacement must be {5, 10, N}% higher
>>>> than that of every original transaction.
>>>>
>>>> * The ancestor score of the replacement must be 1sat/vB higher than
>>>> that of every original transaction.
>>>>
>>>> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>>> mempool, apply the current rules (absolute fees must increase and
>>>> pay for the replacement transaction's new bandwidth). Otherwise, use a
>>>> feerate-only rule.
>>>>
>>>> * If fees don't increase, the size of the replacement transaction must
>>>> decrease by at least N%.
>>>>
>>>> * Rate-limit how many replacements we allow per prevout.
>>>>
>>>> * Rate-limit transaction validation in general, per peer.
>>>>
>>>> Perhaps some others on the mailing list can chime in to throw other
>>>> ideas into the ring and/or combine some of these rules into a sensible
>>>> policy.
>>>>
>>>> #### Replace by Feerate Only
>>>>
>>>> I don't think there's going to be a single-line feerate-based
>>>> rule that can incorporate everything we need.
>>>> On one hand, a feerate-only approach helps eliminate the issues
>>>> associated with Rule #3. On the other hand, I believe the main concern
>>>> with a feerate-only approach is how to rate limit replacements. We
>>>> don't want to enable an attack such as:
>>>>
>>>> 1. Attacker broadcasts large, low-feerate transaction, and attaches a
>>>> chain of descendants.
>>>>
>>>> 2. The attacker replaces the transaction with a smaller but higher
>>>> feerate transaction, attaching a new chain of descendants.
>>>>
>>>> 3. Repeat 1000 times.
>>>>
>>>> #### Fees in Next Block and Feerate for the Rest of the Mempool
>>>>
>>>> Perhaps we can look at replacements like this:
>>>>
>>>> 1. Calculate the directly conflicting transactions and, with their
>>>> descendants, the original transactions. Check signaling. Limit the
>>>> total volume (e.g. can't be more than 100 total or 1MvB or something).
>>>>
>>>> 2. Find which original transactions would be in the next ~1 block. The
>>>> replacement must pay at least this amount + X% in absolute fees. This
>>>> guarantees that the fees of the next block doesn't decrease.
>>>>
>>>> 3. Find which transactions would be left in the mempool after that ~1
>>>> block. The replacement's feerate must be Y% higher than the maximum
>>>> mining score of these transactions. This guarantees that you now have
>>>> only *better* candidates in your after-this-block mempool than you did
>>>> before, even if the size and fees the transactions decrease.
>>>>
>>>> 4. Now you have two numbers: a minimum absolute fee amount and a
>>>> minimum feerate. Check to see if the replacement(s) meet these
>>>> minimums. Also, a wallet would be able to ask the node "What fee and
>>>> feerate would I need to put on a transaction replacing this?" and use
>>>> this information to fund a replacement transaction, without needing to
>>>> guess or overshoot.
>>>>
>>>> Obviously, there are some magic numbers missing here. X and Y are
>>>> TBD constants to ensure we have some kind of rate limiting for the
>>>> number of replacements allowed using some set of fees.
>>>>
>>>> What should they be? We can do some arithmetic to see what happens if
>>>> you start with the biggest/lowest feerate transaction and do a bunch
>>>> of replacements. Maybe we end up with values that are high enough to
>>>> prevent abuse and make sense for applications/users that do RBF.
>>>>
>>>> ### Mempool Changes Need for Implementation
>>>>
>>>> As described in the mining score section above,
>>>> we may want additional tooling to more accurately assess
>>>> the economic gain of replacing transactions in our mempool.
>>>>
>>>> A few options have been discussed:
>>>>
>>>> * Calculate block templates on the fly when we need to consider a
>>>> replacement. However, since replacements are [quite common][11]
>>>> and the information might be useful for other things as well,
>>>> it may be worth it to cache a block template.
>>>>
>>>> * Keep a persistent block template so that we know what transactions
>>>> we would put in the next block. We need to remember the feerate
>>>> at which each transaction was included in the template, because an
>>>> ancestor package may be included in the same block template in
>>>> multiple subsets. Transactions included earlier alter the ancestor
>>>> feerate of the remaining transactions in the package. We also need
>>>> to keep track of the new feerates of transactions left over.
>>>>
>>>> * Divide the mempool into two layers, "high feerate" and "low
>>>> feerate." The high feerate layer contains ~1 block of packages with
>>>> the highest ancestor feerates, and the low feerate layer contains
>>>> everything else. At the edge of a block, we have a Knapsacky problem
>>>> where the next highest ancestor feerate package might not fit, so we
>>>> would probably want the high feerate layer ~2MvB or something to avoid
>>>> underestimating the fees.
>>>>
>>>> ## Acknowledgements
>>>>
>>>> Thank you to everyone whose RBF-related suggestions, grievances,
>>>> criticisms and ideas were incorporated in this document:
>>>> Andrew Chow, Matt Corallo, Suhas Daftuar, Christian Decker,
>>>> Mark Erhardt, Lloyd Fournier, Lisa Neigut, John Newbery,
>>>> Antoine Poinsot, Antoine Riard, Larry Ruane,
>>>> S3RK and Bastien Teinturier.
>>>>
>>>> Thanks for reading!
>>>>
>>>> Best,
>>>> Gloria
>>>>
>>>> [1]:
>>>> https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
>>>> [2]:
>>>> https://github.com/bitcoin/bitcoin/pull/23121#issuecomment-929475999
>>>> [3]:
>>>> https://github.com/Xekyo/bitcoin/commit/d754b0242ec69d42c570418aebf9c1335af0b8ea
>>>> [4]: https://github.com/bitcoindevkit/bdk/issues/144
>>>> [5]: https://github.com/bitcoindevkit/bdk/issues/414
>>>> [6]:
>>>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
>>>> [7]:
>>>> https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a#new-unconfirmed-inputs-rule-2
>>>> [8]:
>>>> https://github.com/bitcoin/bitcoin/pull/23121#discussion_r777131366
>>>> [9]:
>>>> https://github.com/bitcoin/bitcoin/pull/22290#issuecomment-865887922
>>>> [10]:
>>>> https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
>>>> [11]:
>>>> https://github.com/bitcoin/bitcoin/pull/22539#issuecomment-885763670
>>>> [12]: https://github.com/bitcoin/bitcoin/pull/24007
>>>> [13]:
>>>> https://github.com/bitcoin/bitcoin/blob/1a369f006fd0bec373b95001ed84b480e852f191/src/wallet/feebumper.cpp#L114
>>>> [14]:
>>>> https://github.com/bitcoin/bitcoin/blob/cf5bb048e80d4cde8828787b266b7f5f2e3b6d7b/src/node/miner.cpp#L310-L320
>>>> _______________________________________________
>>>> bitcoin-dev mailing list
>>>> bitcoin-dev@lists.linuxfoundation.org
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>
>> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 55813 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Bastien TEINTURIER]

[-- Attachment #1: Type: text/plain, Size: 42034 bytes --]

Good morning,

> The tricky question is what happens when X arrives on its own and it
> might be that no one ever sends a replacement for B,C,D)

It feels ok to me, but this is definitely arguable.

It covers the fact that B,C,D could have been fake transactions whose
sole purpose was to do a pinning attack: in that case the attacker would
have found a way to ensure these transactions don't confirm anyway (or
pay minimal/negligible fees).

If these transactions were legitimate, I believe that their owners would
remake them at some point (because these transactions reflect a business
relationship that needed to happen, so it should very likely still
happen). It's probably hard to verify because the new corresponding
transactions may have nothing in common with the first, but I think the
simplifications it offers for wallets is worth it (which is just my
opinion and needs more scrutiny/feedback).

> But if your backlog's feerate does drop off, *and* that matters, then
> I don't think you can ignore the impact of the descendent transactions
> that you might not get a replacement for.

That is because you're only taking into account the current backlog, and
not taking into account the fact that new items will be added to it soon
to replace the evicted descendants. But I agree that this is a bet: we
can't predict the future and guarantee these replacements will come.

It is really a trade-off, ignoring descendents provides a much simpler
contract that doesn't vary from one mempool to another, but when your
backlog isn't full enough, you may lose some future profits if
transactions don't come in later.

> I think "Y% higher" rather than just "higher" is only useful for
> rate-limiting, not incentive compatibility. (Though maybe it helps
> stabilise a greedy algorithm in some cases?)

That's true. I claimed these policies only address incentives, but using
a percentage increase addresses rate-limiting a bit as well (I couldn't
resist trying to do at least something for it!). I find it a very easy
mechanism to implement, while choosing an absolute value is hard (it's
always easier to think in relatives than absolutes).

> This is why I think it is important to understand the rationales for
introducing the rules in the first place

I completely agree. As you mentioned, we are still in brainstorming
phase, once (if?) we start to converge on what could be better policies,
we do need to clearly explain each policy's expected goal. That will let
future Bastien writing code in 2030 clearly highlight why the 2022 rules
don't make sense anymore!

Cheers,
Bastien

Le sam. 5 févr. 2022 à 14:22, Michael Folkson <michaelfolkson@protonmail.com>
a écrit :

> Thanks for this Bastien (and Gloria for initially posting about this).
>
> I sympathetically skimmed the eclair PR (
> https://github.com/ACINQ/eclair/pull/2113) dealing with replaceable
> transactions fee bumping.
>
> There will continue to be a (hopefully) friendly tug of war on this
> probably for the rest of Bitcoin's existence. I am sure people like Luke,
> Prayank etc will (rightfully) continue to raise that Lightning and other
> second layer protocols shouldn't demand that policy rules be changed if
> there is a reason (e.g. DoS vector) for those rules on the base network.
> But if there are rules that have no upside, introduce unnecessary
> complexity for no reason and make Lightning implementers like Bastien's
> life miserable attempting to deal with them I really hope we can make
> progress on removing or simplifying them.
>
> This is why I think it is important to understand the rationales for
> introducing the rules in the first place (and why it is safe to remove them
> if indeed it is) and being as rigorous as possible on the rationales for
> introducing additional rules. It sounds like from Gloria's initial post we
> are still at a brainstorming phase (which is fine) but knowing what we know
> today I really hope we can learn from the mistakes of the original BIP 125,
> namely the Core implementation not matching the BIP and the sparse
> rationales for the rules. As Bastien says this is not criticizing the
> original BIP 125 authors, 7 years is a long time especially in Bitcoin
> world and they probably weren't thinking about Bastien sitting down to
> write an eclair PR in late 2021 (and reviewers of that PR) when they wrote
> the BIP in 2015.
>
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>
>
>
> ------- Original Message -------
> On Monday, January 31st, 2022 at 3:57 PM, Bastien TEINTURIER via
> bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> Hi Gloria,
>
> Many thanks for raising awareness on these issues and constantly pushing
> towards finding a better model. This work will highly improve the
> security of any multi-party contract trying to build on top of bitcoin
> (because most multi-party contracts will need to have timeout conditions
> and participants will need to make some transactions confirm before a
> timeout happens - otherwise they may lose funds).
>
> For starters, let me quickly explain why the current rules are hard to
> work with in the context of lightning (but I believe most L2 protocols
> will have the same issues). Feel free to skip this part if you are
> already convinced.
>
> ## Motivation
>
> The biggest pain point is BIP 125 rule 2.
> If I need to increase the fees of a time-sensitive transaction because
> the feerate has been rising since I broadcast it, I may need to also pay
> high fees just to produce a confirmed utxo that I can use. I'm actually
> paying a high fee twice instead of once (and needlessly using on-chain
> space, our scarcest asset, because we could have avoided that additional
> transaction!).
>
> It also has some annoying "non-determinism".
> Imagine that my transaction has been evicted from my mempool because its
> feerate was too low. I could think "Great, that means I don't have to
> apply BIP 125 restrictions, I can just fund this transaction as if it
> were a new one!". But actually I do, because my transaction could still
> be in miner's mempools and I have no way of knowing it...this means that
> whenever I have broadcast a transaction, I must assume that I will
> always need to abide by whatever replacement rules the network applies.
>
> Fortunately, as far as I understand it, this rule only exists because of
> a previous implementation detail of bitcoin core, so there's simply no
> good reason to keep it.
>
> The second biggest pain point is rule 3. It prevents me from efficiently
> using my capital while it's unconfirmed. Whenever I'm using a big utxo
> to fund a transaction, I will get a big change output, and it would
> really be a waste to be unable to use that change output to fund other
> transactions. In order to be capital-efficient, I will end up creating
> descendant trees for my time-sensitive transactions. But as Gloria
> explained, replacing all my children will cost me an absurdly large
> amount of fees. So what I'm actually planning to do instead is to RBF
> one of the descendants high enough to get the whole tree confirmed.
> But if those descendants' timeouts were far in the future, that's a
> waste, I paid a lot more fees for them than I should have. I'd like to
> just replace my transaction and republish the invalidated children
> independently.
>
> Rule 4 doesn't hurt as much as the two previous ones, I don't have too
> much to say about it.
>
> To be fair to the BIP 125 authors, all of these scenarios were very hard
> to forecast at the time this BIP was created. We needed years to build
> on those rules to get a better understanding of their limitations and if
> the rationale behind them made sense in the long term.
>
> ## Proposals
>
> I believe that now is a good time to re-think those, and I really like
> Gloria's categorization of the design constraints.
>
> I'd like to propose a different way of looking at descendants that makes
> it easier to design the new rules. The way I understand it, limiting the
> impact on descendant transactions is only important for DoS protection,
> not for incentive compatibility. I would argue that after evictions,
> descendant transactions will be submitted again (because they represent
> transactions that people actually want to make), so evicting them does
> not have a negative impact on mining incentives (in a world where blocks
> are full most of the time).
>
> I'm curious to hear other people's thoughts on that. If it makes sense,
> I would propose the following very simple rules:
>
> 1. The transaction's ancestor absolute fees must be X% higher than the
> previous transaction's ancestor fees
> 2. The transaction's ancestor feerate must be Y% higher than the
> previous transaction's ancestor feerate
>
> I believe it's completely ok to require increasing both the fees and
> feerate if we don't take descendants into account, because you control
> your ancestor set - whereas the descendant set may be completely out of
> your control.
>
> This is very easy to use by wallets, because the ancestor set is easy to
> obtain. And an important point is that the ancestor set is the same in
> every mempool, whereas the descendant set is not (your mempool may have
> rejected the last descendants, while other people's mempools may still
> contain them).
>
> Because of that reason, I'd like to avoid having a rule that relies on
> some size of the replaced descendant set: it may be valid in your
> mempool but invalid in someone else's, which makes it exploitable for
> pinning attacks.
>
> I believe these rules are incentive compatible (again, if you accept
> the fact that the descendants will be re-submitted and mined as well,
> so their fees aren't lost).
>
> Can we choose X and Y so that these two rules are also DoS-resistant?
> Unfortunately I'm not sure, so maybe we'll need to add a third rule to
> address that. But before we do, can someone detail what it costs for a
> node to evict a descendant tree? Given that bitcoin core doesn't allow
> chains of more than 25 transactions, the maximum number of transactions
> being replaced will be bounded by 25 * N (where N is the number of
> outputs of the transaction being replaced). If it's just O(n) pruning of
> a graph, maybe that's ok? Or maybe we make X or Y depend on the number
> of outputs of the transaction being replaced (this would need very
> careful thoughts)?
>
> If you made it this far, thanks for reading!
> A couple of comments on the previous messages:
>
> > Currently, if we see a transaction
> > that has the same txid as one in the mempool, we reject it as a
> > duplicate, even if the feerate is much higher. It's unclear to me if
> > we have a very strong reason to change this, but noting it as a
> > limitation of our current replacement policy.
>
> I don't see a strong reason from an L2 protocol's point of view yet, but
> there are many unkown unknowns. But from a miner incentive's point of
> view, we should keep the transaction with the higher feerate, shouldn't
> we? In that case it's also a more efficient use of on-chain space, which
> is a win, right?
>
> > We might have a more-or-less long transition period during which we
> support both...
>
> Yes, this is a long term thing.
> Even if bitcoin core releases a new version with updated RBF rules, as a
> wallet you'll need to keep using the old rules for a long time if you
> want to be safe.
>
> But it's all the more reason to try to ship this as soon as possible,
> this way maybe our grand-children will be able to benefit from it ;)
> (just kidding on the timespan obviously).
>
> Cheers,
> Bastien
>
> Le lun. 31 janv. 2022 à 00:11, Antoine Riard via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> a écrit :
>
>> Hi Gloria,
>>
>> Thanks for this RBF sum up. Few thoughts and more context comments if it
>> can help other readers.
>>
>> > For starters, the absolute fee pinning attack is especially
>> > problematic if we apply the same rules (i.e. Rule #3 and #4) in
>> > Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>> > LN channel. The mempool is rather full, so their pre-negotiated
>> > commitment transactions' feerates would not be considered high
>> > priority by miners. Bob broadcasts his commitment transaction and
>> > attaches a very large child (100KvB with 100,000sat in fees) to his
>> > anchor output. Alice broadcasts her commitment transaction with a
>> > fee-bumping child (200vB with 50,000sat fees which is a generous
>> > 250sat/vB), but this does not meet the absolute fee requirement. She
>> > would need to add another 50,000sat to replace Bob's commitment
>> > transaction.
>>
>> Solving LN pinning attacks, what we're aiming for is enabling a fair
>> feerate bid between the counterparties, thus either forcing the adversary
>> to overbid or to disengage from the confirmation competition. If the
>> replace-by-feerate rule is adopted, there shouldn't be an incentive for Bob
>> to
>> pick up the first option. Though if he does, that's a winning outcome for
>> Alice, as one of the commitment transactions confirms and her
>> time-sensitive second-stage HTLC can be subsequently confirmed.
>>
>> > It's unclear to me if
>> > we have a very strong reason to change this, but noting it as a
>> > limitation of our current replacement policy. See [#24007][12].
>>
>> Deployment of Taproot opens interesting possibilities in the
>> vaults/payment channels design space, where the tapscripts can commit to
>> different set of timelocks/quorum of keys. Even if the pre-signed states
>> stay symmetric, whoever is the publisher, the feerate cost to spend can
>> fluctuate.
>>
>> > While this isn't completely broken, and the user interface is
>> > secondary to the safety of the mempool policy
>>
>> I think with L2s transaction broadcast backend, the stability and clarity
>> of the RBF user interface is primary. What we could be worried about is a
>> too-much complex interface easing the way for an attacker to trigger your
>> L2 node to issue policy-invalid chain of transactions. Especially, when we
>> consider that an attacker might have leverage on chain of transactions
>> composition ("force broadcast of commitment A then commitment B, knowing
>> they will share a CPFP") or even transactions size ("overload commitment A
>> with HTLCs").
>>
>> > * If the original transaction is in the top {0.75MvB, 1MvB} of the
>> > mempool, apply the current rules (absolute fees must increase and
>> > pay for the replacement transaction's new bandwidth). Otherwise, use a
>> > feerate-only rule.
>>
>> How this new replacement rule would behave if you have a parent in the
>> "replace-by-feerate" half but the child is in the "replace-by-fee" one ?
>>
>> If we allow the replacement of the parent based on the feerate, we might
>> decrease the top block absolute fees.
>>
>> If we block the replacement of the parent based on the feerate because
>> the replacement absolute fees aren't above the replaced package, we still
>> preclude a pinning vector. The child might be low-feerate junk and even
>> attached to a low ancestor-score branch.
>>
>> If I'm correct on this limitation, maybe we could turn off the
>> "replace-by-fee" behavior as soon as the mempool is fulfilled with a few
>> blocks ?
>>
>> > * Rate-limit how many replacements we allow per prevout.
>>
>> Depending on how it is implemented, though I would be concerned it
>> introduces a new pinning vector in the context of shared-utxo. If it's a
>> hardcoded constant, it could be exhausted by an adversary starting at the
>> lowest acceptable feerate then slowly increasing while still not reaching
>> the top of the mempool. Same if it's time-based or block-based, no
>> guarantee the replacement slot is honestly used by your counterparty.
>>
>> Further, an above-the-average replacement frequency might just be the
>> reflection of your confirmation strategy reacting to block schedule or
>> mempools historical data. As long as the feerate penalty is paid, I lean to
>> allow replacement.
>>
>> (One solution could be to associate per-user "tag" to the LN
>> transactions, where each "tag" would have its own replacement slots, but
>> privacy?)
>>
>> > * Rate-limit transaction validation in general, per peer.
>>
>> I think we could improve on the Core's new transaction requester logic.
>> Maybe we could bind the peer announced flow based on the feerate score
>> (modulo validation time) of the previously validated transactions from that
>> peer ? That said, while related to RBF, it sounds to me that enhancing
>> Core's rate-limiting transaction strategy is a whole discussion in itself
>> [0]. Especially ensuring it's tolerant to the specific requirements of LN &
>> consorts.
>>
>> > What should they be? We can do some arithmetic to see what happens if
>> > you start with the biggest/lowest feerate transaction and do a bunch
>> > of replacements. Maybe we end up with values that are high enough to
>> > prevent abuse and make sense for applications/users that do RBF.
>>
>> That's a good question.
>>
>> One observation is that the attacker can always renew the set of DoSy
>> utxos to pursue the attack. So maybe we could pick up constants scaled on
>> the block size ? That way an attacker would have to burn fees, thus
>> deterring them from launching an attack. Even if the attackers are miners,
>> they have to renounce their income to acquire new DoSy utxos. If a low-fee
>> period, we could scale up the constants ?
>>
>>
>> Overall, I think there is the deployment issue to warn of. Moving to a
>> new set of RBF rules implies for a lot of Bitcoin applications to rewrite
>> their RBF logics. We might have a more-or-less long transition period
>> during which we support both...
>>
>> Cheers,
>> Antoine
>>
>> [0] https://github.com/bitcoin/bitcoin/pull/21224
>>
>> Le jeu. 27 janv. 2022 à 09:10, Gloria Zhao via bitcoin-dev <
>> bitcoin-dev@lists.linuxfoundation.org> a écrit :
>>
>>> Hi everyone,
>>>
>>> This post discusses limitations of current Bitcoin Core RBF policy and
>>> attempts to start a conversation about how we can improve it,
>>> summarizing some ideas that have been discussed. Please reply if you
>>> have any new input on issues to be solved and ideas for improvement!
>>>
>>> Just in case I've screwed up the text wrapping again, another copy can be
>>> found here:
>>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff
>>>
>>> ## Background
>>>
>>> Please feel free to skip this section if you are already familiar
>>> with RBF.
>>>
>>> Nodes may receive *conflicting* unconfirmed transactions, aka
>>> "double spends" of the same inputs. Instead of always keeping the
>>> first transaction, since v0.12, Bitcoin Core mempool policy has
>>> included a set of Replace-by-Fee (RBF) criteria that allows the second
>>> transaction to replace the first one and any descendants it may have.
>>>
>>> Bitcoin Core RBF policy was previously documented as BIP 125.
>>> The current RBF policy is documented [here][1]. In summary:
>>>
>>> 1. The directly conflicting transactions all signal replaceability
>>> explicitly.
>>>
>>> 2. The replacement transaction only includes an unconfirmed input if
>>> that input was included in one of the directly conflicting
>>> transactions.
>>>
>>> 3. The replacement transaction pays an absolute fee of at least the
>>> sum paid by the original transactions.
>>>
>>> 4. The additional fees pays for the replacement transaction's
>>> bandwidth at or above the rate set by the node's *incremental relay
>>> feerate*.
>>>
>>> 5. The sum of all directly conflicting transactions' descendant counts
>>> (number of transactions inclusive of itself and its descendants)
>>> does not exceed 100.
>>>
>>> We can split these rules into 3 categories/goals:
>>>
>>> - **Allow Opting Out**: Some applications/businesses are unable to
>>> handle transactions that are replaceable (e.g. merchants that use
>>> zero-confirmation transactions). We (try to) help these businesses by
>>> honoring BIP125 signaling; we won't replace transactions that have not
>>> opted in.
>>>
>>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>> accept replacement transactions which would decrease fee profits
>>> of a miner. In general, if our mempool policy deviates from what is
>>> economically rational, it's likely that the transactions in our
>>> mempool will not match the ones in miners' mempools, making our
>>> fee estimation, compact block relay, and other mempool-dependent
>>> functions unreliable. Incentive-incompatible policy may also
>>> encourage transaction submission through routes other than the p2p
>>> network, harming censorship-resistance and privacy of Bitcoin payments.
>>>
>>> - **DoS Protection**: Limit two types of DoS attacks on the node's
>>> mempool: (1) the number of times a transaction can be replaced and
>>> (2) the volume of transactions that can be evicted during a
>>> replacement.
>>>
>>> Even more abstract: our goal is to make a replacement policy that
>>> results in a useful interface for users and safe policy for
>>> node operators.
>>>
>>> ## Motivation
>>>
>>> There are a number of known problems with the current RBF policy.
>>> Many of these shortcomings exist due to mempool limitations at the
>>> time RBF was implemented or result from new types of Bitcoin usage;
>>> they are not criticisms of the original design.
>>>
>>> ### Pinning Attacks
>>>
>>> The most pressing concern is that attackers may take advantage of
>>> limitations in RBF policy to prevent other users' transactions from
>>> being mined or getting accepted as a replacement.
>>>
>>> #### SIGHASH_ANYONECANPAY Pinning
>>>
>>> BIP125#2 can be bypassed by creating intermediary transactions to be
>>> replaced together. Anyone can simply split a 1-input 1-output
>>> transaction off from the replacement transaction, then broadcast the
>>> transaction as is. This can always be done, and quite cheaply. More
>>> details in [this comment][2].
>>>
>>> In general, if a transaction is signed with SIGHASH\_ANYONECANPAY,
>>> anybody can just attach a low feerate parent to this transaction and
>>> lower its ancestor feerate. Even if you require SIGHASH\_ALL which
>>> prevents an attacker from changing any outputs, the input can be a
>>> very low amount (e.g. just above the dust limit) from a low-fee
>>> ancestor and still bring down the ancestor feerate of the transaction.
>>>
>>> TLDR: if your transaction is signed with SIGHASH\_ANYONECANPAY and
>>> signals replaceability, regardless of the feerate you broadcast at, an
>>> attacker can lower its mining priority by adding an ancestor.
>>>
>>> #### Absolute Fee
>>>
>>> The restriction of requiring replacement transactions to increase the
>>> absolute fee of the mempool has been described as "bonkers." If the
>>> original transaction has a very large descendant that pays a large
>>> amount of fees, even if it has a low feerate, the replacement
>>> transaction must now pay those fees in order to meet Rule #3.
>>>
>>> #### Package RBF
>>>
>>> There are a number of reasons why, in order to enable Package RBF, we
>>> cannot use the same criteria.
>>>
>>> For starters, the absolute fee pinning attack is especially
>>> problematic if we apply the same rules (i.e. Rule #3 and #4) in
>>> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>>> LN channel. The mempool is rather full, so their pre-negotiated
>>> commitment transactions' feerates would not be considered high
>>> priority by miners. Bob broadcasts his commitment transaction and
>>> attaches a very large child (100KvB with 100,000sat in fees) to his
>>> anchor output. Alice broadcasts her commitment transaction with a
>>> fee-bumping child (200vB with 50,000sat fees which is a generous
>>> 250sat/vB), but this does not meet the absolute fee requirement. She
>>> would need to add another 50,000sat to replace Bob's commitment
>>> transaction.
>>>
>>> Disallowing new unconfirmed inputs (Rule #2) in Package RBF would be
>>> broken for packages containing transactions already in the mempool,
>>> explained [here][7].
>>>
>>> Note: I originally [proposed][6] Package RBF using the same Rule #3
>>> and #4 before I realized how significant this pinning attack is. I'm
>>> retracting that proposal, and a new set of Package RBF rules would
>>> follow from whatever the new individual RBF rules end up being.
>>>
>>> #### Same Txid Different Witness
>>>
>>> Two transactions with the same non-witness data but different
>>> witnesses have the same txid but different wtxid, and the same fee but
>>> not necessarily the same feerate. Currently, if we see a transaction
>>> that has the same txid as one in the mempool, we reject it as a
>>> duplicate, even if the feerate is much higher. It's unclear to me if
>>> we have a very strong reason to change this, but noting it as a
>>> limitation of our current replacement policy. See [#24007][12].
>>>
>>> ### User Interface
>>>
>>> #### Using Unconfirmed UTXOs to Fund Replacements
>>>
>>> The restriction of only allowing confirmed UTXOs for funding a
>>> fee-bump (Rule #2) can hurt users trying to fee-bump their
>>> transactions and complicate wallet implementations. If the original
>>> transaction's output value isn't sufficient to fund a fee-bump and/or
>>> all of the user's other UTXOs are unconfirmed, they might not be able
>>> to fund a replacement transaction. Wallet developers also need to
>>> treat self-owned unconfirmed UTXOs as unusable for fee-bumping, which
>>> adds complexity to wallet logic. For example, see BDK issues [#144][4]
>>> and [#414][5].
>>>
>>> #### Interface Not Suitable for Coin Selection
>>>
>>> Currently, a user cannot simply create a replacement transaction
>>> targeting a specific feerate or meeting a minimum fee amount and
>>> expect to meet the RBF criteria. The fee amount depends on the size of
>>> the replacement transaction, and feerate is almost irrelevant.
>>>
>>> Bitcoin Core's `bumpfee` doesn't use the RBF rules when funding the
>>> replacement. It [estimates][13] a feerate which is "wallet incremental
>>> relay fee" (a conservative overestimation of the node's incremental
>>> relay fee) higher than the original transaction, selects coins for
>>> that feerate, and hopes that it meets the RBF rules. It never fails
>>> Rule #3 and #4 because it uses all original inputs and refuses to
>>> bump a transaction with mempool descendants.
>>>
>>> This is suboptimal, but is designed to work with the coin selection
>>> engine: select a feerate first, and then add fees to cover it.
>>> Following the exact RBF rules would require working the other way
>>> around: based on how much fees we've added to the transaction and its
>>> current size, calculate the feerate to see if we meet Rule #4.
>>>
>>> While this isn't completely broken, and the user interface is
>>> secondary to the safety of the mempool policy, we can do much better.
>>> A much more user-friendly interface would depend *only* on the
>>> fee and size of the original transactions.
>>>
>>> ### Updates to Mempool and Mining
>>>
>>> Since RBF was first implemented, a number of improvements have been
>>> made to mempool and mining logic. For example, we now use ancestor
>>> feerates in mining (allowing CPFP), and keep track of ancestor
>>> packages in the mempool.
>>>
>>> ## Ideas for Improvements
>>>
>>> ### Goals
>>>
>>> To summarize, these seem to be desired changes, in order of priority:
>>>
>>> 1. Remove Rule #3. The replacement should not be *required* to pay
>>> higher absolute fees.
>>>
>>> 2. Make it impossible for a replacement transaction to have a lower
>>> mining score than the original transaction(s). This would eliminate
>>> the `SIGHASH\_ANYONECANPAY` pinning attack.
>>>
>>> 3. Remove Rule #2. Adding new unconfirmed inputs should be allowed.
>>>
>>> 4. Create a more helpful interface that helps wallet fund replacement
>>> transactions that aim for a feerate and fee.
>>>
>>> ### A Different Model for Fees
>>>
>>> For incentive compatibility, I believe there are different
>>> formulations we should consider. Most importantly, if we want to get
>>> rid of the absolute fee rule, we can no longer think of it as "the
>>> transaction needs to pay for its own bandwidth," since we won't always
>>> be getting additional fees. That means we need a new method of
>>> rate-limiting replacements that doesn't require additional fees every
>>> time.
>>>
>>> While it makes sense to think about monetary costs when launching a
>>> specific type of attack, given that the fees are paid to the miner and
>>> not to the mempool operators, maybe it doesn't make much sense to
>>> think about "paying for bandwidth". Maybe we should implement
>>> transaction validation rate-limiting differently, e.g. building it
>>> into the P2P layer instead of the mempool policy layer.
>>>
>>> Recently, Suhas gave a [formulation][8] for incentive compatibility
>>> that made sense to me: "are the fees expected to be paid in the next
>>> (N?) blocks higher or lower if we process this transaction?"
>>>
>>> I started by thinking about this where N=1 or `1 + p`.
>>> Here, a rational miner is looking at what fees they would
>>> collect in the next block, and then some proportion `p` of the rest of
>>> the blocks based on their hashrate. We're assuming `p` isn't *so high*
>>> that they would be okay with lower absolute fees in the next 1 block.
>>> We're also assuming `p` isn't *so low* that the miner doesn't care
>>> about what's left of the mempool after this block.
>>>
>>> A tweak to this formulation is "if we process this transaction, would
>>> the fees in the next 1 block higher or lower, and is the feerate
>>> density of the rest of the mempool higher or lower?" This is pretty
>>> similar, where N=1, but we consider the rest of the mempool by feerate
>>> rather than fees.
>>>
>>> ### Mining Score of a Mempool Transaction
>>>
>>> We are often interested in finding out what
>>> the "mining score" of a transaction in the mempool is. That is, when
>>> the transaction is considered in block template building, what is the
>>> feerate it is considered at?
>>>
>>> Obviously, it's not the transaction's individual feerate. Bitcoin Core
>>> [mining code sorts][14] transactions by their ancestor feerate and
>>> includes them packages at a time, keeping track of how this affects the
>>> package feerates of remaining transactions in the mempool.
>>>
>>> *ancestor feerate*: Ancestor feerate is easily accessible information,
>>> but it's not accurate either, because it doesn't take into account the
>>> fact that subsets of a transaction's ancestor set can be included
>>> without it. For example, ancestors may have high feerates on their own
>>> or we may have [high feerate siblings][8].
>>>
>>> TLDR: *Looking at the current ancestor feerate of a transaction is
>>> insufficient to tell us what feerate it will be considered at when
>>> building a block template in the future.*
>>>
>>> *min(individual feerate, ancestor feerate)*: Another
>>> heuristic that is simple to calculate based on current mempool tooling
>>> is to use the [minimum of a transaction's individual score and its
>>> ancestor score][10] as a conservative measure. But this can
>>> overestimate as well (see the example below).
>>>
>>> *min ancestor feerate(tx + possible ancestor subsets)* We can also
>>> take the minimum of every possible ancestor subset, but this can be
>>> computationally expensive since there can be lots and lots of ancestor
>>> subsets.
>>>
>>> *max ancestor feerate(tx + possible descendant subsets)*: Another idea
>>> is to use the [maximum ancestor score of the transaction + each of its
>>> descendants][9]. This doesn't work either; it has the same blindspot
>>> of ancestor subsets being mined on their own.
>>>
>>> #### Mining Score Example
>>>
>>> Here's an example illustrating why mining score is tricky to
>>> efficiently calculate for mempool transactions:
>>>
>>> Let's say you have same-size transactions A (21sat/vB), B (1sat/vB),
>>> C(9sat/vB), D(5sat/vB).
>>> The layout is: grandparent A, parent B, and two children C and D.
>>>
>>> ```
>>> A
>>> ^
>>> B
>>> ^ ^
>>> C D
>>> ```
>>>
>>> A miner using ancestor packages to build block templates will first
>>> include A with a mining score of 21. Next, the miner will include B and
>>> C with a mining score of 6. This leaves D, with a mining score of 5.
>>>
>>> Note: in this case, mining by ancestor feerate results in the most
>>> rational decisions, but [a candidate set-based approach][10] which
>>> makes ancestor feerate much less relevant could
>>> be more advantageous in other situations.
>>>
>>> Here is a chart showing the "true" mining score alongside the values
>>> calculating using imperfect heuristics described above. All of them
>>> can overestimate or underestimate.
>>>
>>> ```
>>> A B C D
>>> mining score | 21 | 6 | 6 | 5 |
>>> ancestor feerate | 21 | 11 | 10.3 | 9 |
>>> min(individual, ancestor) | 21 | 1 | 9 | 5 |
>>> min(tx + ancestor subsets) | 21 | 1 | 5 | 3 |
>>> max(tx + descendants subsets) | 21 | 9 | 9 | 5 |
>>>
>>> ```
>>>
>>> Possibly the best solution for finding the "mining score" of a
>>> transaction is to build a block template, see what feerate each
>>> package is included at. Perhaps at some cutoff, remaining mempool
>>> transactions can be estimated using some heuristic that leans
>>> {overestimating, underestimating} depending on the situation.
>>>
>>> Mining score seems to be relevant in multiple places: Murch and I
>>> recently [found][3] that it would be very important in
>>> "ancestor-aware" funding of transactions (the wallet doesn't
>>> incorporate ancestor fees when using unconfirmed transactions in coin
>>> selection, which is a bug we want to fix).
>>>
>>> In general, it would be nice to know the exact mining priority of
>>> one's unconfirmed transaction is. I can think of a few block/mempool
>>> explorers who might want to display this information for users.
>>>
>>> ### RBF Improvement Proposals
>>>
>>> After speaking to quite a few people, here are some suggestions
>>> for improvements that I have heard:
>>>
>>> * The ancestor score of the replacement must be {5, 10, N}% higher
>>> than that of every original transaction.
>>>
>>> * The ancestor score of the replacement must be 1sat/vB higher than
>>> that of every original transaction.
>>>
>>> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>> mempool, apply the current rules (absolute fees must increase and
>>> pay for the replacement transaction's new bandwidth). Otherwise, use a
>>> feerate-only rule.
>>>
>>> * If fees don't increase, the size of the replacement transaction must
>>> decrease by at least N%.
>>>
>>> * Rate-limit how many replacements we allow per prevout.
>>>
>>> * Rate-limit transaction validation in general, per peer.
>>>
>>> Perhaps some others on the mailing list can chime in to throw other
>>> ideas into the ring and/or combine some of these rules into a sensible
>>> policy.
>>>
>>> #### Replace by Feerate Only
>>>
>>> I don't think there's going to be a single-line feerate-based
>>> rule that can incorporate everything we need.
>>> On one hand, a feerate-only approach helps eliminate the issues
>>> associated with Rule #3. On the other hand, I believe the main concern
>>> with a feerate-only approach is how to rate limit replacements. We
>>> don't want to enable an attack such as:
>>>
>>> 1. Attacker broadcasts large, low-feerate transaction, and attaches a
>>> chain of descendants.
>>>
>>> 2. The attacker replaces the transaction with a smaller but higher
>>> feerate transaction, attaching a new chain of descendants.
>>>
>>> 3. Repeat 1000 times.
>>>
>>> #### Fees in Next Block and Feerate for the Rest of the Mempool
>>>
>>> Perhaps we can look at replacements like this:
>>>
>>> 1. Calculate the directly conflicting transactions and, with their
>>> descendants, the original transactions. Check signaling. Limit the
>>> total volume (e.g. can't be more than 100 total or 1MvB or something).
>>>
>>> 2. Find which original transactions would be in the next ~1 block. The
>>> replacement must pay at least this amount + X% in absolute fees. This
>>> guarantees that the fees of the next block doesn't decrease.
>>>
>>> 3. Find which transactions would be left in the mempool after that ~1
>>> block. The replacement's feerate must be Y% higher than the maximum
>>> mining score of these transactions. This guarantees that you now have
>>> only *better* candidates in your after-this-block mempool than you did
>>> before, even if the size and fees the transactions decrease.
>>>
>>> 4. Now you have two numbers: a minimum absolute fee amount and a
>>> minimum feerate. Check to see if the replacement(s) meet these
>>> minimums. Also, a wallet would be able to ask the node "What fee and
>>> feerate would I need to put on a transaction replacing this?" and use
>>> this information to fund a replacement transaction, without needing to
>>> guess or overshoot.
>>>
>>> Obviously, there are some magic numbers missing here. X and Y are
>>> TBD constants to ensure we have some kind of rate limiting for the
>>> number of replacements allowed using some set of fees.
>>>
>>> What should they be? We can do some arithmetic to see what happens if
>>> you start with the biggest/lowest feerate transaction and do a bunch
>>> of replacements. Maybe we end up with values that are high enough to
>>> prevent abuse and make sense for applications/users that do RBF.
>>>
>>> ### Mempool Changes Need for Implementation
>>>
>>> As described in the mining score section above,
>>> we may want additional tooling to more accurately assess
>>> the economic gain of replacing transactions in our mempool.
>>>
>>> A few options have been discussed:
>>>
>>> * Calculate block templates on the fly when we need to consider a
>>> replacement. However, since replacements are [quite common][11]
>>> and the information might be useful for other things as well,
>>> it may be worth it to cache a block template.
>>>
>>> * Keep a persistent block template so that we know what transactions
>>> we would put in the next block. We need to remember the feerate
>>> at which each transaction was included in the template, because an
>>> ancestor package may be included in the same block template in
>>> multiple subsets. Transactions included earlier alter the ancestor
>>> feerate of the remaining transactions in the package. We also need
>>> to keep track of the new feerates of transactions left over.
>>>
>>> * Divide the mempool into two layers, "high feerate" and "low
>>> feerate." The high feerate layer contains ~1 block of packages with
>>> the highest ancestor feerates, and the low feerate layer contains
>>> everything else. At the edge of a block, we have a Knapsacky problem
>>> where the next highest ancestor feerate package might not fit, so we
>>> would probably want the high feerate layer ~2MvB or something to avoid
>>> underestimating the fees.
>>>
>>> ## Acknowledgements
>>>
>>> Thank you to everyone whose RBF-related suggestions, grievances,
>>> criticisms and ideas were incorporated in this document:
>>> Andrew Chow, Matt Corallo, Suhas Daftuar, Christian Decker,
>>> Mark Erhardt, Lloyd Fournier, Lisa Neigut, John Newbery,
>>> Antoine Poinsot, Antoine Riard, Larry Ruane,
>>> S3RK and Bastien Teinturier.
>>>
>>> Thanks for reading!
>>>
>>> Best,
>>> Gloria
>>>
>>> [1]:
>>> https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
>>> [2]:
>>> https://github.com/bitcoin/bitcoin/pull/23121#issuecomment-929475999
>>> [3]:
>>> https://github.com/Xekyo/bitcoin/commit/d754b0242ec69d42c570418aebf9c1335af0b8ea
>>> [4]: https://github.com/bitcoindevkit/bdk/issues/144
>>> [5]: https://github.com/bitcoindevkit/bdk/issues/414
>>> [6]:
>>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
>>> [7]:
>>> https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a#new-unconfirmed-inputs-rule-2
>>> [8]: https://github.com/bitcoin/bitcoin/pull/23121#discussion_r777131366
>>> [9]:
>>> https://github.com/bitcoin/bitcoin/pull/22290#issuecomment-865887922
>>> [10]:
>>> https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
>>> [11]:
>>> https://github.com/bitcoin/bitcoin/pull/22539#issuecomment-885763670
>>> [12]: https://github.com/bitcoin/bitcoin/pull/24007
>>> [13]:
>>> https://github.com/bitcoin/bitcoin/blob/1a369f006fd0bec373b95001ed84b480e852f191/src/wallet/feebumper.cpp#L114
>>> [14]:
>>> https://github.com/bitcoin/bitcoin/blob/cf5bb048e80d4cde8828787b266b7f5f2e3b6d7b/src/node/miner.cpp#L310-L320
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
>

[-- Attachment #2: Type: text/html, Size: 48283 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Michael Folkson]

[-- Attachment #1: Type: text/plain, Size: 38188 bytes --]

Thanks for this Bastien (and Gloria for initially posting about this).

I sympathetically skimmed the eclair PR (https://github.com/ACINQ/eclair/pull/2113) dealing with replaceable transactions fee bumping.

There will continue to be a (hopefully) friendly tug of war on this probably for the rest of Bitcoin's existence. I am sure people like Luke, Prayank etc will (rightfully) continue to raise that Lightning and other second layer protocols shouldn't demand that policy rules be changed if there is a reason (e.g. DoS vector) for those rules on the base network. But if there are rules that have no upside, introduce unnecessary complexity for no reason and make Lightning implementers like Bastien's life miserable attempting to deal with them I really hope we can make progress on removing or simplifying them.

This is why I think it is important to understand the rationales for introducing the rules in the first place (and why it is safe to remove them if indeed it is) and being as rigorous as possible on the rationales for introducing additional rules. It sounds like from Gloria's initial post we are still at a brainstorming phase (which is fine) but knowing what we know today I really hope we can learn from the mistakes of the original BIP 125, namely the Core implementation not matching the BIP and the sparse rationales for the rules. As Bastien says this is not criticizing the original BIP 125 authors, 7 years is a long time especially in Bitcoin world and they probably weren't thinking about Bastien sitting down to write an eclair PR in late 2021 (and reviewers of that PR) when they wrote the BIP in 2015.

--
Michael Folkson
Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

------- Original Message -------
On Monday, January 31st, 2022 at 3:57 PM, Bastien TEINTURIER via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi Gloria,
>
> Many thanks for raising awareness on these issues and constantly pushing
> towards finding a better model. This work will highly improve the
> security of any multi-party contract trying to build on top of bitcoin
> (because most multi-party contracts will need to have timeout conditions
> and participants will need to make some transactions confirm before a
> timeout happens - otherwise they may lose funds).
>
> For starters, let me quickly explain why the current rules are hard to
> work with in the context of lightning (but I believe most L2 protocols
> will have the same issues). Feel free to skip this part if you are
> already convinced.
>
> ## Motivation
>
> The biggest pain point is BIP 125 rule 2.
> If I need to increase the fees of a time-sensitive transaction because
> the feerate has been rising since I broadcast it, I may need to also pay
> high fees just to produce a confirmed utxo that I can use. I'm actually
> paying a high fee twice instead of once (and needlessly using on-chain
> space, our scarcest asset, because we could have avoided that additional
> transaction!).
>
> It also has some annoying "non-determinism".
> Imagine that my transaction has been evicted from my mempool because its
> feerate was too low. I could think "Great, that means I don't have to
> apply BIP 125 restrictions, I can just fund this transaction as if it
> were a new one!". But actually I do, because my transaction could still
> be in miner's mempools and I have no way of knowing it...this means that
> whenever I have broadcast a transaction, I must assume that I will
> always need to abide by whatever replacement rules the network applies.
>
> Fortunately, as far as I understand it, this rule only exists because of
> a previous implementation detail of bitcoin core, so there's simply no
> good reason to keep it.
>
> The second biggest pain point is rule 3. It prevents me from efficiently
> using my capital while it's unconfirmed. Whenever I'm using a big utxo
> to fund a transaction, I will get a big change output, and it would
> really be a waste to be unable to use that change output to fund other
> transactions. In order to be capital-efficient, I will end up creating
> descendant trees for my time-sensitive transactions. But as Gloria
> explained, replacing all my children will cost me an absurdly large
> amount of fees. So what I'm actually planning to do instead is to RBF
> one of the descendants high enough to get the whole tree confirmed.
> But if those descendants' timeouts were far in the future, that's a
> waste, I paid a lot more fees for them than I should have. I'd like to
> just replace my transaction and republish the invalidated children
> independently.
>
> Rule 4 doesn't hurt as much as the two previous ones, I don't have too
> much to say about it.
>
> To be fair to the BIP 125 authors, all of these scenarios were very hard
> to forecast at the time this BIP was created. We needed years to build
> on those rules to get a better understanding of their limitations and if
> the rationale behind them made sense in the long term.
>
> ## Proposals
>
> I believe that now is a good time to re-think those, and I really like
> Gloria's categorization of the design constraints.
>
> I'd like to propose a different way of looking at descendants that makes
> it easier to design the new rules. The way I understand it, limiting the
> impact on descendant transactions is only important for DoS protection,
> not for incentive compatibility. I would argue that after evictions,
> descendant transactions will be submitted again (because they represent
> transactions that people actually want to make), so evicting them does
> not have a negative impact on mining incentives (in a world where blocks
> are full most of the time).
>
> I'm curious to hear other people's thoughts on that. If it makes sense,
> I would propose the following very simple rules:
>
> 1. The transaction's ancestor absolute fees must be X% higher than the
> previous transaction's ancestor fees
> 2. The transaction's ancestor feerate must be Y% higher than the
> previous transaction's ancestor feerate
>
> I believe it's completely ok to require increasing both the fees and
> feerate if we don't take descendants into account, because you control
> your ancestor set - whereas the descendant set may be completely out of
> your control.
>
> This is very easy to use by wallets, because the ancestor set is easy to
> obtain. And an important point is that the ancestor set is the same in
> every mempool, whereas the descendant set is not (your mempool may have
> rejected the last descendants, while other people's mempools may still
> contain them).
>
> Because of that reason, I'd like to avoid having a rule that relies on
> some size of the replaced descendant set: it may be valid in your
> mempool but invalid in someone else's, which makes it exploitable for
> pinning attacks.
>
> I believe these rules are incentive compatible (again, if you accept
> the fact that the descendants will be re-submitted and mined as well,
> so their fees aren't lost).
>
> Can we choose X and Y so that these two rules are also DoS-resistant?
> Unfortunately I'm not sure, so maybe we'll need to add a third rule to
> address that. But before we do, can someone detail what it costs for a
> node to evict a descendant tree? Given that bitcoin core doesn't allow
> chains of more than 25 transactions, the maximum number of transactions
> being replaced will be bounded by 25 * N (where N is the number of
> outputs of the transaction being replaced). If it's just O(n) pruning of
> a graph, maybe that's ok? Or maybe we make X or Y depend on the number
> of outputs of the transaction being replaced (this would need very
> careful thoughts)?
>
> If you made it this far, thanks for reading!
> A couple of comments on the previous messages:
>
>> Currently, if we see a transaction
>> that has the same txid as one in the mempool, we reject it as a
>> duplicate, even if the feerate is much higher. It's unclear to me if
>> we have a very strong reason to change this, but noting it as a
>> limitation of our current replacement policy.
>
> I don't see a strong reason from an L2 protocol's point of view yet, but
> there are many unkown unknowns. But from a miner incentive's point of
> view, we should keep the transaction with the higher feerate, shouldn't
> we? In that case it's also a more efficient use of on-chain space, which
> is a win, right?
>
>> We might have a more-or-less long transition period during which we support both...
>
> Yes, this is a long term thing.
> Even if bitcoin core releases a new version with updated RBF rules, as a
> wallet you'll need to keep using the old rules for a long time if you
> want to be safe.
>
> But it's all the more reason to try to ship this as soon as possible,
> this way maybe our grand-children will be able to benefit from it ;)
> (just kidding on the timespan obviously).
>
> Cheers,
> Bastien
>
> Le lun. 31 janv. 2022 à 00:11, Antoine Riard via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> a écrit :
>
>> Hi Gloria,
>>
>> Thanks for this RBF sum up. Few thoughts and more context comments if it can help other readers.
>>
>>> For starters, the absolute fee pinning attack is especially
>>> problematic if we apply the same rules (i.e. Rule #3 and #4) in
>>> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>>> LN channel. The mempool is rather full, so their pre-negotiated
>>> commitment transactions' feerates would not be considered high
>>> priority by miners. Bob broadcasts his commitment transaction and
>>> attaches a very large child (100KvB with 100,000sat in fees) to his
>>> anchor output. Alice broadcasts her commitment transaction with a
>>> fee-bumping child (200vB with 50,000sat fees which is a generous
>>> 250sat/vB), but this does not meet the absolute fee requirement. She
>>> would need to add another 50,000sat to replace Bob's commitment
>>> transaction.
>>
>> Solving LN pinning attacks, what we're aiming for is enabling a fair feerate bid between the counterparties, thus either forcing the adversary to overbid or to disengage from the confirmation competition. If the replace-by-feerate rule is adopted, there shouldn't be an incentive for Bob to
>> pick up the first option. Though if he does, that's a winning outcome for Alice, as one of the commitment transactions confirms and her time-sensitive second-stage HTLC can be subsequently confirmed.
>>
>>> It's unclear to me if
>>> we have a very strong reason to change this, but noting it as a
>>> limitation of our current replacement policy. See [#24007][12].
>>
>> Deployment of Taproot opens interesting possibilities in the vaults/payment channels design space, where the tapscripts can commit to different set of timelocks/quorum of keys. Even if the pre-signed states stay symmetric, whoever is the publisher, the feerate cost to spend can fluctuate.
>>
>>> While this isn't completely broken, and the user interface is
>>> secondary to the safety of the mempool policy
>>
>> I think with L2s transaction broadcast backend, the stability and clarity of the RBF user interface is primary. What we could be worried about is a too-much complex interface easing the way for an attacker to trigger your L2 node to issue policy-invalid chain of transactions. Especially, when we consider that an attacker might have leverage on chain of transactions composition ("force broadcast of commitment A then commitment B, knowing they will share a CPFP") or even transactions size ("overload commitment A with HTLCs").
>>
>>> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>> mempool, apply the current rules (absolute fees must increase and
>>> pay for the replacement transaction's new bandwidth). Otherwise, use a
>>> feerate-only rule.
>>
>> How this new replacement rule would behave if you have a parent in the "replace-by-feerate" half but the child is in the "replace-by-fee" one ?
>>
>> If we allow the replacement of the parent based on the feerate, we might decrease the top block absolute fees.
>>
>> If we block the replacement of the parent based on the feerate because the replacement absolute fees aren't above the replaced package, we still preclude a pinning vector. The child might be low-feerate junk and even attached to a low ancestor-score branch.
>>
>> If I'm correct on this limitation, maybe we could turn off the "replace-by-fee" behavior as soon as the mempool is fulfilled with a few blocks ?
>>
>>> * Rate-limit how many replacements we allow per prevout.
>>
>> Depending on how it is implemented, though I would be concerned it introduces a new pinning vector in the context of shared-utxo. If it's a hardcoded constant, it could be exhausted by an adversary starting at the lowest acceptable feerate then slowly increasing while still not reaching
>> the top of the mempool. Same if it's time-based or block-based, no guarantee the replacement slot is honestly used by your counterparty.
>>
>> Further, an above-the-average replacement frequency might just be the reflection of your confirmation strategy reacting to block schedule or mempools historical data. As long as the feerate penalty is paid, I lean to allow replacement.
>> (One solution could be to associate per-user "tag" to the LN transactions, where each "tag" would have its own replacement slots, but privacy?)
>>
>>> * Rate-limit transaction validation in general, per peer.
>>
>> I think we could improve on the Core's new transaction requester logic. Maybe we could bind the peer announced flow based on the feerate score (modulo validation time) of the previously validated transactions from that peer ? That said, while related to RBF, it sounds to me that enhancing Core's rate-limiting transaction strategy is a whole discussion in itself [0]. Especially ensuring it's tolerant to the specific requirements of LN & consorts.
>>
>>> What should they be? We can do some arithmetic to see what happens if
>>> you start with the biggest/lowest feerate transaction and do a bunch
>>> of replacements. Maybe we end up with values that are high enough to
>>> prevent abuse and make sense for applications/users that do RBF.
>>
>> That's a good question.
>>
>> One observation is that the attacker can always renew the set of DoSy utxos to pursue the attack. So maybe we could pick up constants scaled on the block size ? That way an attacker would have to burn fees, thus deterring them from launching an attack. Even if the attackers are miners, they have to renounce their income to acquire new DoSy utxos. If a low-fee period, we could scale up the constants ?
>>
>> Overall, I think there is the deployment issue to warn of. Moving to a new set of RBF rules implies for a lot of Bitcoin applications to rewrite their RBF logics. We might have a more-or-less long transition period during which we support both...
>>
>> Cheers,
>> Antoine
>>
>> [0] https://github.com/bitcoin/bitcoin/pull/21224
>>
>> Le jeu. 27 janv. 2022 à 09:10, Gloria Zhao via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> a écrit :
>>
>>> Hi everyone,
>>>
>>> This post discusses limitations of current Bitcoin Core RBF policy and
>>> attempts to start a conversation about how we can improve it,
>>> summarizing some ideas that have been discussed. Please reply if you
>>> have any new input on issues to be solved and ideas for improvement!
>>>
>>> Just in case I've screwed up the text wrapping again, another copy can be
>>> found here: https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff
>>>
>>> ## Background
>>>
>>> Please feel free to skip this section if you are already familiar
>>> with RBF.
>>>
>>> Nodes may receive *conflicting* unconfirmed transactions, aka
>>> "double spends" of the same inputs. Instead of always keeping the
>>> first transaction, since v0.12, Bitcoin Core mempool policy has
>>> included a set of Replace-by-Fee (RBF) criteria that allows the second
>>> transaction to replace the first one and any descendants it may have.
>>>
>>> Bitcoin Core RBF policy was previously documented as BIP 125.
>>> The current RBF policy is documented [here][1]. In summary:
>>>
>>> 1. The directly conflicting transactions all signal replaceability
>>> explicitly.
>>>
>>> 2. The replacement transaction only includes an unconfirmed input if
>>> that input was included in one of the directly conflicting
>>> transactions.
>>>
>>> 3. The replacement transaction pays an absolute fee of at least the
>>> sum paid by the original transactions.
>>>
>>> 4. The additional fees pays for the replacement transaction's
>>> bandwidth at or above the rate set by the node's *incremental relay
>>> feerate*.
>>>
>>> 5. The sum of all directly conflicting transactions' descendant counts
>>> (number of transactions inclusive of itself and its descendants)
>>> does not exceed 100.
>>>
>>> We can split these rules into 3 categories/goals:
>>>
>>> - **Allow Opting Out**: Some applications/businesses are unable to
>>> handle transactions that are replaceable (e.g. merchants that use
>>> zero-confirmation transactions). We (try to) help these businesses by
>>> honoring BIP125 signaling; we won't replace transactions that have not
>>> opted in.
>>>
>>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>> accept replacement transactions which would decrease fee profits
>>> of a miner. In general, if our mempool policy deviates from what is
>>> economically rational, it's likely that the transactions in our
>>> mempool will not match the ones in miners' mempools, making our
>>> fee estimation, compact block relay, and other mempool-dependent
>>> functions unreliable. Incentive-incompatible policy may also
>>> encourage transaction submission through routes other than the p2p
>>> network, harming censorship-resistance and privacy of Bitcoin payments.
>>>
>>> - **DoS Protection**: Limit two types of DoS attacks on the node's
>>> mempool: (1) the number of times a transaction can be replaced and
>>> (2) the volume of transactions that can be evicted during a
>>> replacement.
>>>
>>> Even more abstract: our goal is to make a replacement policy that
>>> results in a useful interface for users and safe policy for
>>> node operators.
>>>
>>> ## Motivation
>>>
>>> There are a number of known problems with the current RBF policy.
>>> Many of these shortcomings exist due to mempool limitations at the
>>> time RBF was implemented or result from new types of Bitcoin usage;
>>> they are not criticisms of the original design.
>>>
>>> ### Pinning Attacks
>>>
>>> The most pressing concern is that attackers may take advantage of
>>> limitations in RBF policy to prevent other users' transactions from
>>> being mined or getting accepted as a replacement.
>>>
>>> #### SIGHASH_ANYONECANPAY Pinning
>>>
>>> BIP125#2 can be bypassed by creating intermediary transactions to be
>>> replaced together. Anyone can simply split a 1-input 1-output
>>> transaction off from the replacement transaction, then broadcast the
>>> transaction as is. This can always be done, and quite cheaply. More
>>> details in [this comment][2].
>>>
>>> In general, if a transaction is signed with SIGHASH\_ANYONECANPAY,
>>> anybody can just attach a low feerate parent to this transaction and
>>> lower its ancestor feerate. Even if you require SIGHASH\_ALL which
>>> prevents an attacker from changing any outputs, the input can be a
>>> very low amount (e.g. just above the dust limit) from a low-fee
>>> ancestor and still bring down the ancestor feerate of the transaction.
>>>
>>> TLDR: if your transaction is signed with SIGHASH\_ANYONECANPAY and
>>> signals replaceability, regardless of the feerate you broadcast at, an
>>> attacker can lower its mining priority by adding an ancestor.
>>>
>>> #### Absolute Fee
>>>
>>> The restriction of requiring replacement transactions to increase the
>>> absolute fee of the mempool has been described as "bonkers." If the
>>> original transaction has a very large descendant that pays a large
>>> amount of fees, even if it has a low feerate, the replacement
>>> transaction must now pay those fees in order to meet Rule #3.
>>>
>>> #### Package RBF
>>>
>>> There are a number of reasons why, in order to enable Package RBF, we
>>> cannot use the same criteria.
>>>
>>> For starters, the absolute fee pinning attack is especially
>>> problematic if we apply the same rules (i.e. Rule #3 and #4) in
>>> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>>> LN channel. The mempool is rather full, so their pre-negotiated
>>> commitment transactions' feerates would not be considered high
>>> priority by miners. Bob broadcasts his commitment transaction and
>>> attaches a very large child (100KvB with 100,000sat in fees) to his
>>> anchor output. Alice broadcasts her commitment transaction with a
>>> fee-bumping child (200vB with 50,000sat fees which is a generous
>>> 250sat/vB), but this does not meet the absolute fee requirement. She
>>> would need to add another 50,000sat to replace Bob's commitment
>>> transaction.
>>>
>>> Disallowing new unconfirmed inputs (Rule #2) in Package RBF would be
>>> broken for packages containing transactions already in the mempool,
>>> explained [here][7].
>>>
>>> Note: I originally [proposed][6] Package RBF using the same Rule #3
>>> and #4 before I realized how significant this pinning attack is. I'm
>>> retracting that proposal, and a new set of Package RBF rules would
>>> follow from whatever the new individual RBF rules end up being.
>>>
>>> #### Same Txid Different Witness
>>>
>>> Two transactions with the same non-witness data but different
>>> witnesses have the same txid but different wtxid, and the same fee but
>>> not necessarily the same feerate. Currently, if we see a transaction
>>> that has the same txid as one in the mempool, we reject it as a
>>> duplicate, even if the feerate is much higher. It's unclear to me if
>>> we have a very strong reason to change this, but noting it as a
>>> limitation of our current replacement policy. See [#24007][12].
>>>
>>> ### User Interface
>>>
>>> #### Using Unconfirmed UTXOs to Fund Replacements
>>>
>>> The restriction of only allowing confirmed UTXOs for funding a
>>> fee-bump (Rule #2) can hurt users trying to fee-bump their
>>> transactions and complicate wallet implementations. If the original
>>> transaction's output value isn't sufficient to fund a fee-bump and/or
>>> all of the user's other UTXOs are unconfirmed, they might not be able
>>> to fund a replacement transaction. Wallet developers also need to
>>> treat self-owned unconfirmed UTXOs as unusable for fee-bumping, which
>>> adds complexity to wallet logic. For example, see BDK issues [#144][4]
>>> and [#414][5].
>>>
>>> #### Interface Not Suitable for Coin Selection
>>>
>>> Currently, a user cannot simply create a replacement transaction
>>> targeting a specific feerate or meeting a minimum fee amount and
>>> expect to meet the RBF criteria. The fee amount depends on the size of
>>> the replacement transaction, and feerate is almost irrelevant.
>>>
>>> Bitcoin Core's `bumpfee` doesn't use the RBF rules when funding the
>>> replacement. It [estimates][13] a feerate which is "wallet incremental
>>> relay fee" (a conservative overestimation of the node's incremental
>>> relay fee) higher than the original transaction, selects coins for
>>> that feerate, and hopes that it meets the RBF rules. It never fails
>>> Rule #3 and #4 because it uses all original inputs and refuses to
>>> bump a transaction with mempool descendants.
>>>
>>> This is suboptimal, but is designed to work with the coin selection
>>> engine: select a feerate first, and then add fees to cover it.
>>> Following the exact RBF rules would require working the other way
>>> around: based on how much fees we've added to the transaction and its
>>> current size, calculate the feerate to see if we meet Rule #4.
>>>
>>> While this isn't completely broken, and the user interface is
>>> secondary to the safety of the mempool policy, we can do much better.
>>> A much more user-friendly interface would depend *only* on the
>>> fee and size of the original transactions.
>>>
>>> ### Updates to Mempool and Mining
>>>
>>> Since RBF was first implemented, a number of improvements have been
>>> made to mempool and mining logic. For example, we now use ancestor
>>> feerates in mining (allowing CPFP), and keep track of ancestor
>>> packages in the mempool.
>>>
>>> ## Ideas for Improvements
>>>
>>> ### Goals
>>>
>>> To summarize, these seem to be desired changes, in order of priority:
>>>
>>> 1. Remove Rule #3. The replacement should not be *required* to pay
>>> higher absolute fees.
>>>
>>> 2. Make it impossible for a replacement transaction to have a lower
>>> mining score than the original transaction(s). This would eliminate
>>> the `SIGHASH\_ANYONECANPAY` pinning attack.
>>>
>>> 3. Remove Rule #2. Adding new unconfirmed inputs should be allowed.
>>>
>>> 4. Create a more helpful interface that helps wallet fund replacement
>>> transactions that aim for a feerate and fee.
>>>
>>> ### A Different Model for Fees
>>>
>>> For incentive compatibility, I believe there are different
>>> formulations we should consider. Most importantly, if we want to get
>>> rid of the absolute fee rule, we can no longer think of it as "the
>>> transaction needs to pay for its own bandwidth," since we won't always
>>> be getting additional fees. That means we need a new method of
>>> rate-limiting replacements that doesn't require additional fees every
>>> time.
>>>
>>> While it makes sense to think about monetary costs when launching a
>>> specific type of attack, given that the fees are paid to the miner and
>>> not to the mempool operators, maybe it doesn't make much sense to
>>> think about "paying for bandwidth". Maybe we should implement
>>> transaction validation rate-limiting differently, e.g. building it
>>> into the P2P layer instead of the mempool policy layer.
>>>
>>> Recently, Suhas gave a [formulation][8] for incentive compatibility
>>> that made sense to me: "are the fees expected to be paid in the next
>>> (N?) blocks higher or lower if we process this transaction?"
>>>
>>> I started by thinking about this where N=1 or `1 + p`.
>>> Here, a rational miner is looking at what fees they would
>>> collect in the next block, and then some proportion `p` of the rest of
>>> the blocks based on their hashrate. We're assuming `p` isn't *so high*
>>> that they would be okay with lower absolute fees in the next 1 block.
>>> We're also assuming `p` isn't *so low* that the miner doesn't care
>>> about what's left of the mempool after this block.
>>>
>>> A tweak to this formulation is "if we process this transaction, would
>>> the fees in the next 1 block higher or lower, and is the feerate
>>> density of the rest of the mempool higher or lower?" This is pretty
>>> similar, where N=1, but we consider the rest of the mempool by feerate
>>> rather than fees.
>>>
>>> ### Mining Score of a Mempool Transaction
>>>
>>> We are often interested in finding out what
>>> the "mining score" of a transaction in the mempool is. That is, when
>>> the transaction is considered in block template building, what is the
>>> feerate it is considered at?
>>>
>>> Obviously, it's not the transaction's individual feerate. Bitcoin Core
>>> [mining code sorts][14] transactions by their ancestor feerate and
>>> includes them packages at a time, keeping track of how this affects the
>>> package feerates of remaining transactions in the mempool.
>>>
>>> *ancestor feerate*: Ancestor feerate is easily accessible information,
>>> but it's not accurate either, because it doesn't take into account the
>>> fact that subsets of a transaction's ancestor set can be included
>>> without it. For example, ancestors may have high feerates on their own
>>> or we may have [high feerate siblings][8].
>>>
>>> TLDR: *Looking at the current ancestor feerate of a transaction is
>>> insufficient to tell us what feerate it will be considered at when
>>> building a block template in the future.*
>>>
>>> *min(individual feerate, ancestor feerate)*: Another
>>> heuristic that is simple to calculate based on current mempool tooling
>>> is to use the [minimum of a transaction's individual score and its
>>> ancestor score][10] as a conservative measure. But this can
>>> overestimate as well (see the example below).
>>>
>>> *min ancestor feerate(tx + possible ancestor subsets)* We can also
>>> take the minimum of every possible ancestor subset, but this can be
>>> computationally expensive since there can be lots and lots of ancestor
>>> subsets.
>>>
>>> *max ancestor feerate(tx + possible descendant subsets)*: Another idea
>>> is to use the [maximum ancestor score of the transaction + each of its
>>> descendants][9]. This doesn't work either; it has the same blindspot
>>> of ancestor subsets being mined on their own.
>>>
>>> #### Mining Score Example
>>>
>>> Here's an example illustrating why mining score is tricky to
>>> efficiently calculate for mempool transactions:
>>>
>>> Let's say you have same-size transactions A (21sat/vB), B (1sat/vB),
>>> C(9sat/vB), D(5sat/vB).
>>> The layout is: grandparent A, parent B, and two children C and D.
>>>
>>> ```
>>> A
>>> ^
>>> B
>>> ^ ^
>>> C D
>>> ```
>>>
>>> A miner using ancestor packages to build block templates will first
>>> include A with a mining score of 21. Next, the miner will include B and
>>> C with a mining score of 6. This leaves D, with a mining score of 5.
>>>
>>> Note: in this case, mining by ancestor feerate results in the most
>>> rational decisions, but [a candidate set-based approach][10] which
>>> makes ancestor feerate much less relevant could
>>> be more advantageous in other situations.
>>>
>>> Here is a chart showing the "true" mining score alongside the values
>>> calculating using imperfect heuristics described above. All of them
>>> can overestimate or underestimate.
>>>
>>> ```
>>> A B C D
>>> mining score | 21 | 6 | 6 | 5 |
>>> ancestor feerate | 21 | 11 | 10.3 | 9 |
>>> min(individual, ancestor) | 21 | 1 | 9 | 5 |
>>> min(tx + ancestor subsets) | 21 | 1 | 5 | 3 |
>>> max(tx + descendants subsets) | 21 | 9 | 9 | 5 |
>>>
>>> ```
>>>
>>> Possibly the best solution for finding the "mining score" of a
>>> transaction is to build a block template, see what feerate each
>>> package is included at. Perhaps at some cutoff, remaining mempool
>>> transactions can be estimated using some heuristic that leans
>>> {overestimating, underestimating} depending on the situation.
>>>
>>> Mining score seems to be relevant in multiple places: Murch and I
>>> recently [found][3] that it would be very important in
>>> "ancestor-aware" funding of transactions (the wallet doesn't
>>> incorporate ancestor fees when using unconfirmed transactions in coin
>>> selection, which is a bug we want to fix).
>>>
>>> In general, it would be nice to know the exact mining priority of
>>> one's unconfirmed transaction is. I can think of a few block/mempool
>>> explorers who might want to display this information for users.
>>>
>>> ### RBF Improvement Proposals
>>>
>>> After speaking to quite a few people, here are some suggestions
>>> for improvements that I have heard:
>>>
>>> * The ancestor score of the replacement must be {5, 10, N}% higher
>>> than that of every original transaction.
>>>
>>> * The ancestor score of the replacement must be 1sat/vB higher than
>>> that of every original transaction.
>>>
>>> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>> mempool, apply the current rules (absolute fees must increase and
>>> pay for the replacement transaction's new bandwidth). Otherwise, use a
>>> feerate-only rule.
>>>
>>> * If fees don't increase, the size of the replacement transaction must
>>> decrease by at least N%.
>>>
>>> * Rate-limit how many replacements we allow per prevout.
>>>
>>> * Rate-limit transaction validation in general, per peer.
>>>
>>> Perhaps some others on the mailing list can chime in to throw other
>>> ideas into the ring and/or combine some of these rules into a sensible
>>> policy.
>>>
>>> #### Replace by Feerate Only
>>>
>>> I don't think there's going to be a single-line feerate-based
>>> rule that can incorporate everything we need.
>>> On one hand, a feerate-only approach helps eliminate the issues
>>> associated with Rule #3. On the other hand, I believe the main concern
>>> with a feerate-only approach is how to rate limit replacements. We
>>> don't want to enable an attack such as:
>>>
>>> 1. Attacker broadcasts large, low-feerate transaction, and attaches a
>>> chain of descendants.
>>>
>>> 2. The attacker replaces the transaction with a smaller but higher
>>> feerate transaction, attaching a new chain of descendants.
>>>
>>> 3. Repeat 1000 times.
>>>
>>> #### Fees in Next Block and Feerate for the Rest of the Mempool
>>>
>>> Perhaps we can look at replacements like this:
>>>
>>> 1. Calculate the directly conflicting transactions and, with their
>>> descendants, the original transactions. Check signaling. Limit the
>>> total volume (e.g. can't be more than 100 total or 1MvB or something).
>>>
>>> 2. Find which original transactions would be in the next ~1 block. The
>>> replacement must pay at least this amount + X% in absolute fees. This
>>> guarantees that the fees of the next block doesn't decrease.
>>>
>>> 3. Find which transactions would be left in the mempool after that ~1
>>> block. The replacement's feerate must be Y% higher than the maximum
>>> mining score of these transactions. This guarantees that you now have
>>> only *better* candidates in your after-this-block mempool than you did
>>> before, even if the size and fees the transactions decrease.
>>>
>>> 4. Now you have two numbers: a minimum absolute fee amount and a
>>> minimum feerate. Check to see if the replacement(s) meet these
>>> minimums. Also, a wallet would be able to ask the node "What fee and
>>> feerate would I need to put on a transaction replacing this?" and use
>>> this information to fund a replacement transaction, without needing to
>>> guess or overshoot.
>>>
>>> Obviously, there are some magic numbers missing here. X and Y are
>>> TBD constants to ensure we have some kind of rate limiting for the
>>> number of replacements allowed using some set of fees.
>>>
>>> What should they be? We can do some arithmetic to see what happens if
>>> you start with the biggest/lowest feerate transaction and do a bunch
>>> of replacements. Maybe we end up with values that are high enough to
>>> prevent abuse and make sense for applications/users that do RBF.
>>>
>>> ### Mempool Changes Need for Implementation
>>>
>>> As described in the mining score section above,
>>> we may want additional tooling to more accurately assess
>>> the economic gain of replacing transactions in our mempool.
>>>
>>> A few options have been discussed:
>>>
>>> * Calculate block templates on the fly when we need to consider a
>>> replacement. However, since replacements are [quite common][11]
>>> and the information might be useful for other things as well,
>>> it may be worth it to cache a block template.
>>>
>>> * Keep a persistent block template so that we know what transactions
>>> we would put in the next block. We need to remember the feerate
>>> at which each transaction was included in the template, because an
>>> ancestor package may be included in the same block template in
>>> multiple subsets. Transactions included earlier alter the ancestor
>>> feerate of the remaining transactions in the package. We also need
>>> to keep track of the new feerates of transactions left over.
>>>
>>> * Divide the mempool into two layers, "high feerate" and "low
>>> feerate." The high feerate layer contains ~1 block of packages with
>>> the highest ancestor feerates, and the low feerate layer contains
>>> everything else. At the edge of a block, we have a Knapsacky problem
>>> where the next highest ancestor feerate package might not fit, so we
>>> would probably want the high feerate layer ~2MvB or something to avoid
>>> underestimating the fees.
>>>
>>> ## Acknowledgements
>>>
>>> Thank you to everyone whose RBF-related suggestions, grievances,
>>> criticisms and ideas were incorporated in this document:
>>> Andrew Chow, Matt Corallo, Suhas Daftuar, Christian Decker,
>>> Mark Erhardt, Lloyd Fournier, Lisa Neigut, John Newbery,
>>> Antoine Poinsot, Antoine Riard, Larry Ruane,
>>>
>>> S3RK and Bastien Teinturier.
>>>
>>> Thanks for reading!
>>>
>>> Best,
>>> Gloria
>>> [1]: https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
>>> [2]: https://github.com/bitcoin/bitcoin/pull/23121#issuecomment-929475999
>>> [3]: https://github.com/Xekyo/bitcoin/commit/d754b0242ec69d42c570418aebf9c1335af0b8ea
>>> [4]: https://github.com/bitcoindevkit/bdk/issues/144
>>> [5]: https://github.com/bitcoindevkit/bdk/issues/414
>>> [6]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
>>> [7]: https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a#new-unconfirmed-inputs-rule-2
>>> [8]: https://github.com/bitcoin/bitcoin/pull/23121#discussion_r777131366
>>> [9]: https://github.com/bitcoin/bitcoin/pull/22290#issuecomment-865887922
>>> [10]: https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
>>> [11]: https://github.com/bitcoin/bitcoin/pull/22539#issuecomment-885763670
>>> [12]: https://github.com/bitcoin/bitcoin/pull/24007
>>> [13]: https://github.com/bitcoin/bitcoin/blob/1a369f006fd0bec373b95001ed84b480e852f191/src/wallet/feebumper.cpp#L114
>>>
>>> [14]: https://github.com/bitcoin/bitcoin/blob/cf5bb048e80d4cde8828787b266b7f5f2e3b6d7b/src/node/miner.cpp#L310-L320
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 44594 bytes --]

Re: [bitcoin-dev] Improving RBF policy

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 3651 bytes --]


> On Feb 1, 2022, at 00:32, Bram Cohen <bram@chia.net> wrote:
> 
>> On Mon, Jan 31, 2022 at 4:08 PM Eric Voskuil <eric@voskuil.org> wrote:
>> 
>> 
>>>> On Jan 31, 2022, at 15:15, Bram Cohen via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>> Is it still verboten to acknowledge that RBF is normal behavior and disallowing it is the feature, and that feature is mostly there to appease some people's delusions that zeroconf is a thing? It seems a bit overdue to disrespect the RBF flag in the direction of always assuming it's on.
>> What flag?
> 
> The opt-in RBF flag in transactions.

Was being facetious. The “disrespect” referred to above assumes respect that some implementations have never given.

>>> There are two different common regimes which result in different incentivized behavior. One of them is that there's more than a block's backlog in the mempool in which case between two conflicting transactions the one with the higher fee rate should win. In the other case where there isn't a whole block's worth of transactions the one with higher total value should win.
>> These are not distinct scenarios. The rational choice is the highest fee block-valid subgraph of the set of unconfirmed transactions, in both cases (within the limits of what is computationally feasible of course).
> 
> It's weird because which of two or more conflicting transactions should win can oscillate back and forth depending on other stuff going on in the mempool.

The assumption of RAM storage is an error and unrelated to network protocol. There is nothing “going on” in a set of unconfirmed valid transactions. They are logically unchanging.

> There's already a bit of that with child pays but this is stranger and has more oddball edge cases about which transactions to route.

There’s really no such thing. The p2p network is necessarily permissionless. A person can route whatever he wants. Presumably people will not generally waste their own bandwidth by routing what they believe to be unconfirmable. And whatever they would retain themselves is their presumption of confirmable.

This decision of what to retain one’s self is just a graph traversal to determine the most valuable subset - an optimizing CSP (though generally suboptimal due to the time constraint).

Short of DoS, the most profitable model is to retain *all* valid transactions. [Note that a spend conflict is not an invalidity. Two valid transactions can be confirmed in sibling branch blocks - both valid in some context.]

So the only consideration is low cost storage fill. The fee is a proof of spend, which like proof of work (for headers/blocks), is the basis of DoS protection (for unconfirmed transactions). The issue with two conflicting subgraphs is that one or the other is ultimately unspendable. As such the fee on each is non-cumulative and therefore only one (the highest) is providing DoS protection. Any subsequent conflicting subgraph must pay not only for itself, but for all preceding conflicting subgraphs.

This pays for the storage, which is a trade accepted by the owner of the node in order to have a preview of confirmable transactions. This supports both mining generation of candidate blocks and rapid validation/confirmation of blocks.

It’s a rather straightforward system when considered in terms of how it actually works (ie from a consensus standpoint). The only p2p issue is the need to package transactions for consideration as a set, as otherwise parents may be discarded before children can pay for them. Any set up to a full block is entirely reasonable for consideration.

e

[-- Attachment #2: Type: text/html, Size: 5317 bytes --]

Re: [bitcoin-dev] Improving RBF policy

[Bram Cohen]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On Mon, Jan 31, 2022 at 4:08 PM Eric Voskuil <eric@voskuil.org> wrote:

>
>
> On Jan 31, 2022, at 15:15, Bram Cohen via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> Is it still verboten to acknowledge that RBF is normal behavior and
> disallowing it is the feature, and that feature is mostly there to appease
> some people's delusions that zeroconf is a thing? It seems a bit overdue to
> disrespect the RBF flag in the direction of always assuming it's on.
>
> What flag?
>

The opt-in RBF flag in transactions.


> There are two different common regimes which result in different
> incentivized behavior. One of them is that there's more than a block's
> backlog in the mempool in which case between two conflicting transactions
> the one with the higher fee rate should win. In the other case where there
> isn't a whole block's worth of transactions the one with higher total value
> should win.
>
> These are not distinct scenarios. The rational choice is the highest fee
> block-valid subgraph of the set of unconfirmed transactions, in both cases
> (within the limits of what is computationally feasible of course).
>

It's weird because which of two or more conflicting transactions should win
can oscillate back and forth depending on other stuff going on in the
mempool. There's already a bit of that with child pays but this is stranger
and has more oddball edge cases about which transactions to route.

[-- Attachment #2: Type: text/html, Size: 2473 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Anthony Towns]

On Mon, Jan 31, 2022 at 04:57:52PM +0100, Bastien TEINTURIER via bitcoin-dev wrote:
> I'd like to propose a different way of looking at descendants that makes
> it easier to design the new rules. The way I understand it, limiting the
> impact on descendant transactions is only important for DoS protection,
> not for incentive compatibility. I would argue that after evictions,
> descendant transactions will be submitted again (because they represent
> transactions that people actually want to make),

I think that's backwards: we're trying to discourage people from wasting
the network's bandwidth, which they would do by publishing transactions
that will never get confirmed -- if they were to eventually get confirmed
it wouldn't be a waste of bandwith, after all. But if the original
descendent txs were that sort of spam, then they may well not be
submitted again if the ancestor tx reaches a fee rate that's actually
likely to confirm.

I wonder sometimes if it could be sufficient to just have a relay rate
limit and prioritise by ancestor feerate though. Maybe something like:

 - instead of adding txs to each peers setInventoryTxToSend immediately,
   set a mempool flag "relayed=false"

 - on a time delay, add the top N (by fee rate) "relayed=false" txs to
   each peer's setInventoryTxToSend and mark them as "relayed=true";
   calculate how much kB those txs were, and do this again after
   SIZE/RATELIMIT seconds

 - don't include "relayed=false" txs when building blocks?

 - keep high-feerate evicted txs around for a while in case they get
   mined by someone else to improve compact block relay, a la the
   orphan pool?

That way if the network is busy, any attempt to do low fee rate tx spam
will just cause those txs to sit as relayed=false until they're replaced
or the network becomes less busy and they're worth relaying. And your
actual mempool accept policy can just be "is this tx a higher fee rate
than the txs it replaces"...

> Even if bitcoin core releases a new version with updated RBF rules, as a
> wallet you'll need to keep using the old rules for a long time if you
> want to be safe.

All you need is for there to be *a* path that follows the new relay rules
and gets from your node/wallet to perhaps 10% of hashpower, which seems
like something wallet providers could construct relatively quickly?

Cheers,
aj

Re: [bitcoin-dev] Improving RBF policy

[Antoine Riard]

[-- Attachment #1: Type: text/plain, Size: 3090 bytes --]

> Is it still verboten to acknowledge that RBF is normal behavior and
disallowing it is the feature, and that feature is mostly there to appease
some people's delusions that zeroconf is a thing? It seems a bit overdue to
disrespect the RBF flag in the direction of always assuming it's on.

If you're thinking about the opt-in flag, not the RBF rules, please see
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-June/019074.html
The latest state of the discussion is here :
https://gnusha.org/bitcoin-core-dev/2021-10-21.log
A gradual, multi-year deprecation sounds to be preferred to ease adaptation
of the affected Bitcoin applications.

Ultimately, I think it might not be the last time we have to change
high-impact tx-relay/mempool rules and a more formalized Core policy
deprecation process would be good.



Le lun. 31 janv. 2022 à 18:15, Bram Cohen via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> a écrit :

> Gloria Zhao wrote:
>
>>
>> This post discusses limitations of current Bitcoin Core RBF policy and
>> attempts to start a conversation about how we can improve it,
>> summarizing some ideas that have been discussed. Please reply if you
>> have any new input on issues to be solved and ideas for improvement!
>>
>
> Is it still verboten to acknowledge that RBF is normal behavior and
> disallowing it is the feature, and that feature is mostly there to appease
> some people's delusions that zeroconf is a thing? It seems a bit overdue to
> disrespect the RBF flag in the direction of always assuming it's on.
>
>
>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>   accept replacement transactions which would decrease fee profits
>>   of a miner. In general, if our mempool policy deviates from what is
>> economically rational, it's likely that the transactions in our
>> mempool will not match the ones in miners' mempools, making our
>> fee estimation, compact block relay, and other mempool-dependent
>> functions unreliable. Incentive-incompatible policy may also
>> encourage transaction submission through routes other than the p2p
>> network, harming censorship-resistance and privacy of Bitcoin payments.
>>
>
> There are two different common regimes which result in different
> incentivized behavior. One of them is that there's more than a block's
> backlog in the mempool in which case between two conflicting transactions
> the one with the higher fee rate should win. In the other case where there
> isn't a whole block's worth of transactions the one with higher total value
> should win. It would be nice to have consolidated logic which handles both,
> it seems the issue has to do with the slope of the supply/demand curve
> which in the first case is gentle enough to keep the one transaction from
> hitting the rate but in the second one is basically infinite.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 4240 bytes --]

Re: [bitcoin-dev] Improving RBF policy

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 2159 bytes --]



> On Jan 31, 2022, at 15:15, Bram Cohen via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:


…

> Is it still verboten to acknowledge that RBF is normal behavior and disallowing it is the feature, and that feature is mostly there to appease some people's delusions that zeroconf is a thing? It seems a bit overdue to disrespect the RBF flag in the direction of always assuming it's on.

What flag?

>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>   accept replacement transactions which would decrease fee profits
>>   of a miner. In general, if our mempool policy deviates from what is
>> economically rational, it's likely that the transactions in our
>> mempool will not match the ones in miners' mempools, making our
>> fee estimation, compact block relay, and other mempool-dependent
>> functions unreliable. Incentive-incompatible policy may also
>> encourage transaction submission through routes other than the p2p
>> network, harming censorship-resistance and privacy of Bitcoin payments.
> 
> There are two different common regimes which result in different incentivized behavior. One of them is that there's more than a block's backlog in the mempool in which case between two conflicting transactions the one with the higher fee rate should win. In the other case where there isn't a whole block's worth of transactions the one with higher total value should win.

These are not distinct scenarios. The rational choice is the highest fee block-valid subgraph of the set of unconfirmed transactions, in both cases (within the limits of what is computationally feasible of course).

When collecting pooled txs the only issue is DoS protection, which is simply a question of what any given miner is willing to pay, in terms of disk space, to archive conflicts for the opportunity to optimize block reward.

> It would be nice to have consolidated logic which handles both, it seems the issue has to do with the slope of the supply/demand curve which in the first case is gentle enough to keep the one transaction from hitting the rate but in the second one is basically infinite.

[-- Attachment #2: Type: text/html, Size: 3237 bytes --]

[bitcoin-dev] Improving RBF policy

[Bram Cohen]

[-- Attachment #1: Type: text/plain, Size: 1857 bytes --]

Gloria Zhao wrote:

>
> This post discusses limitations of current Bitcoin Core RBF policy and
> attempts to start a conversation about how we can improve it,
> summarizing some ideas that have been discussed. Please reply if you
> have any new input on issues to be solved and ideas for improvement!
>

Is it still verboten to acknowledge that RBF is normal behavior and
disallowing it is the feature, and that feature is mostly there to appease
some people's delusions that zeroconf is a thing? It seems a bit overdue to
disrespect the RBF flag in the direction of always assuming it's on.


> - **Incentive Compatibility**: Ensure that our RBF policy would not
>   accept replacement transactions which would decrease fee profits
>   of a miner. In general, if our mempool policy deviates from what is
> economically rational, it's likely that the transactions in our
> mempool will not match the ones in miners' mempools, making our
> fee estimation, compact block relay, and other mempool-dependent
> functions unreliable. Incentive-incompatible policy may also
> encourage transaction submission through routes other than the p2p
> network, harming censorship-resistance and privacy of Bitcoin payments.
>

There are two different common regimes which result in different
incentivized behavior. One of them is that there's more than a block's
backlog in the mempool in which case between two conflicting transactions
the one with the higher fee rate should win. In the other case where there
isn't a whole block's worth of transactions the one with higher total value
should win. It would be nice to have consolidated logic which handles both,
it seems the issue has to do with the slope of the supply/demand curve
which in the first case is gentle enough to keep the one transaction from
hitting the rate but in the second one is basically infinite.

[-- Attachment #2: Type: text/html, Size: 2326 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Bastien TEINTURIER]

[-- Attachment #1: Type: text/plain, Size: 36506 bytes --]

Hi Gloria,

Many thanks for raising awareness on these issues and constantly pushing
towards finding a better model. This work will highly improve the
security of any multi-party contract trying to build on top of bitcoin
(because most multi-party contracts will need to have timeout conditions
and participants will need to make some transactions confirm before a
timeout happens - otherwise they may lose funds).

For starters, let me quickly explain why the current rules are hard to
work with in the context of lightning (but I believe most L2 protocols
will have the same issues). Feel free to skip this part if you are
already convinced.

## Motivation

The biggest pain point is BIP 125 rule 2.
If I need to increase the fees of a time-sensitive transaction because
the feerate has been rising since I broadcast it, I may need to also pay
high fees just to produce a confirmed utxo that I can use. I'm actually
paying a high fee twice instead of once (and needlessly using on-chain
space, our scarcest asset, because we could have avoided that additional
transaction!).

It also has some annoying "non-determinism".
Imagine that my transaction has been evicted from my mempool because its
feerate was too low. I could think "Great, that means I don't have to
apply BIP 125 restrictions, I can just fund this transaction as if it
were a new one!". But actually I do, because my transaction could still
be in miner's mempools and I have no way of knowing it...this means that
whenever I have broadcast a transaction, I must assume that I will
always need to abide by whatever replacement rules the network applies.

Fortunately, as far as I understand it, this rule only exists because of
a previous implementation detail of bitcoin core, so there's simply no
good reason to keep it.

The second biggest pain point is rule 3. It prevents me from efficiently
using my capital while it's unconfirmed. Whenever I'm using a big utxo
to fund a transaction, I will get a big change output, and it would
really be a waste to be unable to use that change output to fund other
transactions. In order to be capital-efficient, I will end up creating
descendant trees for my time-sensitive transactions. But as Gloria
explained, replacing all my children will cost me an absurdly large
amount of fees. So what I'm actually planning to do instead is to RBF
one of the descendants high enough to get the whole tree confirmed.
But if those descendants' timeouts were far in the future, that's a
waste, I paid a lot more fees for them than I should have. I'd like to
just replace my transaction and republish the invalidated children
independently.

Rule 4 doesn't hurt as much as the two previous ones, I don't have too
much to say about it.

To be fair to the BIP 125 authors, all of these scenarios were very hard
to forecast at the time this BIP was created. We needed years to build
on those rules to get a better understanding of their limitations and if
the rationale behind them made sense in the long term.

## Proposals

I believe that now is a good time to re-think those, and I really like
Gloria's categorization of the design constraints.

I'd like to propose a different way of looking at descendants that makes
it easier to design the new rules. The way I understand it, limiting the
impact on descendant transactions is only important for DoS protection,
not for incentive compatibility. I would argue that after evictions,
descendant transactions will be submitted again (because they represent
transactions that people actually want to make), so evicting them does
not have a negative impact on mining incentives (in a world where blocks
are full most of the time).

I'm curious to hear other people's thoughts on that. If it makes sense,
I would propose the following very simple rules:

1. The transaction's ancestor absolute fees must be X% higher than the
previous transaction's ancestor fees
2. The transaction's ancestor feerate must be Y% higher than the
previous transaction's ancestor feerate

I believe it's completely ok to require increasing both the fees and
feerate if we don't take descendants into account, because you control
your ancestor set - whereas the descendant set may be completely out of
your control.

This is very easy to use by wallets, because the ancestor set is easy to
obtain. And an important point is that the ancestor set is the same in
every mempool, whereas the descendant set is not (your mempool may have
rejected the last descendants, while other people's mempools may still
contain them).

Because of that reason, I'd like to avoid having a rule that relies on
some size of the replaced descendant set: it may be valid in your
mempool but invalid in someone else's, which makes it exploitable for
pinning attacks.

I believe these rules are incentive compatible (again, if you accept
the fact that the descendants will be re-submitted and mined as well,
so their fees aren't lost).

Can we choose X and Y so that these two rules are also DoS-resistant?
Unfortunately I'm not sure, so maybe we'll need to add a third rule to
address that. But before we do, can someone detail what it costs for a
node to evict a descendant tree? Given that bitcoin core doesn't allow
chains of more than 25 transactions, the maximum number of transactions
being replaced will be bounded by 25 * N (where N is the number of
outputs of the transaction being replaced). If it's just O(n) pruning of
a graph, maybe that's ok? Or maybe we make X or Y depend on the number
of outputs of the transaction being replaced (this would need very
careful thoughts)?

If you made it this far, thanks for reading!
A couple of comments on the previous messages:

> Currently, if we see a transaction
> that has the same txid as one in the mempool, we reject it as a
> duplicate, even if the feerate is much higher. It's unclear to me if
> we have a very strong reason to change this, but noting it as a
> limitation of our current replacement policy.

I don't see a strong reason from an L2 protocol's point of view yet, but
there are many unkown unknowns. But from a miner incentive's point of
view, we should keep the transaction with the higher feerate, shouldn't
we? In that case it's also a more efficient use of on-chain space, which
is a win, right?

> We might have a more-or-less long transition period during which we
support both...

Yes, this is a long term thing.
Even if bitcoin core releases a new version with updated RBF rules, as a
wallet you'll need to keep using the old rules for a long time if you
want to be safe.

But it's all the more reason to try to ship this as soon as possible,
this way maybe our grand-children will be able to benefit from it ;)
(just kidding on the timespan obviously).

Cheers,
Bastien

Le lun. 31 janv. 2022 à 00:11, Antoine Riard via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> a écrit :

> Hi Gloria,
>
> Thanks for this RBF sum up. Few thoughts and more context comments if it
> can help other readers.
>
> > For starters, the absolute fee pinning attack is especially
> > problematic if we apply the same rules (i.e. Rule #3 and #4) in
> > Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
> > LN channel. The mempool is rather full, so their pre-negotiated
> > commitment transactions' feerates would not be considered high
> > priority by miners.  Bob broadcasts his commitment transaction and
> > attaches a very large child (100KvB with 100,000sat in fees) to his
> > anchor output. Alice broadcasts her commitment transaction with a
> > fee-bumping child (200vB with 50,000sat fees which is a generous
> > 250sat/vB), but this does not meet the absolute fee requirement. She
> > would need to add another 50,000sat to replace Bob's commitment
> > transaction.
>
> Solving LN pinning attacks, what we're aiming for is enabling a fair
> feerate bid between the  counterparties, thus either forcing the adversary
> to overbid or to disengage from the confirmation competition. If the
> replace-by-feerate rule is adopted, there shouldn't be an incentive for Bob
> to
> pick up the first option. Though if he does, that's a winning outcome for
> Alice, as one of the commitment transactions confirms and her
> time-sensitive second-stage HTLC can be subsequently confirmed.
>
> > It's unclear to me if
> > we have a very strong reason to change this, but noting it as a
> > limitation of our current replacement policy. See [#24007][12].
>
> Deployment of Taproot opens interesting possibilities in the
> vaults/payment channels design space, where the tapscripts can commit to
> different set of timelocks/quorum of keys. Even if the pre-signed states
> stay symmetric, whoever is the publisher, the feerate cost to spend can
> fluctuate.
>
> > While this isn't completely broken, and the user interface is
> > secondary to the safety of the mempool policy
>
> I think with L2s transaction broadcast backend, the stability and clarity
> of the RBF user interface is primary. What we could be worried about is a
> too-much complex interface easing the way for an attacker to trigger your
> L2 node to issue policy-invalid chain of transactions. Especially, when we
> consider that an attacker might have leverage on chain of transactions
> composition ("force broadcast of commitment A then commitment B, knowing
> they will share a CPFP") or even transactions size ("overload commitment A
> with HTLCs").
>
> > * If the original transaction is in the top {0.75MvB, 1MvB} of the
> >   mempool, apply the current rules (absolute fees must increase and
> > pay for the replacement transaction's new bandwidth). Otherwise, use a
> > feerate-only rule.
>
> How this new replacement rule would behave if you have a parent in the
> "replace-by-feerate" half but the child is in the "replace-by-fee" one ?
>
> If we allow the replacement of the parent based on the feerate, we might
> decrease the top block absolute fees.
>
> If we block the replacement of the parent based on the feerate because the
> replacement absolute fees aren't above the replaced package, we still
> preclude a pinning vector. The child might be low-feerate junk and even
> attached to a low ancestor-score branch.
>
> If I'm correct on this limitation, maybe we could turn off the
> "replace-by-fee" behavior as soon as the mempool is fulfilled with a few
> blocks ?
>
> > * Rate-limit how many replacements we allow per prevout.
>
> Depending on how it is implemented, though I would be concerned it
> introduces a new pinning vector in the context of shared-utxo. If it's a
> hardcoded constant, it could be exhausted by an adversary starting at the
> lowest acceptable feerate then slowly increasing while still not reaching
> the top of the mempool. Same if it's time-based or block-based, no
> guarantee the replacement slot is honestly used by your counterparty.
>
> Further, an above-the-average replacement frequency might just be the
> reflection of your confirmation strategy reacting to block schedule or
> mempools historical data. As long as the feerate penalty is paid, I lean to
> allow replacement.
>
> (One solution could be to associate per-user "tag" to the LN transactions,
> where each "tag" would have its own replacement slots, but privacy?)
>
> > * Rate-limit transaction validation in general, per peer.
>
> I think we could improve on the Core's new transaction requester logic.
> Maybe we could bind the peer announced flow based on the feerate score
> (modulo validation time) of the previously validated transactions from that
> peer ? That said, while related to RBF, it sounds to me that enhancing
> Core's rate-limiting transaction strategy is a whole discussion in itself
> [0]. Especially ensuring it's tolerant to the specific requirements of LN &
> consorts.
>
> > What should they be? We can do some arithmetic to see what happens if
> > you start with the biggest/lowest feerate transaction and do a bunch
> > of replacements. Maybe we end up with values that are high enough to
> > prevent abuse and make sense for applications/users that do RBF.
>
> That's a good question.
>
> One observation is that the attacker can always renew the set of DoSy
> utxos to pursue the attack. So maybe we could pick up constants scaled on
> the block size ? That way an attacker would have to burn fees, thus
> deterring them from launching an attack. Even if the attackers are miners,
> they have to renounce their income to acquire new DoSy utxos. If a low-fee
> period, we could scale up the constants ?
>
>
> Overall, I think there is the deployment issue to warn of. Moving to a new
> set of RBF rules implies for a lot of Bitcoin applications to rewrite their
> RBF logics. We might have a more-or-less long transition period during
> which we support both...
>
> Cheers,
> Antoine
>
> [0] https://github.com/bitcoin/bitcoin/pull/21224
>
> Le jeu. 27 janv. 2022 à 09:10, Gloria Zhao via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> a écrit :
>
>> Hi everyone,
>>
>> This post discusses limitations of current Bitcoin Core RBF policy and
>> attempts to start a conversation about how we can improve it,
>> summarizing some ideas that have been discussed. Please reply if you
>> have any new input on issues to be solved and ideas for improvement!
>>
>> Just in case I've screwed up the text wrapping again, another copy can be
>> found here:
>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff
>>
>> ## Background
>>
>> Please feel free to skip this section if you are already familiar
>> with RBF.
>>
>> Nodes may receive *conflicting* unconfirmed transactions, aka
>> "double spends" of the same inputs. Instead of always keeping the
>> first transaction, since v0.12, Bitcoin Core mempool policy has
>> included a set of Replace-by-Fee (RBF) criteria that allows the second
>> transaction to replace the first one and any descendants it may have.
>>
>> Bitcoin Core RBF policy was previously documented as BIP 125.
>> The current RBF policy is documented [here][1]. In summary:
>>
>> 1. The directly conflicting transactions all signal replaceability
>>    explicitly.
>>
>> 2. The replacement transaction only includes an unconfirmed input if
>>    that input was included in one of the directly conflicting
>> transactions.
>>
>> 3. The replacement transaction pays an absolute fee of at least the
>>    sum paid by the original transactions.
>>
>> 4. The additional fees pays for the replacement transaction's
>>    bandwidth at or above the rate set by the node's *incremental relay
>> feerate*.
>>
>> 5. The sum of all directly conflicting transactions' descendant counts
>>    (number of transactions inclusive of itself and its descendants)
>> does not exceed 100.
>>
>> We can split these rules into 3 categories/goals:
>>
>> - **Allow Opting Out**: Some applications/businesses are unable to
>>   handle transactions that are replaceable (e.g. merchants that use
>> zero-confirmation transactions). We (try to) help these businesses by
>> honoring BIP125 signaling; we won't replace transactions that have not
>> opted in.
>>
>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>   accept replacement transactions which would decrease fee profits
>>   of a miner. In general, if our mempool policy deviates from what is
>> economically rational, it's likely that the transactions in our
>> mempool will not match the ones in miners' mempools, making our
>> fee estimation, compact block relay, and other mempool-dependent
>> functions unreliable. Incentive-incompatible policy may also
>> encourage transaction submission through routes other than the p2p
>> network, harming censorship-resistance and privacy of Bitcoin payments.
>>
>> - **DoS Protection**: Limit two types of DoS attacks on the node's
>>   mempool: (1) the number of times a transaction can be replaced and
>> (2) the volume of transactions that can be evicted during a
>> replacement.
>>
>> Even more abstract: our goal is to make a replacement policy that
>> results in a useful interface for users and safe policy for
>> node operators.
>>
>> ## Motivation
>>
>> There are a number of known problems with the current RBF policy.
>> Many of these shortcomings exist due to mempool limitations at the
>> time RBF was implemented or result from new types of Bitcoin usage;
>> they are not criticisms of the original design.
>>
>> ### Pinning Attacks
>>
>> The most pressing concern is that attackers may take advantage of
>> limitations in RBF policy to prevent other users' transactions from
>> being mined or getting accepted as a replacement.
>>
>> #### SIGHASH_ANYONECANPAY Pinning
>>
>> BIP125#2 can be bypassed by creating intermediary transactions to be
>> replaced together. Anyone can simply split a 1-input 1-output
>> transaction off from the replacement transaction, then broadcast the
>> transaction as is. This can always be done, and quite cheaply. More
>> details in [this comment][2].
>>
>> In general, if a transaction is signed with SIGHASH\_ANYONECANPAY,
>> anybody can just attach a low feerate parent to this transaction and
>> lower its ancestor feerate.  Even if you require SIGHASH\_ALL which
>> prevents an attacker from changing any outputs, the input can be a
>> very low amount (e.g. just above the dust limit) from a low-fee
>> ancestor and still bring down the ancestor feerate of the transaction.
>>
>> TLDR: if your transaction is signed with SIGHASH\_ANYONECANPAY and
>> signals replaceability, regardless of the feerate you broadcast at, an
>> attacker can lower its mining priority by adding an ancestor.
>>
>> #### Absolute Fee
>>
>> The restriction of requiring replacement transactions to increase the
>> absolute fee of the mempool has been described as "bonkers." If the
>> original transaction has a very large descendant that pays a large
>> amount of fees, even if it has a low feerate, the replacement
>> transaction must now pay those fees in order to meet Rule #3.
>>
>> #### Package RBF
>>
>> There are a number of reasons why, in order to enable Package RBF, we
>> cannot use the same criteria.
>>
>> For starters, the absolute fee pinning attack is especially
>> problematic if we apply the same rules (i.e. Rule #3 and #4) in
>> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>> LN channel. The mempool is rather full, so their pre-negotiated
>> commitment transactions' feerates would not be considered high
>> priority by miners.  Bob broadcasts his commitment transaction and
>> attaches a very large child (100KvB with 100,000sat in fees) to his
>> anchor output. Alice broadcasts her commitment transaction with a
>> fee-bumping child (200vB with 50,000sat fees which is a generous
>> 250sat/vB), but this does not meet the absolute fee requirement. She
>> would need to add another 50,000sat to replace Bob's commitment
>> transaction.
>>
>> Disallowing new unconfirmed inputs (Rule #2) in Package RBF would be
>> broken for packages containing transactions already in the mempool,
>> explained [here][7].
>>
>> Note: I originally [proposed][6] Package RBF using the same Rule #3
>> and #4 before I realized how significant this pinning attack is. I'm
>> retracting that proposal, and a new set of Package RBF rules would
>> follow from whatever the new individual RBF rules end up being.
>>
>> #### Same Txid Different Witness
>>
>> Two transactions with the same non-witness data but different
>> witnesses have the same txid but different wtxid, and the same fee but
>> not necessarily the same feerate. Currently, if we see a transaction
>> that has the same txid as one in the mempool, we reject it as a
>> duplicate, even if the feerate is much higher. It's unclear to me if
>> we have a very strong reason to change this, but noting it as a
>> limitation of our current replacement policy. See [#24007][12].
>>
>> ### User Interface
>>
>> #### Using Unconfirmed UTXOs to Fund Replacements
>>
>> The restriction of only allowing confirmed UTXOs for funding a
>> fee-bump (Rule #2) can hurt users trying to fee-bump their
>> transactions and complicate wallet implementations. If the original
>> transaction's output value isn't sufficient to fund a fee-bump and/or
>> all of the user's other UTXOs are unconfirmed, they might not be able
>> to fund a replacement transaction. Wallet developers also need to
>> treat self-owned unconfirmed UTXOs as unusable for fee-bumping, which
>> adds complexity to wallet logic. For example, see BDK issues [#144][4]
>> and [#414][5].
>>
>> #### Interface Not Suitable for Coin Selection
>>
>> Currently, a user cannot simply create a replacement transaction
>> targeting a specific feerate or meeting a minimum fee amount and
>> expect to meet the RBF criteria. The fee amount depends on the size of
>> the replacement transaction, and feerate is almost irrelevant.
>>
>> Bitcoin Core's `bumpfee` doesn't use the RBF rules when funding the
>> replacement. It [estimates][13] a feerate which is "wallet incremental
>> relay fee" (a conservative overestimation of the node's incremental
>> relay fee) higher than the original transaction, selects coins for
>> that feerate, and hopes that it meets the RBF rules. It never fails
>> Rule #3 and #4 because it uses all original inputs and refuses to
>> bump a transaction with mempool descendants.
>>
>> This is suboptimal, but is designed to work with the coin selection
>> engine: select a feerate first, and then add fees to cover it.
>> Following the exact RBF rules would require working the other way
>> around: based on how much fees we've added to the transaction and its
>> current size, calculate the feerate to see if we meet Rule #4.
>>
>> While this isn't completely broken, and the user interface is
>> secondary to the safety of the mempool policy, we can do much better.
>> A much more user-friendly interface would depend *only* on the
>> fee and size of the original transactions.
>>
>> ### Updates to Mempool and Mining
>>
>> Since RBF was first implemented, a number of improvements have been
>> made to mempool and mining logic. For example, we now use ancestor
>> feerates in mining (allowing CPFP), and keep track of ancestor
>> packages in the mempool.
>>
>> ## Ideas for Improvements
>>
>> ### Goals
>>
>> To summarize, these seem to be desired changes, in order of priority:
>>
>> 1. Remove Rule #3. The replacement should not be *required* to pay
>> higher absolute fees.
>>
>> 2. Make it impossible for a replacement transaction to have a lower
>> mining score than the original transaction(s). This would eliminate
>> the `SIGHASH\_ANYONECANPAY` pinning attack.
>>
>> 3. Remove Rule #2. Adding new unconfirmed inputs should be allowed.
>>
>> 4. Create a more helpful interface that helps wallet fund replacement
>> transactions that aim for a feerate and fee.
>>
>> ### A Different Model for Fees
>>
>> For incentive compatibility, I believe there are different
>> formulations we should consider.  Most importantly, if we want to get
>> rid of the absolute fee rule, we can no longer think of it as "the
>> transaction needs to pay for its own bandwidth," since we won't always
>> be getting additional fees. That means we need a new method of
>> rate-limiting replacements that doesn't require additional fees every
>> time.
>>
>> While it makes sense to think about monetary costs when launching a
>> specific type of attack, given that the fees are paid to the miner and
>> not to the mempool operators, maybe it doesn't make much sense to
>> think about "paying for bandwidth". Maybe we should implement
>> transaction validation rate-limiting differently, e.g. building it
>> into the P2P layer instead of the mempool policy layer.
>>
>> Recently, Suhas gave a [formulation][8] for incentive compatibility
>> that made sense to me: "are the fees expected to be paid in the next
>> (N?) blocks higher or lower if we process this transaction?"
>>
>> I started by thinking about this where N=1 or `1 + p`.
>> Here, a rational miner is looking at what fees they would
>> collect in the next block, and then some proportion `p` of the rest of
>> the blocks based on their hashrate. We're assuming `p` isn't *so high*
>> that they would be okay with lower absolute fees in the next 1 block.
>> We're also assuming `p` isn't *so low* that the miner doesn't care
>> about what's left of the mempool after this block.
>>
>> A tweak to this formulation is "if we process this transaction, would
>> the fees in the next 1 block higher or lower, and is the feerate
>> density of the rest of the mempool higher or lower?" This is pretty
>> similar, where N=1, but we consider the rest of the mempool by feerate
>> rather than fees.
>>
>> ### Mining Score of a Mempool Transaction
>>
>> We are often interested in finding out what
>> the "mining score" of a transaction in the mempool is. That is, when
>> the transaction is considered in block template building, what is the
>> feerate it is considered at?
>>
>> Obviously, it's not the transaction's individual feerate. Bitcoin Core
>> [mining code sorts][14] transactions by their ancestor feerate and
>> includes them packages at a time, keeping track of how this affects the
>> package feerates of remaining transactions in the mempool.
>>
>> *ancestor feerate*: Ancestor feerate is easily accessible information,
>> but it's not accurate either, because it doesn't take into account the
>> fact that subsets of a transaction's ancestor set can be included
>> without it. For example, ancestors may have high feerates on their own
>> or we may have [high feerate siblings][8].
>>
>> TLDR: *Looking at the current ancestor feerate of a transaction is
>> insufficient to tell us what feerate it will be considered at when
>> building a block template in the future.*
>>
>> *min(individual feerate, ancestor feerate)*: Another
>> heuristic that is simple to calculate based on current mempool tooling
>> is to use the [minimum of a transaction's individual score and its
>> ancestor score][10] as a conservative measure.  But this can
>> overestimate as well (see the example below).
>>
>> *min ancestor feerate(tx + possible ancestor subsets)* We can also
>> take the minimum of every possible ancestor subset, but this can be
>> computationally expensive since there can be lots and lots of ancestor
>> subsets.
>>
>> *max ancestor feerate(tx + possible descendant subsets)*: Another idea
>> is to use the [maximum ancestor score of the transaction + each of its
>> descendants][9]. This doesn't work either; it has the same blindspot
>> of ancestor subsets being mined on their own.
>>
>> #### Mining Score Example
>>
>> Here's an example illustrating why mining score is tricky to
>> efficiently calculate for mempool transactions:
>>
>> Let's say you have same-size transactions A (21sat/vB), B (1sat/vB),
>> C(9sat/vB), D(5sat/vB).
>> The layout is: grandparent A, parent B, and two children C and D.
>>
>> ```
>>     A
>>     ^
>>     B
>>    ^ ^
>>    C D
>> ```
>>
>> A miner using ancestor packages to build block templates will first
>> include A with a mining score of 21. Next, the miner will include B and
>> C with a mining score of 6. This leaves D, with a mining score of 5.
>>
>> Note: in this case, mining by ancestor feerate results in the most
>> rational decisions, but [a candidate set-based approach][10] which
>> makes ancestor feerate much less relevant could
>> be more advantageous in other situations.
>>
>> Here is a chart showing the "true" mining score alongside the values
>> calculating using imperfect heuristics described above. All of them
>> can overestimate or underestimate.
>>
>> ```
>>    A     B       C     D
>> mining score |   21   |   6   |   6   |   5   |
>> ancestor feerate   |   21   |  11   | 10.3  |   9   |
>> min(individual, ancestor) |   21   |   1   |   9   |   5   |
>> min(tx + ancestor subsets)      |   21   |   1   |   5   |   3   |
>> max(tx + descendants subsets) |   21   |   9   |   9   |   5   |
>>
>> ```
>>
>> Possibly the best solution for finding the "mining score" of a
>> transaction is to build a block template, see what feerate each
>> package is included at. Perhaps at some cutoff, remaining mempool
>> transactions can be estimated using some heuristic that leans
>> {overestimating, underestimating} depending on the situation.
>>
>> Mining score seems to be relevant in multiple places: Murch and I
>> recently [found][3] that it would be very important in
>> "ancestor-aware" funding of transactions (the wallet doesn't
>> incorporate ancestor fees when using unconfirmed transactions in coin
>> selection, which is a bug we want to fix).
>>
>> In general, it would be nice to know the exact mining priority of
>> one's unconfirmed transaction is.  I can think of a few block/mempool
>> explorers who might want to display this information for users.
>>
>> ### RBF Improvement Proposals
>>
>> After speaking to quite a few people, here are some suggestions
>> for improvements that I have heard:
>>
>> * The ancestor score of the replacement must be {5, 10, N}% higher
>>   than that of every original transaction.
>>
>> * The ancestor score of the replacement must be 1sat/vB higher than
>>   that of every original transaction.
>>
>> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>   mempool, apply the current rules (absolute fees must increase and
>> pay for the replacement transaction's new bandwidth). Otherwise, use a
>> feerate-only rule.
>>
>> * If fees don't increase, the size of the replacement transaction must
>>   decrease by at least N%.
>>
>> * Rate-limit how many replacements we allow per prevout.
>>
>> * Rate-limit transaction validation in general, per peer.
>>
>> Perhaps some others on the mailing list can chime in to throw other
>> ideas into the ring and/or combine some of these rules into a sensible
>> policy.
>>
>> #### Replace by Feerate Only
>>
>> I don't think there's going to be a single-line feerate-based
>> rule that can incorporate everything we need.
>> On one hand, a feerate-only approach helps eliminate the issues
>> associated with Rule #3. On the other hand, I believe the main concern
>> with a feerate-only approach is how to rate limit replacements. We
>> don't want to enable an attack such as:
>>
>> 1. Attacker broadcasts large, low-feerate transaction, and attaches a
>> chain of descendants.
>>
>> 2. The attacker replaces the transaction with a smaller but higher
>> feerate transaction, attaching a new chain of descendants.
>>
>> 3. Repeat 1000 times.
>>
>> #### Fees in Next Block and Feerate for the Rest of the Mempool
>>
>> Perhaps we can look at replacements like this:
>>
>> 1. Calculate the directly conflicting transactions and, with their
>> descendants, the original transactions. Check signaling. Limit the
>> total volume (e.g. can't be more than 100 total or 1MvB or something).
>>
>> 2. Find which original transactions would be in the next ~1 block. The
>> replacement must pay at least this amount + X% in absolute fees. This
>> guarantees that the fees of the next block doesn't decrease.
>>
>> 3. Find which transactions would be left in the mempool after that ~1
>> block. The replacement's feerate must be Y% higher than the maximum
>> mining score of these transactions. This guarantees that you now have
>> only *better* candidates in your after-this-block mempool than you did
>> before, even if the size and fees the transactions decrease.
>>
>> 4. Now you have two numbers: a minimum absolute fee amount and a
>> minimum feerate. Check to see if the replacement(s) meet these
>> minimums. Also, a wallet would be able to ask the node "What fee and
>> feerate would I need to put on a transaction replacing this?" and use
>> this information to fund a replacement transaction, without needing to
>> guess or overshoot.
>>
>> Obviously, there are some magic numbers missing here. X and Y are
>> TBD constants to ensure we have some kind of rate limiting for the
>> number of replacements allowed using some set of fees.
>>
>> What should they be? We can do some arithmetic to see what happens if
>> you start with the biggest/lowest feerate transaction and do a bunch
>> of replacements. Maybe we end up with values that are high enough to
>> prevent abuse and make sense for applications/users that do RBF.
>>
>> ### Mempool Changes Need for Implementation
>>
>> As described in the mining score section above,
>> we may want additional tooling to more accurately assess
>> the economic gain of replacing transactions in our mempool.
>>
>> A few options have been discussed:
>>
>> * Calculate block templates on the fly when we need to consider a
>>   replacement. However, since replacements are [quite common][11]
>>   and the information might be useful for other things as well,
>>   it may be worth it to cache a block template.
>>
>> * Keep a persistent block template so that we know what transactions
>>   we would put in the next block. We need to remember the feerate
>> at which each transaction was included in the template, because an
>> ancestor package may be included in the same block template in
>> multiple subsets. Transactions included earlier alter the ancestor
>> feerate of the remaining transactions in the package. We also need
>> to keep track of the new feerates of transactions left over.
>>
>> * Divide the mempool into two layers, "high feerate" and "low
>>   feerate." The high feerate layer contains ~1 block of packages with
>> the highest ancestor feerates, and the low feerate layer contains
>> everything else. At the edge of a block, we have a Knapsacky problem
>> where the next highest ancestor feerate package might not fit, so we
>> would probably want the high feerate layer ~2MvB or something to avoid
>> underestimating the fees.
>>
>> ## Acknowledgements
>>
>> Thank you to everyone whose RBF-related suggestions, grievances,
>> criticisms and ideas were incorporated in this document:
>> Andrew Chow, Matt Corallo, Suhas Daftuar, Christian Decker,
>> Mark Erhardt, Lloyd Fournier, Lisa Neigut, John Newbery,
>> Antoine Poinsot, Antoine Riard, Larry Ruane,
>> S3RK and Bastien Teinturier.
>>
>> Thanks for reading!
>>
>> Best,
>> Gloria
>>
>> [1]:
>> https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
>> [2]: https://github.com/bitcoin/bitcoin/pull/23121#issuecomment-929475999
>> [3]:
>> https://github.com/Xekyo/bitcoin/commit/d754b0242ec69d42c570418aebf9c1335af0b8ea
>> [4]: https://github.com/bitcoindevkit/bdk/issues/144
>> [5]: https://github.com/bitcoindevkit/bdk/issues/414
>> [6]:
>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
>> [7]:
>> https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a#new-unconfirmed-inputs-rule-2
>> [8]: https://github.com/bitcoin/bitcoin/pull/23121#discussion_r777131366
>> [9]: https://github.com/bitcoin/bitcoin/pull/22290#issuecomment-865887922
>> [10]:
>> https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
>> [11]:
>> https://github.com/bitcoin/bitcoin/pull/22539#issuecomment-885763670
>> [12]: https://github.com/bitcoin/bitcoin/pull/24007
>> [13]:
>> https://github.com/bitcoin/bitcoin/blob/1a369f006fd0bec373b95001ed84b480e852f191/src/wallet/feebumper.cpp#L114
>> [14]:
>> https://github.com/bitcoin/bitcoin/blob/cf5bb048e80d4cde8828787b266b7f5f2e3b6d7b/src/node/miner.cpp#L310-L320
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 39894 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Antoine Riard]

[-- Attachment #1: Type: text/plain, Size: 28589 bytes --]

Hi Gloria,

Thanks for this RBF sum up. Few thoughts and more context comments if it
can help other readers.

> For starters, the absolute fee pinning attack is especially
> problematic if we apply the same rules (i.e. Rule #3 and #4) in
> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
> LN channel. The mempool is rather full, so their pre-negotiated
> commitment transactions' feerates would not be considered high
> priority by miners.  Bob broadcasts his commitment transaction and
> attaches a very large child (100KvB with 100,000sat in fees) to his
> anchor output. Alice broadcasts her commitment transaction with a
> fee-bumping child (200vB with 50,000sat fees which is a generous
> 250sat/vB), but this does not meet the absolute fee requirement. She
> would need to add another 50,000sat to replace Bob's commitment
> transaction.

Solving LN pinning attacks, what we're aiming for is enabling a fair
feerate bid between the  counterparties, thus either forcing the adversary
to overbid or to disengage from the confirmation competition. If the
replace-by-feerate rule is adopted, there shouldn't be an incentive for Bob
to
pick up the first option. Though if he does, that's a winning outcome for
Alice, as one of the commitment transactions confirms and her
time-sensitive second-stage HTLC can be subsequently confirmed.

> It's unclear to me if
> we have a very strong reason to change this, but noting it as a
> limitation of our current replacement policy. See [#24007][12].

Deployment of Taproot opens interesting possibilities in the vaults/payment
channels design space, where the tapscripts can commit to different set of
timelocks/quorum of keys. Even if the pre-signed states stay symmetric,
whoever is the publisher, the feerate cost to spend can fluctuate.

> While this isn't completely broken, and the user interface is
> secondary to the safety of the mempool policy

I think with L2s transaction broadcast backend, the stability and clarity
of the RBF user interface is primary. What we could be worried about is a
too-much complex interface easing the way for an attacker to trigger your
L2 node to issue policy-invalid chain of transactions. Especially, when we
consider that an attacker might have leverage on chain of transactions
composition ("force broadcast of commitment A then commitment B, knowing
they will share a CPFP") or even transactions size ("overload commitment A
with HTLCs").

> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>   mempool, apply the current rules (absolute fees must increase and
> pay for the replacement transaction's new bandwidth). Otherwise, use a
> feerate-only rule.

How this new replacement rule would behave if you have a parent in the
"replace-by-feerate" half but the child is in the "replace-by-fee" one ?

If we allow the replacement of the parent based on the feerate, we might
decrease the top block absolute fees.

If we block the replacement of the parent based on the feerate because the
replacement absolute fees aren't above the replaced package, we still
preclude a pinning vector. The child might be low-feerate junk and even
attached to a low ancestor-score branch.

If I'm correct on this limitation, maybe we could turn off the
"replace-by-fee" behavior as soon as the mempool is fulfilled with a few
blocks ?

> * Rate-limit how many replacements we allow per prevout.

Depending on how it is implemented, though I would be concerned it
introduces a new pinning vector in the context of shared-utxo. If it's a
hardcoded constant, it could be exhausted by an adversary starting at the
lowest acceptable feerate then slowly increasing while still not reaching
the top of the mempool. Same if it's time-based or block-based, no
guarantee the replacement slot is honestly used by your counterparty.

Further, an above-the-average replacement frequency might just be the
reflection of your confirmation strategy reacting to block schedule or
mempools historical data. As long as the feerate penalty is paid, I lean to
allow replacement.

(One solution could be to associate per-user "tag" to the LN transactions,
where each "tag" would have its own replacement slots, but privacy?)

> * Rate-limit transaction validation in general, per peer.

I think we could improve on the Core's new transaction requester logic.
Maybe we could bind the peer announced flow based on the feerate score
(modulo validation time) of the previously validated transactions from that
peer ? That said, while related to RBF, it sounds to me that enhancing
Core's rate-limiting transaction strategy is a whole discussion in itself
[0]. Especially ensuring it's tolerant to the specific requirements of LN &
consorts.

> What should they be? We can do some arithmetic to see what happens if
> you start with the biggest/lowest feerate transaction and do a bunch
> of replacements. Maybe we end up with values that are high enough to
> prevent abuse and make sense for applications/users that do RBF.

That's a good question.

One observation is that the attacker can always renew the set of DoSy utxos
to pursue the attack. So maybe we could pick up constants scaled on the
block size ? That way an attacker would have to burn fees, thus deterring
them from launching an attack. Even if the attackers are miners, they have
to renounce their income to acquire new DoSy utxos. If a low-fee period, we
could scale up the constants ?


Overall, I think there is the deployment issue to warn of. Moving to a new
set of RBF rules implies for a lot of Bitcoin applications to rewrite their
RBF logics. We might have a more-or-less long transition period during
which we support both...

Cheers,
Antoine

[0] https://github.com/bitcoin/bitcoin/pull/21224

Le jeu. 27 janv. 2022 à 09:10, Gloria Zhao via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> a écrit :

> Hi everyone,
>
> This post discusses limitations of current Bitcoin Core RBF policy and
> attempts to start a conversation about how we can improve it,
> summarizing some ideas that have been discussed. Please reply if you
> have any new input on issues to be solved and ideas for improvement!
>
> Just in case I've screwed up the text wrapping again, another copy can be
> found here:
> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff
>
> ## Background
>
> Please feel free to skip this section if you are already familiar
> with RBF.
>
> Nodes may receive *conflicting* unconfirmed transactions, aka
> "double spends" of the same inputs. Instead of always keeping the
> first transaction, since v0.12, Bitcoin Core mempool policy has
> included a set of Replace-by-Fee (RBF) criteria that allows the second
> transaction to replace the first one and any descendants it may have.
>
> Bitcoin Core RBF policy was previously documented as BIP 125.
> The current RBF policy is documented [here][1]. In summary:
>
> 1. The directly conflicting transactions all signal replaceability
>    explicitly.
>
> 2. The replacement transaction only includes an unconfirmed input if
>    that input was included in one of the directly conflicting
> transactions.
>
> 3. The replacement transaction pays an absolute fee of at least the
>    sum paid by the original transactions.
>
> 4. The additional fees pays for the replacement transaction's
>    bandwidth at or above the rate set by the node's *incremental relay
> feerate*.
>
> 5. The sum of all directly conflicting transactions' descendant counts
>    (number of transactions inclusive of itself and its descendants)
> does not exceed 100.
>
> We can split these rules into 3 categories/goals:
>
> - **Allow Opting Out**: Some applications/businesses are unable to
>   handle transactions that are replaceable (e.g. merchants that use
> zero-confirmation transactions). We (try to) help these businesses by
> honoring BIP125 signaling; we won't replace transactions that have not
> opted in.
>
> - **Incentive Compatibility**: Ensure that our RBF policy would not
>   accept replacement transactions which would decrease fee profits
>   of a miner. In general, if our mempool policy deviates from what is
> economically rational, it's likely that the transactions in our
> mempool will not match the ones in miners' mempools, making our
> fee estimation, compact block relay, and other mempool-dependent
> functions unreliable. Incentive-incompatible policy may also
> encourage transaction submission through routes other than the p2p
> network, harming censorship-resistance and privacy of Bitcoin payments.
>
> - **DoS Protection**: Limit two types of DoS attacks on the node's
>   mempool: (1) the number of times a transaction can be replaced and
> (2) the volume of transactions that can be evicted during a
> replacement.
>
> Even more abstract: our goal is to make a replacement policy that
> results in a useful interface for users and safe policy for
> node operators.
>
> ## Motivation
>
> There are a number of known problems with the current RBF policy.
> Many of these shortcomings exist due to mempool limitations at the
> time RBF was implemented or result from new types of Bitcoin usage;
> they are not criticisms of the original design.
>
> ### Pinning Attacks
>
> The most pressing concern is that attackers may take advantage of
> limitations in RBF policy to prevent other users' transactions from
> being mined or getting accepted as a replacement.
>
> #### SIGHASH_ANYONECANPAY Pinning
>
> BIP125#2 can be bypassed by creating intermediary transactions to be
> replaced together. Anyone can simply split a 1-input 1-output
> transaction off from the replacement transaction, then broadcast the
> transaction as is. This can always be done, and quite cheaply. More
> details in [this comment][2].
>
> In general, if a transaction is signed with SIGHASH\_ANYONECANPAY,
> anybody can just attach a low feerate parent to this transaction and
> lower its ancestor feerate.  Even if you require SIGHASH\_ALL which
> prevents an attacker from changing any outputs, the input can be a
> very low amount (e.g. just above the dust limit) from a low-fee
> ancestor and still bring down the ancestor feerate of the transaction.
>
> TLDR: if your transaction is signed with SIGHASH\_ANYONECANPAY and
> signals replaceability, regardless of the feerate you broadcast at, an
> attacker can lower its mining priority by adding an ancestor.
>
> #### Absolute Fee
>
> The restriction of requiring replacement transactions to increase the
> absolute fee of the mempool has been described as "bonkers." If the
> original transaction has a very large descendant that pays a large
> amount of fees, even if it has a low feerate, the replacement
> transaction must now pay those fees in order to meet Rule #3.
>
> #### Package RBF
>
> There are a number of reasons why, in order to enable Package RBF, we
> cannot use the same criteria.
>
> For starters, the absolute fee pinning attack is especially
> problematic if we apply the same rules (i.e. Rule #3 and #4) in
> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
> LN channel. The mempool is rather full, so their pre-negotiated
> commitment transactions' feerates would not be considered high
> priority by miners.  Bob broadcasts his commitment transaction and
> attaches a very large child (100KvB with 100,000sat in fees) to his
> anchor output. Alice broadcasts her commitment transaction with a
> fee-bumping child (200vB with 50,000sat fees which is a generous
> 250sat/vB), but this does not meet the absolute fee requirement. She
> would need to add another 50,000sat to replace Bob's commitment
> transaction.
>
> Disallowing new unconfirmed inputs (Rule #2) in Package RBF would be
> broken for packages containing transactions already in the mempool,
> explained [here][7].
>
> Note: I originally [proposed][6] Package RBF using the same Rule #3
> and #4 before I realized how significant this pinning attack is. I'm
> retracting that proposal, and a new set of Package RBF rules would
> follow from whatever the new individual RBF rules end up being.
>
> #### Same Txid Different Witness
>
> Two transactions with the same non-witness data but different
> witnesses have the same txid but different wtxid, and the same fee but
> not necessarily the same feerate. Currently, if we see a transaction
> that has the same txid as one in the mempool, we reject it as a
> duplicate, even if the feerate is much higher. It's unclear to me if
> we have a very strong reason to change this, but noting it as a
> limitation of our current replacement policy. See [#24007][12].
>
> ### User Interface
>
> #### Using Unconfirmed UTXOs to Fund Replacements
>
> The restriction of only allowing confirmed UTXOs for funding a
> fee-bump (Rule #2) can hurt users trying to fee-bump their
> transactions and complicate wallet implementations. If the original
> transaction's output value isn't sufficient to fund a fee-bump and/or
> all of the user's other UTXOs are unconfirmed, they might not be able
> to fund a replacement transaction. Wallet developers also need to
> treat self-owned unconfirmed UTXOs as unusable for fee-bumping, which
> adds complexity to wallet logic. For example, see BDK issues [#144][4]
> and [#414][5].
>
> #### Interface Not Suitable for Coin Selection
>
> Currently, a user cannot simply create a replacement transaction
> targeting a specific feerate or meeting a minimum fee amount and
> expect to meet the RBF criteria. The fee amount depends on the size of
> the replacement transaction, and feerate is almost irrelevant.
>
> Bitcoin Core's `bumpfee` doesn't use the RBF rules when funding the
> replacement. It [estimates][13] a feerate which is "wallet incremental
> relay fee" (a conservative overestimation of the node's incremental
> relay fee) higher than the original transaction, selects coins for
> that feerate, and hopes that it meets the RBF rules. It never fails
> Rule #3 and #4 because it uses all original inputs and refuses to
> bump a transaction with mempool descendants.
>
> This is suboptimal, but is designed to work with the coin selection
> engine: select a feerate first, and then add fees to cover it.
> Following the exact RBF rules would require working the other way
> around: based on how much fees we've added to the transaction and its
> current size, calculate the feerate to see if we meet Rule #4.
>
> While this isn't completely broken, and the user interface is
> secondary to the safety of the mempool policy, we can do much better.
> A much more user-friendly interface would depend *only* on the
> fee and size of the original transactions.
>
> ### Updates to Mempool and Mining
>
> Since RBF was first implemented, a number of improvements have been
> made to mempool and mining logic. For example, we now use ancestor
> feerates in mining (allowing CPFP), and keep track of ancestor
> packages in the mempool.
>
> ## Ideas for Improvements
>
> ### Goals
>
> To summarize, these seem to be desired changes, in order of priority:
>
> 1. Remove Rule #3. The replacement should not be *required* to pay
> higher absolute fees.
>
> 2. Make it impossible for a replacement transaction to have a lower
> mining score than the original transaction(s). This would eliminate
> the `SIGHASH\_ANYONECANPAY` pinning attack.
>
> 3. Remove Rule #2. Adding new unconfirmed inputs should be allowed.
>
> 4. Create a more helpful interface that helps wallet fund replacement
> transactions that aim for a feerate and fee.
>
> ### A Different Model for Fees
>
> For incentive compatibility, I believe there are different
> formulations we should consider.  Most importantly, if we want to get
> rid of the absolute fee rule, we can no longer think of it as "the
> transaction needs to pay for its own bandwidth," since we won't always
> be getting additional fees. That means we need a new method of
> rate-limiting replacements that doesn't require additional fees every
> time.
>
> While it makes sense to think about monetary costs when launching a
> specific type of attack, given that the fees are paid to the miner and
> not to the mempool operators, maybe it doesn't make much sense to
> think about "paying for bandwidth". Maybe we should implement
> transaction validation rate-limiting differently, e.g. building it
> into the P2P layer instead of the mempool policy layer.
>
> Recently, Suhas gave a [formulation][8] for incentive compatibility
> that made sense to me: "are the fees expected to be paid in the next
> (N?) blocks higher or lower if we process this transaction?"
>
> I started by thinking about this where N=1 or `1 + p`.
> Here, a rational miner is looking at what fees they would
> collect in the next block, and then some proportion `p` of the rest of
> the blocks based on their hashrate. We're assuming `p` isn't *so high*
> that they would be okay with lower absolute fees in the next 1 block.
> We're also assuming `p` isn't *so low* that the miner doesn't care
> about what's left of the mempool after this block.
>
> A tweak to this formulation is "if we process this transaction, would
> the fees in the next 1 block higher or lower, and is the feerate
> density of the rest of the mempool higher or lower?" This is pretty
> similar, where N=1, but we consider the rest of the mempool by feerate
> rather than fees.
>
> ### Mining Score of a Mempool Transaction
>
> We are often interested in finding out what
> the "mining score" of a transaction in the mempool is. That is, when
> the transaction is considered in block template building, what is the
> feerate it is considered at?
>
> Obviously, it's not the transaction's individual feerate. Bitcoin Core
> [mining code sorts][14] transactions by their ancestor feerate and
> includes them packages at a time, keeping track of how this affects the
> package feerates of remaining transactions in the mempool.
>
> *ancestor feerate*: Ancestor feerate is easily accessible information,
> but it's not accurate either, because it doesn't take into account the
> fact that subsets of a transaction's ancestor set can be included
> without it. For example, ancestors may have high feerates on their own
> or we may have [high feerate siblings][8].
>
> TLDR: *Looking at the current ancestor feerate of a transaction is
> insufficient to tell us what feerate it will be considered at when
> building a block template in the future.*
>
> *min(individual feerate, ancestor feerate)*: Another
> heuristic that is simple to calculate based on current mempool tooling
> is to use the [minimum of a transaction's individual score and its
> ancestor score][10] as a conservative measure.  But this can
> overestimate as well (see the example below).
>
> *min ancestor feerate(tx + possible ancestor subsets)* We can also
> take the minimum of every possible ancestor subset, but this can be
> computationally expensive since there can be lots and lots of ancestor
> subsets.
>
> *max ancestor feerate(tx + possible descendant subsets)*: Another idea
> is to use the [maximum ancestor score of the transaction + each of its
> descendants][9]. This doesn't work either; it has the same blindspot
> of ancestor subsets being mined on their own.
>
> #### Mining Score Example
>
> Here's an example illustrating why mining score is tricky to
> efficiently calculate for mempool transactions:
>
> Let's say you have same-size transactions A (21sat/vB), B (1sat/vB),
> C(9sat/vB), D(5sat/vB).
> The layout is: grandparent A, parent B, and two children C and D.
>
> ```
>     A
>     ^
>     B
>    ^ ^
>    C D
> ```
>
> A miner using ancestor packages to build block templates will first
> include A with a mining score of 21. Next, the miner will include B and
> C with a mining score of 6. This leaves D, with a mining score of 5.
>
> Note: in this case, mining by ancestor feerate results in the most
> rational decisions, but [a candidate set-based approach][10] which
> makes ancestor feerate much less relevant could
> be more advantageous in other situations.
>
> Here is a chart showing the "true" mining score alongside the values
> calculating using imperfect heuristics described above. All of them
> can overestimate or underestimate.
>
> ```
>    A     B       C     D
> mining score |   21   |   6   |   6   |   5   |
> ancestor feerate   |   21   |  11   | 10.3  |   9   |
> min(individual, ancestor) |   21   |   1   |   9   |   5   |
> min(tx + ancestor subsets)      |   21   |   1   |   5   |   3   |
> max(tx + descendants subsets) |   21   |   9   |   9   |   5   |
>
> ```
>
> Possibly the best solution for finding the "mining score" of a
> transaction is to build a block template, see what feerate each
> package is included at. Perhaps at some cutoff, remaining mempool
> transactions can be estimated using some heuristic that leans
> {overestimating, underestimating} depending on the situation.
>
> Mining score seems to be relevant in multiple places: Murch and I
> recently [found][3] that it would be very important in
> "ancestor-aware" funding of transactions (the wallet doesn't
> incorporate ancestor fees when using unconfirmed transactions in coin
> selection, which is a bug we want to fix).
>
> In general, it would be nice to know the exact mining priority of
> one's unconfirmed transaction is.  I can think of a few block/mempool
> explorers who might want to display this information for users.
>
> ### RBF Improvement Proposals
>
> After speaking to quite a few people, here are some suggestions
> for improvements that I have heard:
>
> * The ancestor score of the replacement must be {5, 10, N}% higher
>   than that of every original transaction.
>
> * The ancestor score of the replacement must be 1sat/vB higher than
>   that of every original transaction.
>
> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>   mempool, apply the current rules (absolute fees must increase and
> pay for the replacement transaction's new bandwidth). Otherwise, use a
> feerate-only rule.
>
> * If fees don't increase, the size of the replacement transaction must
>   decrease by at least N%.
>
> * Rate-limit how many replacements we allow per prevout.
>
> * Rate-limit transaction validation in general, per peer.
>
> Perhaps some others on the mailing list can chime in to throw other
> ideas into the ring and/or combine some of these rules into a sensible
> policy.
>
> #### Replace by Feerate Only
>
> I don't think there's going to be a single-line feerate-based
> rule that can incorporate everything we need.
> On one hand, a feerate-only approach helps eliminate the issues
> associated with Rule #3. On the other hand, I believe the main concern
> with a feerate-only approach is how to rate limit replacements. We
> don't want to enable an attack such as:
>
> 1. Attacker broadcasts large, low-feerate transaction, and attaches a
> chain of descendants.
>
> 2. The attacker replaces the transaction with a smaller but higher
> feerate transaction, attaching a new chain of descendants.
>
> 3. Repeat 1000 times.
>
> #### Fees in Next Block and Feerate for the Rest of the Mempool
>
> Perhaps we can look at replacements like this:
>
> 1. Calculate the directly conflicting transactions and, with their
> descendants, the original transactions. Check signaling. Limit the
> total volume (e.g. can't be more than 100 total or 1MvB or something).
>
> 2. Find which original transactions would be in the next ~1 block. The
> replacement must pay at least this amount + X% in absolute fees. This
> guarantees that the fees of the next block doesn't decrease.
>
> 3. Find which transactions would be left in the mempool after that ~1
> block. The replacement's feerate must be Y% higher than the maximum
> mining score of these transactions. This guarantees that you now have
> only *better* candidates in your after-this-block mempool than you did
> before, even if the size and fees the transactions decrease.
>
> 4. Now you have two numbers: a minimum absolute fee amount and a
> minimum feerate. Check to see if the replacement(s) meet these
> minimums. Also, a wallet would be able to ask the node "What fee and
> feerate would I need to put on a transaction replacing this?" and use
> this information to fund a replacement transaction, without needing to
> guess or overshoot.
>
> Obviously, there are some magic numbers missing here. X and Y are
> TBD constants to ensure we have some kind of rate limiting for the
> number of replacements allowed using some set of fees.
>
> What should they be? We can do some arithmetic to see what happens if
> you start with the biggest/lowest feerate transaction and do a bunch
> of replacements. Maybe we end up with values that are high enough to
> prevent abuse and make sense for applications/users that do RBF.
>
> ### Mempool Changes Need for Implementation
>
> As described in the mining score section above,
> we may want additional tooling to more accurately assess
> the economic gain of replacing transactions in our mempool.
>
> A few options have been discussed:
>
> * Calculate block templates on the fly when we need to consider a
>   replacement. However, since replacements are [quite common][11]
>   and the information might be useful for other things as well,
>   it may be worth it to cache a block template.
>
> * Keep a persistent block template so that we know what transactions
>   we would put in the next block. We need to remember the feerate
> at which each transaction was included in the template, because an
> ancestor package may be included in the same block template in
> multiple subsets. Transactions included earlier alter the ancestor
> feerate of the remaining transactions in the package. We also need
> to keep track of the new feerates of transactions left over.
>
> * Divide the mempool into two layers, "high feerate" and "low
>   feerate." The high feerate layer contains ~1 block of packages with
> the highest ancestor feerates, and the low feerate layer contains
> everything else. At the edge of a block, we have a Knapsacky problem
> where the next highest ancestor feerate package might not fit, so we
> would probably want the high feerate layer ~2MvB or something to avoid
> underestimating the fees.
>
> ## Acknowledgements
>
> Thank you to everyone whose RBF-related suggestions, grievances,
> criticisms and ideas were incorporated in this document:
> Andrew Chow, Matt Corallo, Suhas Daftuar, Christian Decker,
> Mark Erhardt, Lloyd Fournier, Lisa Neigut, John Newbery,
> Antoine Poinsot, Antoine Riard, Larry Ruane,
> S3RK and Bastien Teinturier.
>
> Thanks for reading!
>
> Best,
> Gloria
>
> [1]:
> https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
> [2]: https://github.com/bitcoin/bitcoin/pull/23121#issuecomment-929475999
> [3]:
> https://github.com/Xekyo/bitcoin/commit/d754b0242ec69d42c570418aebf9c1335af0b8ea
> [4]: https://github.com/bitcoindevkit/bdk/issues/144
> [5]: https://github.com/bitcoindevkit/bdk/issues/414
> [6]:
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
> [7]:
> https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a#new-unconfirmed-inputs-rule-2
> [8]: https://github.com/bitcoin/bitcoin/pull/23121#discussion_r777131366
> [9]: https://github.com/bitcoin/bitcoin/pull/22290#issuecomment-865887922
> [10]:
> https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
> [11]: https://github.com/bitcoin/bitcoin/pull/22539#issuecomment-885763670
> [12]: https://github.com/bitcoin/bitcoin/pull/24007
> [13]:
> https://github.com/bitcoin/bitcoin/blob/1a369f006fd0bec373b95001ed84b480e852f191/src/wallet/feebumper.cpp#L114
> [14]:
> https://github.com/bitcoin/bitcoin/blob/cf5bb048e80d4cde8828787b266b7f5f2e3b6d7b/src/node/miner.cpp#L310-L320
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 31690 bytes --]

Re: [bitcoin-dev] Improving RBF Policy

[Jeremy]

[-- Attachment #1: Type: text/plain, Size: 3717 bytes --]

Gloria,

This is a brilliant post! Great job systematizing many of the issues. Quite
a lot to chew on & I hope other readers of this list digest the post fully.

Three things come to mind as partial responses:

under:

- **DoS Protection**: Limit two types of DoS attacks on the node's
>   mempool: (1) the number of times a transaction can be replaced and
> (2) the volume of transactions that can be evicted during a
> replacement.


I'd more simply put it:

Limiting the amount of work that must be done to consider the replacement

We don't particularly care about goal (1) or goal (2), we care about how
much it costs to do (1) or (2). And there are scenarios where the (1) or
(2) might not be particularly high, but the total work still might be. I
can give you some examples to consider if needed. There are also scenarios
where (1) and (2) might be high, but the cost is low overall. Therefore it
makes sense to be a little more general with what the anti-DoS goal is.




An issue I'd like to toss into the mix is that of iterative / additive
batching. E.g., https://bitcoinops.org/en/cardcoins-rbf-batching/

This is where an business puts a txn in the mempool that pays to N users,
and as they see additional requests for payouts the update it to N+2,
N+2... N+M payouts. This iterative batching can be highly efficient because
the number of transactions per business per 10 minutes is 1 (with variable
number of outputs).

One issue with this approach today is that because of the feerate rule, if
you go from N to N+1 you need to pay 1 sat/byte over the whole txn. Applied
M times, and you have to increase fees quadratically for this approach.
Therefore the less efficient long-chain of batches model ends up being
'rational' with respect to mempool policy and irrational with respect to
"optimally packing blocks with transactions".

If the absolute fee rule is dropped, but feerate remains, one thing you
might see is businesses doing iterative batches with N+2M outputs whereby
they drop 2 outputs for every input they add, allowing the iterative batch
to always increase the fee-rate but possibly not triggering the quadratic
feerate issue since the transaction gets smaller over time.

Another possible solution to this would be to allow relaying "txdiffs"
which only require re-relay of signatures + new/modified outputs, and not
the entire tx.

I think this iterative batching is pretty desirable to support, and so I'd
like to see a RBF model which doesn't make it "unfairly" expensive.

(I'll spare everyone the details on how CTV batching also solves this, but
feel free to ask elsewhere.)

A counterargument to additive batching is that if you instead do non
iterative batches every minute, and you have 100 txns that arrive
uniformly, you'd end up with 10 batches of size 10 on average. The bulk of
the benefit under this model is in the non-batched to batched transition,
and the iterative part only saves on space/fees marginally after that point.



A final point is that a verifiable delay function could be used over, e.g.,
each of the N COutpoints individually to rate-limit transaction
replacement. The VDF period can be made shorter / eliminated depending on
the feerate increase. E.g., always consider a much higher feerate txn
whenever available, for things of equal feerate only consider 1 per minute.
A VDF is like proof-of-work that doesn't parallelize, in case you are
unfamiliar, so no matter how many computers you have it would take about
the same amount of time (you could parallelize across N outputs, of course,
but you're still bound minimally to the time it takes to replace 1 output,
doing all outputs individually just is the most flexible option).


Cheers,

Jeremy

[-- Attachment #2: Type: text/html, Size: 9089 bytes --]

[bitcoin-dev] Improving RBF Policy

[Gloria Zhao]

[-- Attachment #1: Type: text/plain, Size: 21067 bytes --]

Hi everyone,

This post discusses limitations of current Bitcoin Core RBF policy and
attempts to start a conversation about how we can improve it,
summarizing some ideas that have been discussed. Please reply if you
have any new input on issues to be solved and ideas for improvement!

Just in case I've screwed up the text wrapping again, another copy can be
found here: https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff

## Background

Please feel free to skip this section if you are already familiar
with RBF.

Nodes may receive *conflicting* unconfirmed transactions, aka
"double spends" of the same inputs. Instead of always keeping the
first transaction, since v0.12, Bitcoin Core mempool policy has
included a set of Replace-by-Fee (RBF) criteria that allows the second
transaction to replace the first one and any descendants it may have.

Bitcoin Core RBF policy was previously documented as BIP 125.
The current RBF policy is documented [here][1]. In summary:

1. The directly conflicting transactions all signal replaceability
   explicitly.

2. The replacement transaction only includes an unconfirmed input if
   that input was included in one of the directly conflicting
transactions.

3. The replacement transaction pays an absolute fee of at least the
   sum paid by the original transactions.

4. The additional fees pays for the replacement transaction's
   bandwidth at or above the rate set by the node's *incremental relay
feerate*.

5. The sum of all directly conflicting transactions' descendant counts
   (number of transactions inclusive of itself and its descendants)
does not exceed 100.

We can split these rules into 3 categories/goals:

- **Allow Opting Out**: Some applications/businesses are unable to
  handle transactions that are replaceable (e.g. merchants that use
zero-confirmation transactions). We (try to) help these businesses by
honoring BIP125 signaling; we won't replace transactions that have not
opted in.

- **Incentive Compatibility**: Ensure that our RBF policy would not
  accept replacement transactions which would decrease fee profits
  of a miner. In general, if our mempool policy deviates from what is
economically rational, it's likely that the transactions in our
mempool will not match the ones in miners' mempools, making our
fee estimation, compact block relay, and other mempool-dependent
functions unreliable. Incentive-incompatible policy may also
encourage transaction submission through routes other than the p2p
network, harming censorship-resistance and privacy of Bitcoin payments.

- **DoS Protection**: Limit two types of DoS attacks on the node's
  mempool: (1) the number of times a transaction can be replaced and
(2) the volume of transactions that can be evicted during a
replacement.

Even more abstract: our goal is to make a replacement policy that
results in a useful interface for users and safe policy for
node operators.

## Motivation

There are a number of known problems with the current RBF policy.
Many of these shortcomings exist due to mempool limitations at the
time RBF was implemented or result from new types of Bitcoin usage;
they are not criticisms of the original design.

### Pinning Attacks

The most pressing concern is that attackers may take advantage of
limitations in RBF policy to prevent other users' transactions from
being mined or getting accepted as a replacement.

#### SIGHASH_ANYONECANPAY Pinning

BIP125#2 can be bypassed by creating intermediary transactions to be
replaced together. Anyone can simply split a 1-input 1-output
transaction off from the replacement transaction, then broadcast the
transaction as is. This can always be done, and quite cheaply. More
details in [this comment][2].

In general, if a transaction is signed with SIGHASH\_ANYONECANPAY,
anybody can just attach a low feerate parent to this transaction and
lower its ancestor feerate.  Even if you require SIGHASH\_ALL which
prevents an attacker from changing any outputs, the input can be a
very low amount (e.g. just above the dust limit) from a low-fee
ancestor and still bring down the ancestor feerate of the transaction.

TLDR: if your transaction is signed with SIGHASH\_ANYONECANPAY and
signals replaceability, regardless of the feerate you broadcast at, an
attacker can lower its mining priority by adding an ancestor.

#### Absolute Fee

The restriction of requiring replacement transactions to increase the
absolute fee of the mempool has been described as "bonkers." If the
original transaction has a very large descendant that pays a large
amount of fees, even if it has a low feerate, the replacement
transaction must now pay those fees in order to meet Rule #3.

#### Package RBF

There are a number of reasons why, in order to enable Package RBF, we
cannot use the same criteria.

For starters, the absolute fee pinning attack is especially
problematic if we apply the same rules (i.e. Rule #3 and #4) in
Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
LN channel. The mempool is rather full, so their pre-negotiated
commitment transactions' feerates would not be considered high
priority by miners.  Bob broadcasts his commitment transaction and
attaches a very large child (100KvB with 100,000sat in fees) to his
anchor output. Alice broadcasts her commitment transaction with a
fee-bumping child (200vB with 50,000sat fees which is a generous
250sat/vB), but this does not meet the absolute fee requirement. She
would need to add another 50,000sat to replace Bob's commitment
transaction.

Disallowing new unconfirmed inputs (Rule #2) in Package RBF would be
broken for packages containing transactions already in the mempool,
explained [here][7].

Note: I originally [proposed][6] Package RBF using the same Rule #3
and #4 before I realized how significant this pinning attack is. I'm
retracting that proposal, and a new set of Package RBF rules would
follow from whatever the new individual RBF rules end up being.

#### Same Txid Different Witness

Two transactions with the same non-witness data but different
witnesses have the same txid but different wtxid, and the same fee but
not necessarily the same feerate. Currently, if we see a transaction
that has the same txid as one in the mempool, we reject it as a
duplicate, even if the feerate is much higher. It's unclear to me if
we have a very strong reason to change this, but noting it as a
limitation of our current replacement policy. See [#24007][12].

### User Interface

#### Using Unconfirmed UTXOs to Fund Replacements

The restriction of only allowing confirmed UTXOs for funding a
fee-bump (Rule #2) can hurt users trying to fee-bump their
transactions and complicate wallet implementations. If the original
transaction's output value isn't sufficient to fund a fee-bump and/or
all of the user's other UTXOs are unconfirmed, they might not be able
to fund a replacement transaction. Wallet developers also need to
treat self-owned unconfirmed UTXOs as unusable for fee-bumping, which
adds complexity to wallet logic. For example, see BDK issues [#144][4]
and [#414][5].

#### Interface Not Suitable for Coin Selection

Currently, a user cannot simply create a replacement transaction
targeting a specific feerate or meeting a minimum fee amount and
expect to meet the RBF criteria. The fee amount depends on the size of
the replacement transaction, and feerate is almost irrelevant.

Bitcoin Core's `bumpfee` doesn't use the RBF rules when funding the
replacement. It [estimates][13] a feerate which is "wallet incremental
relay fee" (a conservative overestimation of the node's incremental
relay fee) higher than the original transaction, selects coins for
that feerate, and hopes that it meets the RBF rules. It never fails
Rule #3 and #4 because it uses all original inputs and refuses to
bump a transaction with mempool descendants.

This is suboptimal, but is designed to work with the coin selection
engine: select a feerate first, and then add fees to cover it.
Following the exact RBF rules would require working the other way
around: based on how much fees we've added to the transaction and its
current size, calculate the feerate to see if we meet Rule #4.

While this isn't completely broken, and the user interface is
secondary to the safety of the mempool policy, we can do much better.
A much more user-friendly interface would depend *only* on the
fee and size of the original transactions.

### Updates to Mempool and Mining

Since RBF was first implemented, a number of improvements have been
made to mempool and mining logic. For example, we now use ancestor
feerates in mining (allowing CPFP), and keep track of ancestor
packages in the mempool.

## Ideas for Improvements

### Goals

To summarize, these seem to be desired changes, in order of priority:

1. Remove Rule #3. The replacement should not be *required* to pay
higher absolute fees.

2. Make it impossible for a replacement transaction to have a lower
mining score than the original transaction(s). This would eliminate
the `SIGHASH\_ANYONECANPAY` pinning attack.

3. Remove Rule #2. Adding new unconfirmed inputs should be allowed.

4. Create a more helpful interface that helps wallet fund replacement
transactions that aim for a feerate and fee.

### A Different Model for Fees

For incentive compatibility, I believe there are different
formulations we should consider.  Most importantly, if we want to get
rid of the absolute fee rule, we can no longer think of it as "the
transaction needs to pay for its own bandwidth," since we won't always
be getting additional fees. That means we need a new method of
rate-limiting replacements that doesn't require additional fees every
time.

While it makes sense to think about monetary costs when launching a
specific type of attack, given that the fees are paid to the miner and
not to the mempool operators, maybe it doesn't make much sense to
think about "paying for bandwidth". Maybe we should implement
transaction validation rate-limiting differently, e.g. building it
into the P2P layer instead of the mempool policy layer.

Recently, Suhas gave a [formulation][8] for incentive compatibility
that made sense to me: "are the fees expected to be paid in the next
(N?) blocks higher or lower if we process this transaction?"

I started by thinking about this where N=1 or `1 + p`.
Here, a rational miner is looking at what fees they would
collect in the next block, and then some proportion `p` of the rest of
the blocks based on their hashrate. We're assuming `p` isn't *so high*
that they would be okay with lower absolute fees in the next 1 block.
We're also assuming `p` isn't *so low* that the miner doesn't care
about what's left of the mempool after this block.

A tweak to this formulation is "if we process this transaction, would
the fees in the next 1 block higher or lower, and is the feerate
density of the rest of the mempool higher or lower?" This is pretty
similar, where N=1, but we consider the rest of the mempool by feerate
rather than fees.

### Mining Score of a Mempool Transaction

We are often interested in finding out what
the "mining score" of a transaction in the mempool is. That is, when
the transaction is considered in block template building, what is the
feerate it is considered at?

Obviously, it's not the transaction's individual feerate. Bitcoin Core
[mining code sorts][14] transactions by their ancestor feerate and
includes them packages at a time, keeping track of how this affects the
package feerates of remaining transactions in the mempool.

*ancestor feerate*: Ancestor feerate is easily accessible information,
but it's not accurate either, because it doesn't take into account the
fact that subsets of a transaction's ancestor set can be included
without it. For example, ancestors may have high feerates on their own
or we may have [high feerate siblings][8].

TLDR: *Looking at the current ancestor feerate of a transaction is
insufficient to tell us what feerate it will be considered at when
building a block template in the future.*

*min(individual feerate, ancestor feerate)*: Another
heuristic that is simple to calculate based on current mempool tooling
is to use the [minimum of a transaction's individual score and its
ancestor score][10] as a conservative measure.  But this can
overestimate as well (see the example below).

*min ancestor feerate(tx + possible ancestor subsets)* We can also
take the minimum of every possible ancestor subset, but this can be
computationally expensive since there can be lots and lots of ancestor
subsets.

*max ancestor feerate(tx + possible descendant subsets)*: Another idea
is to use the [maximum ancestor score of the transaction + each of its
descendants][9]. This doesn't work either; it has the same blindspot
of ancestor subsets being mined on their own.

#### Mining Score Example

Here's an example illustrating why mining score is tricky to
efficiently calculate for mempool transactions:

Let's say you have same-size transactions A (21sat/vB), B (1sat/vB),
C(9sat/vB), D(5sat/vB).
The layout is: grandparent A, parent B, and two children C and D.

```
    A
    ^
    B
   ^ ^
   C D
```

A miner using ancestor packages to build block templates will first
include A with a mining score of 21. Next, the miner will include B and
C with a mining score of 6. This leaves D, with a mining score of 5.

Note: in this case, mining by ancestor feerate results in the most
rational decisions, but [a candidate set-based approach][10] which
makes ancestor feerate much less relevant could
be more advantageous in other situations.

Here is a chart showing the "true" mining score alongside the values
calculating using imperfect heuristics described above. All of them
can overestimate or underestimate.

```
   A     B       C     D
mining score |   21   |   6   |   6   |   5   |
ancestor feerate   |   21   |  11   | 10.3  |   9   |
min(individual, ancestor) |   21   |   1   |   9   |   5   |
min(tx + ancestor subsets)      |   21   |   1   |   5   |   3   |
max(tx + descendants subsets) |   21   |   9   |   9   |   5   |

```

Possibly the best solution for finding the "mining score" of a
transaction is to build a block template, see what feerate each
package is included at. Perhaps at some cutoff, remaining mempool
transactions can be estimated using some heuristic that leans
{overestimating, underestimating} depending on the situation.

Mining score seems to be relevant in multiple places: Murch and I
recently [found][3] that it would be very important in
"ancestor-aware" funding of transactions (the wallet doesn't
incorporate ancestor fees when using unconfirmed transactions in coin
selection, which is a bug we want to fix).

In general, it would be nice to know the exact mining priority of
one's unconfirmed transaction is.  I can think of a few block/mempool
explorers who might want to display this information for users.

### RBF Improvement Proposals

After speaking to quite a few people, here are some suggestions
for improvements that I have heard:

* The ancestor score of the replacement must be {5, 10, N}% higher
  than that of every original transaction.

* The ancestor score of the replacement must be 1sat/vB higher than
  that of every original transaction.

* If the original transaction is in the top {0.75MvB, 1MvB} of the
  mempool, apply the current rules (absolute fees must increase and
pay for the replacement transaction's new bandwidth). Otherwise, use a
feerate-only rule.

* If fees don't increase, the size of the replacement transaction must
  decrease by at least N%.

* Rate-limit how many replacements we allow per prevout.

* Rate-limit transaction validation in general, per peer.

Perhaps some others on the mailing list can chime in to throw other
ideas into the ring and/or combine some of these rules into a sensible
policy.

#### Replace by Feerate Only

I don't think there's going to be a single-line feerate-based
rule that can incorporate everything we need.
On one hand, a feerate-only approach helps eliminate the issues
associated with Rule #3. On the other hand, I believe the main concern
with a feerate-only approach is how to rate limit replacements. We
don't want to enable an attack such as:

1. Attacker broadcasts large, low-feerate transaction, and attaches a
chain of descendants.

2. The attacker replaces the transaction with a smaller but higher
feerate transaction, attaching a new chain of descendants.

3. Repeat 1000 times.

#### Fees in Next Block and Feerate for the Rest of the Mempool

Perhaps we can look at replacements like this:

1. Calculate the directly conflicting transactions and, with their
descendants, the original transactions. Check signaling. Limit the
total volume (e.g. can't be more than 100 total or 1MvB or something).

2. Find which original transactions would be in the next ~1 block. The
replacement must pay at least this amount + X% in absolute fees. This
guarantees that the fees of the next block doesn't decrease.

3. Find which transactions would be left in the mempool after that ~1
block. The replacement's feerate must be Y% higher than the maximum
mining score of these transactions. This guarantees that you now have
only *better* candidates in your after-this-block mempool than you did
before, even if the size and fees the transactions decrease.

4. Now you have two numbers: a minimum absolute fee amount and a
minimum feerate. Check to see if the replacement(s) meet these
minimums. Also, a wallet would be able to ask the node "What fee and
feerate would I need to put on a transaction replacing this?" and use
this information to fund a replacement transaction, without needing to
guess or overshoot.

Obviously, there are some magic numbers missing here. X and Y are
TBD constants to ensure we have some kind of rate limiting for the
number of replacements allowed using some set of fees.

What should they be? We can do some arithmetic to see what happens if
you start with the biggest/lowest feerate transaction and do a bunch
of replacements. Maybe we end up with values that are high enough to
prevent abuse and make sense for applications/users that do RBF.

### Mempool Changes Need for Implementation

As described in the mining score section above,
we may want additional tooling to more accurately assess
the economic gain of replacing transactions in our mempool.

A few options have been discussed:

* Calculate block templates on the fly when we need to consider a
  replacement. However, since replacements are [quite common][11]
  and the information might be useful for other things as well,
  it may be worth it to cache a block template.

* Keep a persistent block template so that we know what transactions
  we would put in the next block. We need to remember the feerate
at which each transaction was included in the template, because an
ancestor package may be included in the same block template in
multiple subsets. Transactions included earlier alter the ancestor
feerate of the remaining transactions in the package. We also need
to keep track of the new feerates of transactions left over.

* Divide the mempool into two layers, "high feerate" and "low
  feerate." The high feerate layer contains ~1 block of packages with
the highest ancestor feerates, and the low feerate layer contains
everything else. At the edge of a block, we have a Knapsacky problem
where the next highest ancestor feerate package might not fit, so we
would probably want the high feerate layer ~2MvB or something to avoid
underestimating the fees.

## Acknowledgements

Thank you to everyone whose RBF-related suggestions, grievances,
criticisms and ideas were incorporated in this document:
Andrew Chow, Matt Corallo, Suhas Daftuar, Christian Decker,
Mark Erhardt, Lloyd Fournier, Lisa Neigut, John Newbery,
Antoine Poinsot, Antoine Riard, Larry Ruane,
S3RK and Bastien Teinturier.

Thanks for reading!

Best,
Gloria

[1]:
https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
[2]: https://github.com/bitcoin/bitcoin/pull/23121#issuecomment-929475999
[3]:
https://github.com/Xekyo/bitcoin/commit/d754b0242ec69d42c570418aebf9c1335af0b8ea
[4]: https://github.com/bitcoindevkit/bdk/issues/144
[5]: https://github.com/bitcoindevkit/bdk/issues/414
[6]:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
[7]:
https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a#new-unconfirmed-inputs-rule-2
[8]: https://github.com/bitcoin/bitcoin/pull/23121#discussion_r777131366
[9]: https://github.com/bitcoin/bitcoin/pull/22290#issuecomment-865887922
[10]:
https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
[11]: https://github.com/bitcoin/bitcoin/pull/22539#issuecomment-885763670
[12]: https://github.com/bitcoin/bitcoin/pull/24007
[13]:
https://github.com/bitcoin/bitcoin/blob/1a369f006fd0bec373b95001ed84b480e852f191/src/wallet/feebumper.cpp#L114
[14]:
https://github.com/bitcoin/bitcoin/blob/cf5bb048e80d4cde8828787b266b7f5f2e3b6d7b/src/node/miner.cpp#L310-L320

[-- Attachment #2: Type: text/html, Size: 24377 bytes --]