[bitcoin-dev] Revisiting BIP 125 RBF policy.

[bitcoin-dev] Revisiting BIP 125 RBF policy.

[Russell O'Connor]

[-- Attachment #1: Type: text/plain, Size: 4380 bytes --]

I think it is worth revisiting BIP 125's replace-by-fee policy for when to
replace transactions.

The current policy can be problematic. As noted earlier by Rhavar,
sometimes one's transaction becomes pinned making it infeasible to fee bump
with RBF.  This happens when one makes a normal payment to a large
commercial service, and, while the transaction remains unconfirmed, the
commercial service creates a low-fee-rate sweep of one's payment, among a
collection of others.  If one wants to RBF this original payment, for
example to get confirmation of the change output for use in further
transactions, the current BIP 125 rules require that you make a fee bump
that exceeds the combined total fees of the original transaction and the
low-fee-rate sweep of the commercial service.

The problem is that, while the fee rate of the sweep is low, the absolute
size of the fee can still be large, making it infeasible to RBF the
original transaction.  BIP 125 was defined back in 2015, when perhaps
rational miners did care about absolute fee amounts. However, today we are
in an era where rational miners care about fee-rates more than absolute
fees.  The fee-rate of the large sweep transaction is low enough that we do
not expect that miners will be mining it in the same block as the original
transaction.  Today, a rational miner will prefer a fee-bumped version of
original transaction without consideration of the low-fee sweep transaction
(or at least discounting the low-fee sweep in proportion to the miner's
hash-rate fraction).

Let me quote the five rules that define the current BIP 125 policy:

One or more transactions currently in the mempool (original transactions)
> will be replaced by a new transaction (replacement transaction) that spends
> one or more of the same inputs if,
>
>    1. The original transactions signal replaceability explicitly or
>    through inheritance as described in the above Summary section.
>    2. The replacement transaction does not contain any new unconfirmed
>    inputs that did not previously appear in the mempool. (Unconfirmed inputs
>    are inputs spending outputs from currently unconfirmed transactions.)
>    3. The replacement transaction pays an absolute fee of at least the
>    sum paid by the original transactions.
>    4. The replacement transaction must also pay for its own bandwidth at
>    or above the rate set by the node's minimum relay fee setting. For example,
>    if the minimum relay fee is 1 satoshi/byte and the replacement transaction
>    is 500 bytes total, then the replacement must pay a fee at least 500
>    satoshis higher than the sum of the originals.
>    5. The number of original transactions to be replaced and their
>    descendant transactions which will be evicted from the mempool must not
>    exceed a total of 100 transactions.
>
> To address the new reality of rational miners' consideration, I propose
changing rules 3 and 4 to something like the following.

3'. The replacement transaction pays a fee rate of at least the effective
fee rate of any chain of transactions from the set of original transactions
that begins with the root of the original transaction set.

4'. The replacement transaction must also pay for replacing the original
transactions at or above the rate set by the node's minimum relay fee
setting. For example, if the minimum relay fee is 1 satoshi/byte and the
replacement transaction and the original transactions are 1000 bytes total,
then the replacement must pay a fee at least 1000 satoshis higher than the
fee of the root transaction of the original transactions.

Rule 3' is a fancy way of saying that the replacement transaction must have
a fee rate that is larger than the package fee rate of the root of the set
of transactions it replaces, where the package fee rate is the fee rate
implied by considering CPFP.

Rule 4' is an amended anti-spam rule that is intended to avoid DOS attacks
from churning the mempool. I don't know if it is really necessary to pay
for the size of the original transactions being evicted, but some people I
chatted with thought it could be important.

Other people on the mailing list have been thinking about RBF policy for
far longer than I have, so I wouldn't be surprised if my proposal above is
naive.  However, I think it can start a conversation about addressing the
problems with the current RBF policy.

[-- Attachment #2: Type: text/html, Size: 4764 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[rhavar]

Thank you very much for writing this up.  It's worth noting that there can be multiple roots for the transactions that are getting replaced.

So for rule 3, you probably want a feeRate >= the max "package fee rate" of all replaced roots.


I am very happy with this proposal in general, as it's clearly a step in the right direction for making transaction replacement practically usable for todays services.


However, I think your new rule 4 is a bit weak. The logical extension of your proposal would be to allow a transaction (say B) be able to replace transactions (say A) by purely paying a higher fee rate, /even if it's less absolute fee/. In this simple example of B replacing A -- B should pay at least:   (a.FeeRate * b.size) + relayFeeRate*(a.size + b.size)



​-Ryan

​

-------- Original Message --------
 On February 12, 2018 10:52 AM, Russell O'Connor via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

>I think it is worth revisiting BIP 125's replace-by-fee policy for when to replace transactions.
>The current policy can be problematic. As noted earlier by Rhavar, sometimes one's transaction becomes pinned making it infeasible to fee bump with RBF.  This happens when one makes a normal payment to a large commercial service, and, while the transaction remains unconfirmed, the commercial service creates a low-fee-rate sweep of one's payment, among a collection of others.  If one wants to RBF this original payment, for example to get confirmation of the change output for use in further transactions, the current BIP 125 rules require that you make a fee bump that exceeds the combined total fees of the original transaction and the low-fee-rate sweep of the commercial service.
>
>The problem is that, while the fee rate of the sweep is low, the absolute size of the fee can still be large, making it infeasible to RBF the original transaction.  BIP 125 was defined back in 2015, when perhaps rational miners did care about absolute fee amounts. However, today we are in an era where rational miners care about fee-rates more than absolute fees.  The fee-rate of the large sweep transaction is low enough that we do not expect that miners will be mining it in the same block as the original transaction.  Today, a rational miner will prefer a fee-bumped version of original transaction without consideration of the low-fee sweep transaction (or at least discounting the low-fee sweep in proportion to the miner's hash-rate fraction).
>
>Let me quote the five rules that define the current BIP 125 policy:
>
>>
>>One or more transactions currently in the mempool (original
>>transactions) will be replaced by a new transaction (replacement
>>transaction) that spends one or more of the same inputs if,
>>
>>
>>1. The original transactions signal replaceability explicitly or through inheritance as described in the above Summary section.
>>
>>2. The
>> replacement transaction does not contain any new unconfirmed inputs 
>>that did not previously appear in the mempool. (Unconfirmed inputs are 
>>inputs spending outputs from currently unconfirmed transactions.)
>>
>>3. The replacement transaction pays an absolute fee of at least the sum paid by the original transactions.
>>
>>4. The
>> replacement transaction must also pay for its own bandwidth at or above
>> the rate set by the node's minimum relay fee setting.  For example, if 
>>the minimum relay fee is 1 satoshi/byte and the replacement transaction 
>>is 500 bytes total, then the replacement must pay a fee at least 500 
>>satoshis higher than the sum of the originals.
>>
>>5. The number of 
>>original transactions to be replaced and their descendant transactions 
>>which will be evicted from the mempool must not exceed a total of 100 
>>transactions.
>>
>>To address the new reality of rational miners' consideration, I propose changing rules 3 and 4 to something like the following.
>3'. The replacement transaction pays a fee rate of at least the effective fee rate of any chain of transactions from the set of original transactions that begins with the root of the original transaction set.
>
>4'. The
> replacement transaction must also pay for replacing the original transactions at or above
> the rate set by the node's minimum relay fee setting.  For example, if 
>the minimum relay fee is 1 satoshi/byte and the replacement transaction and the original transactions are 1000 bytes total, then the replacement must pay a fee at least 1000 
>satoshis higher than the fee of the root transaction of the original transactions.
>Rule 3' is a fancy way of saying that the replacement transaction must have a fee rate that is larger than the package fee rate of the root of the set of transactions it replaces, where the package fee rate is the fee rate implied by considering CPFP.
>Rule 4' is an amended anti-spam rule that is intended to avoid DOS attacks from churning the mempool. I don't know if it is really necessary to pay for the size of the original transactions being evicted, but some people I chatted with thought it could be important.
>
>Other people on the mailing list have been thinking about RBF policy for far longer than I have, so I wouldn't be surprised if my proposal above is naive.  However, I think it can start a conversation about addressing the problems with the current RBF policy.
>

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 7606 bytes --]

On Mon, Feb 12, 2018 at 10:52:30AM -0500, Russell O'Connor via bitcoin-dev wrote:
> I think it is worth revisiting BIP 125's replace-by-fee policy for when to
> replace transactions.
> 
> The current policy can be problematic. As noted earlier by Rhavar,
> sometimes one's transaction becomes pinned making it infeasible to fee bump
> with RBF.  This happens when one makes a normal payment to a large
> commercial service, and, while the transaction remains unconfirmed, the
> commercial service creates a low-fee-rate sweep of one's payment, among a
> collection of others.  If one wants to RBF this original payment, for
> example to get confirmation of the change output for use in further
> transactions, the current BIP 125 rules require that you make a fee bump
> that exceeds the combined total fees of the original transaction and the
> low-fee-rate sweep of the commercial service.
> 
> The problem is that, while the fee rate of the sweep is low, the absolute
> size of the fee can still be large, making it infeasible to RBF the
> original transaction.  BIP 125 was defined back in 2015, when perhaps
> rational miners did care about absolute fee amounts. However, today we are
> in an era where rational miners care about fee-rates more than absolute
> fees.  The fee-rate of the large sweep transaction is low enough that we do
> not expect that miners will be mining it in the same block as the original
> transaction.  Today, a rational miner will prefer a fee-bumped version of
> original transaction without consideration of the low-fee sweep transaction
> (or at least discounting the low-fee sweep in proportion to the miner's
> hash-rate fraction).

I don't actually see where the problem is here. First of all, suppose we have a
transaction T_a that already pays Alice with a feerate sufficiently high that
we expect it to get mined in the near future. If we want to pay Bob, we can do
that by simply creating a double-spend of T_a that pays both Bob and Alice,
T_{ab}. BIP125 only requires that double-spend to have an absolute fee higher
than the minimum relay feerate * size of the transaction.

I just checked one of my nodes, and the absolute minimum relay fee is about
1/5th that of what estimatefee returns for the longest possible estimate, 48
blocks. Depends on the exact circumstances, but it'll likely be worth it to pay
Bob with a replacement of T_a rather than create a second transaction due to
that difference.

Secondly, if for some reason you need to broadcast a separate transaction
paying Bob before you do the replacement, again I don't see an issue: just make
a minimum fee T_b that pays Bob, and replace both with T_{ab}. Again, the big
difference between minimum fee and what you might actually pay in fees means
that you'll still save money in most cases, so long as your wallet is
intelligent enough to pick a low feerate for T_b.

> Let me quote the five rules that define the current BIP 125 policy:
> 
> One or more transactions currently in the mempool (original transactions)
> > will be replaced by a new transaction (replacement transaction) that spends
> > one or more of the same inputs if,
> >
> >    1. The original transactions signal replaceability explicitly or
> >    through inheritance as described in the above Summary section.
> >    2. The replacement transaction does not contain any new unconfirmed
> >    inputs that did not previously appear in the mempool. (Unconfirmed inputs
> >    are inputs spending outputs from currently unconfirmed transactions.)
> >    3. The replacement transaction pays an absolute fee of at least the
> >    sum paid by the original transactions.
> >    4. The replacement transaction must also pay for its own bandwidth at
> >    or above the rate set by the node's minimum relay fee setting. For example,
> >    if the minimum relay fee is 1 satoshi/byte and the replacement transaction
> >    is 500 bytes total, then the replacement must pay a fee at least 500
> >    satoshis higher than the sum of the originals.
> >    5. The number of original transactions to be replaced and their
> >    descendant transactions which will be evicted from the mempool must not
> >    exceed a total of 100 transactions.
> >
> > To address the new reality of rational miners' consideration, I propose
> changing rules 3 and 4 to something like the following.
> 
> 3'. The replacement transaction pays a fee rate of at least the effective
> fee rate of any chain of transactions from the set of original transactions
> that begins with the root of the original transaction set.

I think what you mean here should be the effective fee rate of the maximum
feerate package that can be built from the set of transactions that begins with
the candidate replacement. But actually calculating this is I believe
non-trivial, which is why I didn't implement it this way when RBF was first
implemented.

> 4'. The replacement transaction must also pay for replacing the original
> transactions at or above the rate set by the node's minimum relay fee
> setting. For example, if the minimum relay fee is 1 satoshi/byte and the
> replacement transaction and the original transactions are 1000 bytes total,
> then the replacement must pay a fee at least 1000 satoshis higher than the
> fee of the root transaction of the original transactions.

So the previous version of condition #4 does this implicitly because the
absolute fee isn't allowed to go down; you're effectively re-adding this
condition. But as I've shown above, you can get the same *behavior* by simply
ensuring that the transactions you broadcast that you'll want to double-spend
have a minimum feerate in the first place.

> Rule 3' is a fancy way of saying that the replacement transaction must have
> a fee rate that is larger than the package fee rate of the root of the set
> of transactions it replaces, where the package fee rate is the fee rate
> implied by considering CPFP.
> 
> Rule 4' is an amended anti-spam rule that is intended to avoid DOS attacks
> from churning the mempool. I don't know if it is really necessary to pay
> for the size of the original transactions being evicted, but some people I
> chatted with thought it could be important.

I think this is very important. For example, without this condition I could do
a DoS attack by repeatedly broadcasting a transaction, then spending the
outputs of that transaction with a very large number of child transactions, all
of minimum fee. With up to 100 transactions allowed for consideration, and a
100KB max transaction size, that could be up to ~10MB of transactions.

Next I double spend the root, increasing it's feerate but *not* paying for the
child transactions. Those ~10MB are now evicted from the mempool, and I can
repeat the cycle again. The cost is whatever the root tx replacement cost,
which will be much less than the cost of broadcasting 10MB should have been.

> Other people on the mailing list have been thinking about RBF policy for
> far longer than I have, so I wouldn't be surprised if my proposal above is
> naive.  However, I think it can start a conversation about addressing the
> problems with the current RBF policy.

A better way to solve this class of problems may be diffed tx replacement
propagation: basically broadcast a diff between the original tx and the
proposed replacement, allowing you to do the minimum bandwidth accounting based
on the size of the diff instead.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Russell O'Connor]

[-- Attachment #1: Type: text/plain, Size: 1774 bytes --]

On Mon, Feb 12, 2018 at 5:58 PM, Peter Todd <pete@petertodd.org> wrote:

>
> I don't actually see where the problem is here. First of all, suppose we
> have a
> transaction T_a that already pays Alice with a feerate sufficiently high
> that
> we expect it to get mined in the near future. If we want to pay Bob, we
> can do
> that by simply creating a double-spend of T_a that pays both Bob and Alice,
> T_{ab}. BIP125 only requires that double-spend to have an absolute fee
> higher
> than the minimum relay feerate * size of the transaction.
>

The problem is that rule 3 of BIP 125 requires you pay a fee that is higher
than the the fee of T_a *plus* the fee of the sweep-transaction that the
Alice has added as a unconfirmed child transaction to T_a because
double-spending to pay Alice and Bob invalidates Alice's
sweep-transaction.  Alice's sweep-transaction is very large, and hence pays
a large absolute fee even though her fee-rate is very low.  We do not have
any control over its value, hence Alice has "pinned" our RBF transaction.

> 3'. The replacement transaction pays a fee rate of at least the effective
> > fee rate of any chain of transactions from the set of original
> transactions
> > that begins with the root of the original transaction set.
>
> I think what you mean here should be the effective fee rate of the maximum
> feerate package that can be built from the set of transactions that begins
> with
> the candidate replacement. But actually calculating this is I believe
> non-trivial, which is why I didn't implement it this way when RBF was first
> implemented.
>

Yes, that is what I mean.  My proposal was off-the-mark.

Surely CPFP is already computing the package-fee rates of mempool
transactions.  That is the value we need to compute.

[-- Attachment #2: Type: text/html, Size: 2469 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[rhavar]

 On February 12, 2018 5:58 PM, Peter Todd via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> I don't actually see where the problem is here. First of all, suppose we have a
> transaction T_a that already pays Alice with a feerate sufficiently high that
> we expect it to get mined in the near future. If we want to pay Bob, we can do
> that by simply creating a double-spend of T_a that pays both Bob and Alice,
> T_{ab}. BIP125 only requires that double-spend to have an absolute fee higher
> than the minimum relay feerate * size of the transaction.

It's a bit of a made up term, but when Russel said "pinned" it refers to a child transaction that was made  (i.e. in this case by Alice) and for us to replace our transaction we need to "unpin" it.

However to unpin it, the current bip125 rules require you pay not only the relay of the transactions you're throwing away, but the absolute fee. This is general is cost prohibitive, as even a normalish transaction can be spending $20 in fees. 

Also FWIW this whole idea of T_a and T_ab  is good, but it's also pretty impractical at the moment due to the sheer amount complexity it introduces (i.e. monitoring, seeing which confirms, trying to rebroadcast the missing one in a way that is safe against reorgs, blah blah).



>
> I just checked one of my nodes, and the absolute minimum relay fee is about
> 1/5th that of what estimatefee returns for the longest possible estimate, 48
> blocks.

If you use estimatesmartfee you should be able to get estimates all the way to 1008 or something btw

> [...]
> I think this is very important. For example, without this condition I could do
> a DoS attack by repeatedly broadcasting a transaction, then spending the
> outputs of that transaction with a very large number of child transactions, all
> of minimum fee. 

I agree.

>
> A better way to solve this class of problems may be diffed tx replacement
> propagation: basically broadcast a diff between the original tx and the
> proposed replacement, allowing you to do the minimum bandwidth accounting based
> on the size of the diff instead.

This would definitely work for some specific use-case. For instance currently if you do n replacements of a transaction, each time adding an additional output .. you need to pay something like O(n^2) relay fee. If you used a diff instead, you could probably get it to O(n)ish. 

But relay fee (and n) at the moment, mean it's not a big deal at all. The big flaw (imo) in bip125 is that you need to pay the absolute fee from the transactions you are evicting. And that can be from transactions you didn't even generate yourself.  We can already compactly represent the diff  (the new transaction invalidates it)  the debate is more "Should you have to pay the absolute fee or just relay fee for the stuff you invalidate"

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 2260 bytes --]

On Mon, Feb 12, 2018 at 06:19:40PM -0500, Russell O'Connor wrote:
> On Mon, Feb 12, 2018 at 5:58 PM, Peter Todd <pete@petertodd.org> wrote:
> 
> >
> > I don't actually see where the problem is here. First of all, suppose we
> > have a
> > transaction T_a that already pays Alice with a feerate sufficiently high
> > that
> > we expect it to get mined in the near future. If we want to pay Bob, we
> > can do
> > that by simply creating a double-spend of T_a that pays both Bob and Alice,
> > T_{ab}. BIP125 only requires that double-spend to have an absolute fee
> > higher
> > than the minimum relay feerate * size of the transaction.
> >
> 
> The problem is that rule 3 of BIP 125 requires you pay a fee that is higher
> than the the fee of T_a *plus* the fee of the sweep-transaction that the
> Alice has added as a unconfirmed child transaction to T_a because
> double-spending to pay Alice and Bob invalidates Alice's
> sweep-transaction.  Alice's sweep-transaction is very large, and hence pays
> a large absolute fee even though her fee-rate is very low.  We do not have
> any control over its value, hence Alice has "pinned" our RBF transaction.

Ah ok, I misunderstood and didn't realise you were talking about the case where
Alice re-spends her unconfirmed payment. Unfortunately I don't think that case
is possible to solve without putting some kind of restriction on spending
unconfirmed outputs; with a restriction it's fairly simple to solve.

> > I think what you mean here should be the effective fee rate of the maximum
> > feerate package that can be built from the set of transactions that begins
> > with
> > the candidate replacement. But actually calculating this is I believe
> > non-trivial, which is why I didn't implement it this way when RBF was first
> > implemented.
> >
> 
> Yes, that is what I mean.  My proposal was off-the-mark.
> 
> Surely CPFP is already computing the package-fee rates of mempool
> transactions.  That is the value we need to compute.

True, maybe we can just reuse the CPFP calculation now. That said, AFAIK that's
only done in the miner code, not the mempool, so that may not be trivial to
actually do.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Russell O'Connor]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

On Mon, Feb 12, 2018 at 6:42 PM, Peter Todd <pete@petertodd.org> wrote:

> On Mon, Feb 12, 2018 at 06:19:40PM -0500, Russell O'Connor wrote:
> > On Mon, Feb 12, 2018 at 5:58 PM, Peter Todd <pete@petertodd.org> wrote:
> >
> > >
> > > I don't actually see where the problem is here. First of all, suppose
> we
> > > have a
> > > transaction T_a that already pays Alice with a feerate sufficiently
> high
> > > that
> > > we expect it to get mined in the near future. If we want to pay Bob, we
> > > can do
> > > that by simply creating a double-spend of T_a that pays both Bob and
> Alice,
> > > T_{ab}. BIP125 only requires that double-spend to have an absolute fee
> > > higher
> > > than the minimum relay feerate * size of the transaction.
> > >
> >
> > The problem is that rule 3 of BIP 125 requires you pay a fee that is
> higher
> > than the the fee of T_a *plus* the fee of the sweep-transaction that the
> > Alice has added as a unconfirmed child transaction to T_a because
> > double-spending to pay Alice and Bob invalidates Alice's
> > sweep-transaction.  Alice's sweep-transaction is very large, and hence
> pays
> > a large absolute fee even though her fee-rate is very low.  We do not
> have
> > any control over its value, hence Alice has "pinned" our RBF transaction.
>
> Ah ok, I misunderstood and didn't realise you were talking about the case
> where
> Alice re-spends her unconfirmed payment. Unfortunately I don't think that
> case
> is possible to solve without putting some kind of restriction on spending
> unconfirmed outputs; with a restriction it's fairly simple to solve.
>

Adding such a restriction was Rhavar's original suggestion in
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-July/014688.html,
but it seems the proposal wasn't well received because it kinda destroys
CPFP.

[-- Attachment #2: Type: text/html, Size: 2579 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 3345 bytes --]

On Mon, Feb 12, 2018 at 06:23:12PM -0500, rhavar@protonmail.com wrote:
> 
>  On February 12, 2018 5:58 PM, Peter Todd via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> > I don't actually see where the problem is here. First of all, suppose we have a
> > transaction T_a that already pays Alice with a feerate sufficiently high that
> > we expect it to get mined in the near future. If we want to pay Bob, we can do
> > that by simply creating a double-spend of T_a that pays both Bob and Alice,
> > T_{ab}. BIP125 only requires that double-spend to have an absolute fee higher
> > than the minimum relay feerate * size of the transaction.
> 
> It's a bit of a made up term, but when Russel said "pinned" it refers to a child transaction that was made  (i.e. in this case by Alice) and for us to replace our transaction we need to "unpin" it.

Yeah, sorry, I just misread what scenario you guys were talking about. IIRC the
term "pinned" may have even been invented by myself, as IIRC I noticed the
issue when the RBF patch was being developed years ago. I don't think I had a
solution at the time so I just punted on it.

> However to unpin it, the current bip125 rules require you pay not only the relay of the transactions you're throwing away, but the absolute fee. This is general is cost prohibitive, as even a normalish transaction can be spending $20 in fees. 
> 
> Also FWIW this whole idea of T_a and T_ab  is good, but it's also pretty impractical at the moment due to the sheer amount complexity it introduces (i.e. monitoring, seeing which confirms, trying to rebroadcast the missing one in a way that is safe against reorgs, blah blah).

I'm not sure that's actually true, as you're only creating transactions sets
that are reorg safe. Though I don't have a detailed design in mind so I may be
missing something.

> > A better way to solve this class of problems may be diffed tx replacement
> > propagation: basically broadcast a diff between the original tx and the
> > proposed replacement, allowing you to do the minimum bandwidth accounting based
> > on the size of the diff instead.
> 
> This would definitely work for some specific use-case. For instance currently if you do n replacements of a transaction, each time adding an additional output .. you need to pay something like O(n^2) relay fee. If you used a diff instead, you could probably get it to O(n)ish. 
> 
> But relay fee (and n) at the moment, mean it's not a big deal at all. The big flaw (imo) in bip125 is that you need to pay the absolute fee from the transactions you are evicting. And that can be from transactions you didn't even generate yourself.  We can already compactly represent the diff  (the new transaction invalidates it)  the debate is more "Should you have to pay the absolute fee or just relay fee for the stuff you invalidate"

Yes, the diff approach doesn't help for the pinned case.

Unfortunately the only solution I have is basically the same as what you
proposed(1) months ago: limit spends of unconfirmed outputs in some way.

So here's a question: how many wallets have actually implemented CPFP fee bumps
for incoming transactions?

1) https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-July/014688.html

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[rhavar]

 On February 13, 2018 1:40 PM, Peter Todd <pete@petertodd.org> wrote:

> Yeah, sorry, I just misread what scenario you guys were talking about. IIRC the
> term "pinned" may have even been invented by myself, as IIRC I noticed the
> issue when the RBF patch was being developed years ago. I don't think I had a
> solution at the time so I just punted on it.

Yeah. I posted that before it was clarified, it's just my message got held up in the moderation queue so it came out of order at an inconvenient time ><



> I'm not sure that's actually true, as you're only creating transactions sets
> that are reorg safe. Though I don't have a detailed design in mind so I may be
> missing something.

It is. T_a and T_ab are "reorg" safe, but if T_a confirms you will still need to pay Bob in way. But you need to pay him such that in a reorg occurs and suddenly T_ab is mined, you haven't doubled paid him. 

I've been working on it's implementation, but it's honestly really complex and hard to test. I outlined the procedure here: https://gist.github.com/RHavar/cff76a026ece8446c898470db4f35682  which I call "Super Withdrawals".


My point though isn't that it's impossible, it's that it's sufficiently complex that it's unreasonable to expect anyone to be doing it any time soon. By relaxing any unnecessary restrictions on bip125, just makes it _drastically_ easier to do certain things.

> So here's a question: how many wallets have actually implemented CPFP fee bumps
> for incoming transactions?

Never tried it, but I recall seeing it in the electrum gui. I originally tried supporting this myself, but it's kind of annoying. It's  generally a bit cost-prohibitive to create a transaction specifically for the purpose of a CPFP fee bump, but since I made transactions pretty frequently (averaged say every 8 minutes) it doesn't add an additional input for the purpose of bumping selected incoming transactions.

The work flow is reasonably smooth: Alice has sent me 1 BTC with low fees, I owe Bob some money. I source Alice's output in the payment to Bob, giving her transaction a fee bump. Both transactions confirm, everyone is happy.

However during the whole time I need to watch Alice's transaction because if it ever is replaced/conflicted, I need to immediately pay Bob (in a reorg safe way, so I don't double-pay). It's not terribly hard to do, by making sure when I pay Bob I use an additional input that I also use for any "repayment" but it's enough complexity and hard enough to test that I gave up.

The really nice thing of most current send systems (and now especially so with segwit) is everything is pretty much fire and forget.  (although I just schedule in 0.5, 1, 2, 4, .... 32 hours fee bump attempts. But that's just background that can fail/succeed blindly)






>
>1. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-July/014688.html
>
> --
>https://petertodd.org 'peter'[:-1]@petertodd.org
>
>

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Russell O'Connor]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

On Mon, Feb 12, 2018 at 6:42 PM, Peter Todd <pete@petertodd.org> wrote:

> On Mon, Feb 12, 2018 at 06:19:40PM -0500, Russell O'Connor wrote:
> > Surely CPFP is already computing the package-fee rates of mempool
> > transactions.  That is the value we need to compute.
>
> True, maybe we can just reuse the CPFP calculation now. That said, AFAIK
> that's
> only done in the miner code, not the mempool, so that may not be trivial to
> actually do.
>

Do you (or anyone else) know if the package fee rate is considered when
ejecting transactions from the bottom of the mempool when the mempool gets
too large?

[-- Attachment #2: Type: text/html, Size: 998 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Greg Sanders]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

Yes.

On Wed, Feb 14, 2018 at 9:08 AM, Russell O'Connor via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Mon, Feb 12, 2018 at 6:42 PM, Peter Todd <pete@petertodd.org> wrote:
>
>> On Mon, Feb 12, 2018 at 06:19:40PM -0500, Russell O'Connor wrote:
>> > Surely CPFP is already computing the package-fee rates of mempool
>> > transactions.  That is the value we need to compute.
>>
>> True, maybe we can just reuse the CPFP calculation now. That said, AFAIK
>> that's
>> only done in the miner code, not the mempool, so that may not be trivial
>> to
>> actually do.
>>
>
> Do you (or anyone else) know if the package fee rate is considered when
> ejecting transactions from the bottom of the mempool when the mempool gets
> too large?
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 1873 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Russell O'Connor]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

On Mon, Feb 12, 2018 at 6:42 PM, Peter Todd <pete@petertodd.org> wrote:

>
> Ah ok, I misunderstood and didn't realise you were talking about the case
> where
> Alice re-spends her unconfirmed payment. Unfortunately I don't think that
> case
> is possible to solve without putting some kind of restriction on spending
> unconfirmed outputs; with a restriction it's fairly simple to solve.


When you say that you don't think it is possible to solve, do you mean that
there is a specific problem with this proposal of replacing transactions
when offered a new transaction whose fee rate exceeds the package fee rate
of the original transaction (and ensuring that the fee increase covers the
size of the transactions being ejected)?  Is your concern only about the
ability to computing and track the package fee rate for transactions within
the mempool or is there some other issue you foresee?

[-- Attachment #2: Type: text/html, Size: 1275 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]

On Tue, Feb 27, 2018 at 11:25:59AM -0500, Russell O'Connor wrote:
> On Mon, Feb 12, 2018 at 6:42 PM, Peter Todd <pete@petertodd.org> wrote:
> 
> >
> > Ah ok, I misunderstood and didn't realise you were talking about the case
> > where
> > Alice re-spends her unconfirmed payment. Unfortunately I don't think that
> > case
> > is possible to solve without putting some kind of restriction on spending
> > unconfirmed outputs; with a restriction it's fairly simple to solve.
> 
> 
> When you say that you don't think it is possible to solve, do you mean that
> there is a specific problem with this proposal of replacing transactions
> when offered a new transaction whose fee rate exceeds the package fee rate
> of the original transaction (and ensuring that the fee increase covers the
> size of the transactions being ejected)?  Is your concern only about the
> ability to computing and track the package fee rate for transactions within
> the mempool or is there some other issue you foresee?

I mean, I think in general solving this problem is probably not possible.
Basically, the fundamental problem is someone else has consumed network
bandwidth that should be paid for with fees. What you're trying to do is
replace a transaction without paying those fees, which is identical to what an
attacker is trying to do, and thus any such scheme will be as vulnerable to
attack as not having that protection in the first place.

...which does give you an out: maybe the attack isn't important enough to
matter. :)

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Russell O'Connor]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

On Thu, Mar 1, 2018 at 10:11 AM, Peter Todd <pete@petertodd.org> wrote:

> On Tue, Feb 27, 2018 at 11:25:59AM -0500, Russell O'Connor wrote:
> > When you say that you don't think it is possible to solve, do you mean
> that
> > there is a specific problem with this proposal of replacing transactions
> > when offered a new transaction whose fee rate exceeds the package fee
> rate
> > of the original transaction (and ensuring that the fee increase covers
> the
> > size of the transactions being ejected)?  Is your concern only about the
> > ability to computing and track the package fee rate for transactions
> within
> > the mempool or is there some other issue you foresee?
>
> I mean, I think in general solving this problem is probably not possible.
> Basically, the fundamental problem is someone else has consumed network
> bandwidth that should be paid for with fees. What you're trying to do is
> replace a transaction without paying those fees, which is identical to
> what an
> attacker is trying to do, and thus any such scheme will be as vulnerable to
> attack as not having that protection in the first place.
>
> ...which does give you an out: maybe the attack isn't important enough to
> matter. :)
>

Thanks, that makes sense.

I still think it is worthwhile pursuing this proposed change in RBF policy
as it would seem that the current policy is problematic in practice today
where participants are just performing normal transactions and are not
trying to attack each other.

[-- Attachment #2: Type: text/html, Size: 1969 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

On Thu, Mar 08, 2018 at 10:39:46AM -0500, Russell O'Connor wrote:
> On Thu, Mar 1, 2018 at 10:11 AM, Peter Todd <pete@petertodd.org> wrote:
> > I mean, I think in general solving this problem is probably not possible.
> > Basically, the fundamental problem is someone else has consumed network
> > bandwidth that should be paid for with fees. What you're trying to do is
> > replace a transaction without paying those fees, which is identical to
> > what an
> > attacker is trying to do, and thus any such scheme will be as vulnerable to
> > attack as not having that protection in the first place.
> >
> > ...which does give you an out: maybe the attack isn't important enough to
> > matter. :)
> >
> 
> Thanks, that makes sense.
> 
> I still think it is worthwhile pursuing this proposed change in RBF policy
> as it would seem that the current policy is problematic in practice today
> where participants are just performing normal transactions and are not
> trying to attack each other.

But that's not a good argument: whether or not normal users are trying to
attack each other has nothing to do with whether or not you're opening up an
attack by relaxing anti-DoS protections.

Equally, how often are normal users who aren't attacking each other creating
issues anyway? You can always have your wallet code just skip use of RBF
replacements in the event that someone does spend an unconfirmed output that
you sent them; how often does this actually happen in practice? Not many
wallets let you spend unconfirmed outputs that you didn't create.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 614 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Russell O'Connor]

[-- Attachment #1: Type: text/plain, Size: 2875 bytes --]

On Thu, Mar 8, 2018 at 1:34 PM, Peter Todd <pete@petertodd.org> wrote:

> On Thu, Mar 08, 2018 at 10:39:46AM -0500, Russell O'Connor wrote:
> > On Thu, Mar 1, 2018 at 10:11 AM, Peter Todd <pete@petertodd.org> wrote:
> > > I mean, I think in general solving this problem is probably not
> possible.
> > > Basically, the fundamental problem is someone else has consumed network
> > > bandwidth that should be paid for with fees. What you're trying to do
> is
> > > replace a transaction without paying those fees, which is identical to
> > > what an
> > > attacker is trying to do, and thus any such scheme will be as
> vulnerable to
> > > attack as not having that protection in the first place.
> > >
> > > ...which does give you an out: maybe the attack isn't important enough
> to
> > > matter. :)
> > >
> >
> > Thanks, that makes sense.
> >
> > I still think it is worthwhile pursuing this proposed change in RBF
> policy
> > as it would seem that the current policy is problematic in practice today
> > where participants are just performing normal transactions and are not
> > trying to attack each other.
>
> But that's not a good argument: whether or not normal users are trying to
> attack each other has nothing to do with whether or not you're opening up
> an
> attack by relaxing anti-DoS protections.
>

I'm not suggesting removing the anti-DoS protections.  I'm suggesting that
replaced transaction require a fee increase of at least the min-fee-rate
times the size of all the transactions being ejected (in addition to the
other proposed requirements).


> Equally, how often are normal users who aren't attacking each other
> creating
> issues anyway? You can always have your wallet code just skip use of RBF
>
replacements in the event that someone does spend an unconfirmed output that
> you sent them; how often does this actually happen in practice?


Just ask rhavar.  It happens regularly.

Not many wallets let you spend unconfirmed outputs that you didn't create.
>

The problem is with institutional wallets sweeping incoming payments.  It
seems that in practice they are happy to sweep unconfirmed outputs.

Setting all of the above aside for a moment.  We need to understand that
rational miners are going to prefer to transactions with higher package fee
rates regardless of whatever your personal preferred RBF policy is.  If we
do not bring the RBF policy to alignment with what is economically
rational, then miners are going to change their own policies anyways,
probably all in slightly different ways.  It behooves everyone to develop a
reasonable standard RBF policy, that is still robust against possible DoS
vectors, and aligns with miner incentives, so that all participants know
what behaviour they can reasonably expect.  It is simply a bonus that this
change in RBF policy also partially mitigates the problem of pinned
transactions.

[-- Attachment #2: Type: text/html, Size: 4039 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 2896 bytes --]

On Thu, Mar 08, 2018 at 03:07:43PM -0500, Russell O'Connor wrote:
> On Thu, Mar 8, 2018 at 1:34 PM, Peter Todd <pete@petertodd.org> wrote:
> > But that's not a good argument: whether or not normal users are trying to
> > attack each other has nothing to do with whether or not you're opening up
> > an
> > attack by relaxing anti-DoS protections.
> >
> 
> I'm not suggesting removing the anti-DoS protections.  I'm suggesting that
> replaced transaction require a fee increase of at least the min-fee-rate
> times the size of all the transactions being ejected (in addition to the
> other proposed requirements).

Fair: you're not removing them entirely, but you are weakening them compared to
the status quo.

> > Equally, how often are normal users who aren't attacking each other
> > creating
> > issues anyway? You can always have your wallet code just skip use of RBF
> >
> replacements in the event that someone does spend an unconfirmed output that
> > you sent them; how often does this actually happen in practice?
> 
> 
> Just ask rhavar.  It happens regularly.
> 
> Not many wallets let you spend unconfirmed outputs that you didn't create.
> >
> 
> The problem is with institutional wallets sweeping incoming payments.  It
> seems that in practice they are happy to sweep unconfirmed outputs.

Pity, that does sound like a problem. :(

> Setting all of the above aside for a moment.  We need to understand that
> rational miners are going to prefer to transactions with higher package fee
> rates regardless of whatever your personal preferred RBF policy is.  If we
> do not bring the RBF policy to alignment with what is economically
> rational, then miners are going to change their own policies anyways,
> probably all in slightly different ways.  It behooves everyone to develop a
> reasonable standard RBF policy, that is still robust against possible DoS
> vectors, and aligns with miner incentives, so that all participants know
> what behaviour they can reasonably expect.  It is simply a bonus that this
> change in RBF policy also partially mitigates the problem of pinned
> transactions.

Miners and full nodes have slightly different priorities here; it's not clear
to me why it matters that they implement slightly different policies.


Still, re-reading your initital post, I'm convinced that the weakening of the
DoS protections is probably not a huge problem, so maybe lets try this in a
release and see what happens.

Notably, if people actually use this new replacement behavior, the institutions
doing these sweeps of unconfirmed outputs might stop doing that! That's
probably a good thing, as respends of potentially conflicted unconfirmed
outputs can be dangerous in reorgs; we're better off if outputs are buried
deeply before being spent again.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 614 bytes --]

Re: [bitcoin-dev] Revisiting BIP 125 RBF policy.

[rhavar]

> Still, re-reading your initital post, I'm convinced that the weakening of the
> DoS protections is probably not a huge problem, so maybe lets try this in a
> release and see what happens.

Awesome! I very much agree. The relaxation of some of these DoS prevention rules I think will really open up a lot of use cases and adoption 

> Notably, if people actually use this new replacement behavior, the institutions
> doing these sweeps of unconfirmed outputs might stop doing that! 

Agree, I'm pretty sure it's unintentional. I know a lot of services struggle with coin selection, so what they do is conceptually have a receive wallet from which they can sweep to their hot wallet (or cold storage) to keep their utxo manageable.

Currently some of them are sweeping unconfirmed inputs with it, but I don't think it's a conscious design choice, just something that happens to be working well now.

(FWIW I observed this behavior like 6+ months ago, I haven't kept track of if it's still happening or how often. But at the time I had to write off the idea of low-fee rbf batch transactions as it was happening too often to be feasible)


​-Ryan​

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On March 9, 2018 1:28 PM, Peter Todd via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> ​​
> 
> On Thu, Mar 08, 2018 at 03:07:43PM -0500, Russell O'Connor wrote:
> 
> > On Thu, Mar 8, 2018 at 1:34 PM, Peter Todd pete@petertodd.org wrote:
> > 
> > > But that's not a good argument: whether or not normal users are trying to
> > > 
> > > attack each other has nothing to do with whether or not you're opening up
> > > 
> > > an
> > > 
> > > attack by relaxing anti-DoS protections.
> > 
> > I'm not suggesting removing the anti-DoS protections. I'm suggesting that
> > 
> > replaced transaction require a fee increase of at least the min-fee-rate
> > 
> > times the size of all the transactions being ejected (in addition to the
> > 
> > other proposed requirements).
> 
> Fair: you're not removing them entirely, but you are weakening them compared to
> 
> the status quo.
> 
> > > Equally, how often are normal users who aren't attacking each other
> > > 
> > > creating
> > > 
> > > issues anyway? You can always have your wallet code just skip use of RBF
> > 
> > replacements in the event that someone does spend an unconfirmed output that
> > 
> > > you sent them; how often does this actually happen in practice?
> > 
> > Just ask rhavar. It happens regularly.
> > 
> > Not many wallets let you spend unconfirmed outputs that you didn't create.
> > 
> > > 
> > 
> > The problem is with institutional wallets sweeping incoming payments. It
> > 
> > seems that in practice they are happy to sweep unconfirmed outputs.
> 
> Pity, that does sound like a problem. :(
> 
> > Setting all of the above aside for a moment. We need to understand that
> > 
> > rational miners are going to prefer to transactions with higher package fee
> > 
> > rates regardless of whatever your personal preferred RBF policy is. If we
> > 
> > do not bring the RBF policy to alignment with what is economically
> > 
> > rational, then miners are going to change their own policies anyways,
> > 
> > probably all in slightly different ways. It behooves everyone to develop a
> > 
> > reasonable standard RBF policy, that is still robust against possible DoS
> > 
> > vectors, and aligns with miner incentives, so that all participants know
> > 
> > what behaviour they can reasonably expect. It is simply a bonus that this
> > 
> > change in RBF policy also partially mitigates the problem of pinned
> > 
> > transactions.
> 
> Miners and full nodes have slightly different priorities here; it's not clear
> 
> to me why it matters that they implement slightly different policies.
> 
> Still, re-reading your initital post, I'm convinced that the weakening of the
> 
> DoS protections is probably not a huge problem, so maybe lets try this in a
> 
> release and see what happens.
> 
> Notably, if people actually use this new replacement behavior, the institutions
> 
> doing these sweeps of unconfirmed outputs might stop doing that! That's
> 
> probably a good thing, as respends of potentially conflicted unconfirmed
> 
> outputs can be dangerous in reorgs; we're better off if outputs are buried
> 
> deeply before being spent again.
> 
> 
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> https://petertodd.org 'peter'\[:-1\]@petertodd.org
> 
> bitcoin-dev mailing list
> 
> bitcoin-dev@lists.linuxfoundation.org
> 
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev