[bitcoin-dev] Fee estimates and RBF

[bitcoin-dev] Fee estimates and RBF

[Prayank]

[-- Attachment #1: Type: text/plain, Size: 2519 bytes --]

Hello World, 

I hope everyone is doing okay. Things are not good in India and even I was tested covid positive few days back. Recovered and feeling better now. Hoping everything gets back to normal soon.

There are different estimations used in wallets, explorers and other Bitcoin projects. For example: `estimatesmartfee` in Bitcoin Core (One of the implementation for Bitcoin which is used more but not official as there is nothing official in Bitcoin).  Are different estimations misleading and affect the way fees are used in Bitcoin transactions? Will it be better if we just share mempool stats and user can decide the fee rate accordingly?

If I compare this with BTCUSD orderbook on any exchange, are we trying to estimate at what price buy order will get filled in certain time? Does that make sense?

Mempool Stats: https://i.imgur.com/r4XKk2p.png
BTCUSD Orderbook: https://i.imgur.com/ylGVHJB.png

I consider it misleading because lot of users think a transaction with fee rate 1-5 sat/vByte will be included in 1 week or maybe a transaction with X sat/VByte will be included in Y time which is not true. Users can decide the fee rate and can do bidding, transaction will be included based on demand, supply and miners.

Will it be better if the wallets used this approach?
1.Show mempool stats
2.Leave the fee rate for user to decide
3.RBF every transaction and follow different algorithms for automated bidding

A basic algorithm for automated bidding can be: https://i.stack.imgur.com/1SlPv.png

Such RBF algos can be helpful for users when Bitcoin wallets are open in background. Maybe it will work better for mobile wallets in which you can see a notification every time transaction is replaced with a new fee rate automatically.

I wanted to know what others think about this approach of creating and using different RBF algos instead of predicting something that is difficult or doesn't make sense. Also if there was a way we could achieve this even if the user goes offline. For example: Alice broadcasts Tx1 with 1 sat/vByte, its replaced with Tx2 (2 sat/vByte) after 2 blocks because Tx1 was not confirmed. Alice decides to shut down her system or switch off mobile or mobile data. Tx2 is still not confirmed after another 2 blocks but it has some information as one OP_RETURN output which is used by Bitcoin nodes that see this transaction in the mempool. Bob's node use this information to replace the transaction with Tx3 and use fee rate 3 sat/vByte.

--
Prayank


[-- Attachment #2: Type: text/html, Size: 3148 bytes --]

Re: [bitcoin-dev] Fee estimates and RBF

[Jeremy]

[-- Attachment #1: Type: text/plain, Size: 4041 bytes --]

Hi Prayank,

Very glad to hear you are weathering the storm OK, wishing you and yours
safety in the weeks ahead.

I think you'll be interested to see
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-September/018168.html
especially with regards to background services.

In the long term, this can be particularly useful since you can use a
separate fee-only wallet to arrange the bumps if your main wallet keys are
offline, and you do not disturb any of the on-chain dependants.

It can also be much cheaper to use this mechanism than actual RBF because
you are not subject to the feerate improvement rule, which forces you to
pay for the bump on all tx data.

I would be very interested to see a system whereby you can make a market
(maybe via LN) for someone to get your TX included (using oracle network
OK) by a particular date. Then you can abstract the whole system a lot
better, so that you can fee bump without having to create any direct on
chain traffic, and a sponsor vector can be offered across a number of txns
by a service provider.

Cheers,

Jeremy
--
@JeremyRubin <https://twitter.com/JeremyRubin>
<https://twitter.com/JeremyRubin>


On Fri, Apr 30, 2021 at 2:40 PM Prayank via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hello World,
>
> I hope everyone is doing okay. Things are not good in India and even I was
> tested covid positive few days back. Recovered and feeling better now.
> Hoping everything gets back to normal soon.
>
> There are different estimations used in wallets, explorers and other
> Bitcoin projects. For example: `estimatesmartfee` in Bitcoin Core (One of
> the implementation for Bitcoin which is used more but not official as there
> is nothing official in Bitcoin).  Are different estimations misleading and
> affect the way fees are used in Bitcoin transactions? Will it be better if
> we just share mempool stats and user can decide the fee rate accordingly?
>
> If I compare this with BTCUSD orderbook on any exchange, are we trying to
> estimate at what price buy order will get filled in certain time? Does that
> make sense?
>
> Mempool Stats: https://i.imgur.com/r4XKk2p.png
> BTCUSD Orderbook: https://i.imgur.com/ylGVHJB.png
>
> I consider it misleading because lot of users think a transaction with fee
> rate 1-5 sat/vByte will be included in 1 week or maybe a transaction with X
> sat/VByte will be included in Y time which is not true. Users can decide
> the fee rate and can do bidding, transaction will be included based on
> demand, supply and miners.
>
> Will it be better if the wallets used this approach?
> 1.Show mempool stats
> 2.Leave the fee rate for user to decide
> 3.RBF every transaction and follow different algorithms for automated
> bidding
>
> A basic algorithm for automated bidding can be:
> https://i.stack.imgur.com/1SlPv.png
>
> Such RBF algos can be helpful for users when Bitcoin wallets are open in
> background. Maybe it will work better for mobile wallets in which you can
> see a notification every time transaction is replaced with a new fee rate
> automatically.
>
> I wanted to know what others think about this approach of creating and
> using different RBF algos instead of predicting something that is difficult
> or doesn't make sense. Also if there was a way we could achieve this even
> if the user goes offline. For example: Alice broadcasts Tx1 with 1
> sat/vByte, its replaced with Tx2 (2 sat/vByte) after 2 blocks because Tx1
> was not confirmed. Alice decides to shut down her system or switch off
> mobile or mobile data. Tx2 is still not confirmed after another 2 blocks
> but it has some information as one OP_RETURN output which is used by
> Bitcoin nodes that see this transaction in the mempool. Bob's node use this
> information to replace the transaction with Tx3 and use fee rate 3
> sat/vByte.
>
> --
> Prayank
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 6992 bytes --]

Re: [bitcoin-dev] Fee estimates and RBF

[Prayank]

[-- Attachment #1: Type: text/plain, Size: 4398 bytes --]

Thanks Jeremy for sharing this link: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-September/018168.html

Trying to understand everything mentioned and "fee-only" wallet sounds interesting.
-- 
Prayank

May 1, 2021, 05:41 by jlrubin@mit.edu:

> Hi Prayank,
>
> Very glad to hear you are weathering the storm OK, wishing you and yours safety in the weeks ahead.
>
> I think you'll be interested to see > https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-September/018168.html>  especially with regards to background services.
>
> In the long term, this can be particularly useful since you can use a separate fee-only wallet to arrange the bumps if your main wallet keys are offline, and you do not disturb any of the on-chain dependants.
>
> It can also be much cheaper to use this mechanism than actual RBF because you are not subject to the feerate improvement rule, which forces you to pay for the bump on all tx data.
>
> I would be very interested to see a system whereby you can make a market (maybe via LN) for someone to get your TX included (using oracle network OK) by a particular date. Then you can abstract the whole system a lot better, so that you can fee bump without having to create any direct on chain traffic, and a sponsor vector can be offered across a number of txns by a service provider.
>
> Cheers,
>
> Jeremy
> --
> @JeremyRubin <https://twitter.com/JeremyRubin> <https://twitter.com/JeremyRubin>
>
>
> On Fri, Apr 30, 2021 at 2:40 PM Prayank via bitcoin-dev <> bitcoin-dev@lists.linuxfoundation.org> > wrote:
>
>> Hello World, 
>>
>> I hope everyone is doing okay. Things are not good in India and even I was tested covid positive few days back. Recovered and feeling better now. Hoping everything gets back to normal soon.
>>
>> There are different estimations used in wallets, explorers and other Bitcoin projects. For example: `estimatesmartfee` in Bitcoin Core (One of the implementation for Bitcoin which is used more but not official as there is nothing official in Bitcoin).  Are different estimations misleading and affect the way fees are used in Bitcoin transactions? Will it be better if we just share mempool stats and user can decide the fee rate accordingly?
>>
>> If I compare this with BTCUSD orderbook on any exchange, are we trying to estimate at what price buy order will get filled in certain time? Does that make sense?
>>
>> Mempool Stats: >> https://i.imgur.com/r4XKk2p.png
>> BTCUSD Orderbook: >> https://i.imgur.com/ylGVHJB.png
>>
>> I consider it misleading because lot of users think a transaction with fee rate 1-5 sat/vByte will be included in 1 week or maybe a transaction with X sat/VByte will be included in Y time which is not true. Users can decide the fee rate and can do bidding, transaction will be included based on demand, supply and miners.
>>
>> Will it be better if the wallets used this approach?
>> 1.Show mempool stats
>> 2.Leave the fee rate for user to decide
>> 3.RBF every transaction and follow different algorithms for automated bidding
>>
>> A basic algorithm for automated bidding can be: >> https://i.stack.imgur.com/1SlPv.png
>>
>> Such RBF algos can be helpful for users when Bitcoin wallets are open in background. Maybe it will work better for mobile wallets in which you can see a notification every time transaction is replaced with a new fee rate automatically.
>>
>> I wanted to know what others think about this approach of creating and using different RBF algos instead of predicting something that is difficult or doesn't make sense. Also if there was a way we could achieve this even if the user goes offline. For example: Alice broadcasts Tx1 with 1 sat/vByte, its replaced with Tx2 (2 sat/vByte) after 2 blocks because Tx1 was not confirmed. Alice decides to shut down her system or switch off mobile or mobile data. Tx2 is still not confirmed after another 2 blocks but it has some information as one OP_RETURN output which is used by Bitcoin nodes that see this transaction in the mempool. Bob's node use this information to replace the transaction with Tx3 and use fee rate 3 sat/vByte.
>>
>> --
>> Prayank
>>
>> _______________________________________________
>>  bitcoin-dev mailing list
>>  >> bitcoin-dev@lists.linuxfoundation.org
>>  >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>


[-- Attachment #2: Type: text/html, Size: 7826 bytes --]

Re: [bitcoin-dev] Fee estimates and RBF

[ZmnSCPxj]

Good morning Prayank,

I believe a "true" full-RBF wallet should be what every onchain wallet aspires to.

However, I think a lot of the effort necessary here has to do with sheer engineering issues.

For example, if you think "RBF does not exist", you can do things like:

* Spend an unconfirmed input from a third party.
  * This is not actually safe since an unconfirmed tx might have a conflicting transaction get confirmed, but a lot of onchain wallets support this for non-RBF unconfirmed inputs because 99.9% of the time this never happens.
* When you spend from a (confirmed or unconfirmed) input, delete it from your db forever (because you do not have to worry about alternate transactions spending the same input).
  * This simplifies db design, you do not have to keep track of states like "has been spent but tx is not confirmed yet", "has two different alternate transactions spending it that have not confirmed", "is on a transaction that is not confirmed and therefore this input might disappear completely" etc.

In particular, if we want a "true" full-RBF wallet:

* Suppose the user wants to spend some amount to address A.
  * The user imposes a limit on up to how much to spend on fees to have this spend happen.
* The wallet optimistically creates a low-fee send transaction.
* After some time, the wallet bumps up the fee by creating a new transaction.
  * The wallet keeps bumping up, up to the designated limit, the longer the transaction is not confirmed.

Of note is that there is a *race condition* in the above case.
When the wallet is bumping up and constructing a new transaction with higher fee, a miner could find a new block that has the old transaction with lower fee.

Now consider the subsequent user story.

* After some time, the user wants to spend another amount to address B.
  * Again the user imposes a limit on how much to spend on fees to have this spend happen.
* The wallet RBFs the existing transaction to include the spend to address B.

Again, a race condition can occur --- while the wallet is feebumping a new transaction that includes the new output, a random miner can find a new block that includes the old transaction.

Thus, the wallet really needs to keep track of any "pending spends" and correlate them with actual transactions.

Further, of course it is convenient to be able to spend money even while it is unconfirmed.
But the sender of the unconfirmed input might be using the same software as this wallet as well, meaning that the actual transaction output might change as the original spender keeps fee-bumping it over time.

I confess I have not been thinking of this as well as I should have.

Regards,
ZmnSCPxj

Re: [bitcoin-dev] Fee estimates and RBF

[Prayank]

[-- Attachment #1: Type: text/plain, Size: 3196 bytes --]

Good morning ZmnSCPxj,

Thanks for your response. I agree there are few exceptions: 

1.Unconfirmed output can be spent resulting in conflict with RBF
2.Race condition and mining pool may include old transaction with low fee
I am trying few things related to RBF and handling such exceptions, will share if I find anything interesting.
-- 
Prayank

May 3, 2021, 09:32 by ZmnSCPxj@protonmail.com:

> Good morning Prayank,
>
> I believe a "true" full-RBF wallet should be what every onchain wallet aspires to.
>
> However, I think a lot of the effort necessary here has to do with sheer engineering issues.
>
> For example, if you think "RBF does not exist", you can do things like:
>
> * Spend an unconfirmed input from a third party.
>  * This is not actually safe since an unconfirmed tx might have a conflicting transaction get confirmed, but a lot of onchain wallets support this for non-RBF unconfirmed inputs because 99.9% of the time this never happens.
> * When you spend from a (confirmed or unconfirmed) input, delete it from your db forever (because you do not have to worry about alternate transactions spending the same input).
>  * This simplifies db design, you do not have to keep track of states like "has been spent but tx is not confirmed yet", "has two different alternate transactions spending it that have not confirmed", "is on a transaction that is not confirmed and therefore this input might disappear completely" etc.
>
> In particular, if we want a "true" full-RBF wallet:
>
> * Suppose the user wants to spend some amount to address A.
>  * The user imposes a limit on up to how much to spend on fees to have this spend happen.
> * The wallet optimistically creates a low-fee send transaction.
> * After some time, the wallet bumps up the fee by creating a new transaction.
>  * The wallet keeps bumping up, up to the designated limit, the longer the transaction is not confirmed.
>
> Of note is that there is a *race condition* in the above case.
> When the wallet is bumping up and constructing a new transaction with higher fee, a miner could find a new block that has the old transaction with lower fee.
>
> Now consider the subsequent user story.
>
> * After some time, the user wants to spend another amount to address B.
>  * Again the user imposes a limit on how much to spend on fees to have this spend happen.
> * The wallet RBFs the existing transaction to include the spend to address B.
>
> Again, a race condition can occur --- while the wallet is feebumping a new transaction that includes the new output, a random miner can find a new block that includes the old transaction.
>
> Thus, the wallet really needs to keep track of any "pending spends" and correlate them with actual transactions.
>
> Further, of course it is convenient to be able to spend money even while it is unconfirmed.
> But the sender of the unconfirmed input might be using the same software as this wallet as well, meaning that the actual transaction output might change as the original spender keeps fee-bumping it over time.
>
> I confess I have not been thinking of this as well as I should have.
>
> Regards,
> ZmnSCPxj
>

[-- Attachment #2: Type: text/html, Size: 4096 bytes --]

Re: [bitcoin-dev] Fee estimates and RBF

[Prayank]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

I have shared response by Jeremy and ZmnSCPxj in an answer to https://bitcoin.stackexchange.com/questions/105860/what-are-we-trying-to-predict-in-fee-estimation-and-why

Also find the recent CVE related to RBF by Antoine Riard and implementation of RBF in Bitcoin Core compared to btcd interesting. Even though I am not sure why inherited signalling is not implemented in Bitcoin Core.
RBF, CPFP and their combinations are something that is less explored IMO. For example: I had discussed one usecase of CPFP with Harding in IRC once in which a project uses maker-taker model. Maker broadcasts transaction with 1 sat/vByte and taker has to confirm this transaction by creating a child transaction with an effective fee rate according to mempool stats. Basically, the idea of receiver paying for the transaction instead of sender. But it will involve lot of exception handling.
-- 
Prayank

[-- Attachment #2: Type: text/html, Size: 1249 bytes --]

Re: [bitcoin-dev] Fee estimates and RBF

[ZmnSCPxj]

Good morning Prayank,

>  But it will involve lot of exception handling.


Yes, that is precisely the problem here.

If you select a fixed feerate and then just broadcast-and-forget, you have no real exceptions you have to handle --- but that means not using RBF at all.


Testing the handling of reorgs in particular is important, as a reorg might use an older version of an RBFed transaction rather than a newer version.
This also implies that further follow-up transactions might need to be recreated in such a case.

As this is financial code, we need a lot of testing, and code that has a lot of branches due to having to handle a lot of possible exceptions and so forth is a headache to completely cover in testing.


C-lightning supposedly supports RBF, in the sense that every transaction it makes always signals RBF, but I am almost certain there are edge cases where it might mishandle a replaced transaction and lose track of onchain funds, and it is difficult to support both "we can spend unconfirmed change outputs" (a very common feature of nearly every onchain wallet) with "we can change the feerate of unconfirmed transactions" (which changes the txid and therefore the UTXO id of the change output spent by use of the previous feature).

Regards,
ZmnSCPxj