[bitcoin-dev] Mempool "Expected Byte Stay" policy

[bitcoin-dev] Mempool "Expected Byte Stay" policy

[Tom Harding]

Spammers out there are being very disrepectful of my fullnode resources 
these days!  I'm making some changes. In case others are interested, 
here's a description:

There is now a maximum size for the memory pool.  Space is allocated 
with a pretty simple rule.  For each tx, I calculate MY COST of 
continuing to hold it in the mempool.  I measure the cost to me by 
"expected byte stay":

expectedByteStay = sizeBytes * expectedBlocksToConfirm(feeRate)


Rule 1: When there's not enough space for a new tx, I try to make space 
by evicting txes with expectedByteStay higher than tx.

I'm NOT worrying about
  - Fees
    EXCEPT via their effect on confirmation time

  - Coin age
    You already made money on your old coins.  Pay up.

  - CPFP
    Child's expectedBlocksToConfirm is max'ed with its
    parent, then parent expectedByteStay is ADDED to child's

  - Replacement
    You'll get another chance in 2 hours (see below).


Rule 2: A transaction and its dependents are evicted on its 2-hour 
anniversary, whether space is required or not


The latest expectedBlocksToConfirm(feeRate) table is applied to the 
entire mempool periodically.

What do you think?  I'll let you know how it works out.  I'm putting a 
lot of faith in the new fee estimation (particularly its size 
independence).  Another possibility is clog-ups by transactions that 
look like they'll confirm next block, but don't because of factors other 
than fees (other people's blacklists?)

Re: [bitcoin-dev] Mempool "Expected Byte Stay" policy

[Thomas Zander]

On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:
> Rule 2: A transaction and its dependents are evicted on its 2-hour 
> anniversary, whether space is required or not

Instead of 2 hours, why not a number of blocks?
-- 
Thomas Zander

Re: [bitcoin-dev] Mempool "Expected Byte Stay" policy

[Tom Harding]

On 7/15/2015 12:18 PM, Thomas Zander via bitcoin-dev wrote:
> On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:
>> Rule 2: A transaction and its dependents are evicted on its 2-hour
>> anniversary, whether space is required or not
> Instead of 2 hours, why not a number of blocks?

So users/wallets can know when they should rebroadcast and consider 
increasing the fee.


Using 12 blocks, there is a 5% chance he has to wait 3 hours.*

Using 120 minutes, there is only a .23% chance that fewer than 4 blocks 
have occurred.**





*
Table[{x, 1 - CDF[ErlangDistribution[12, 1/10], x]}, {x, 0, 240, 10}] 
//N //TableForm
0.      1.
10.    1.
20.    0.999999
30.    0.999929
40.    0.999085
50.    0.994547
60.    0.979908
70.    0.94665
80.    0.888076
90.    0.803008
100.    0.696776
110.    0.579267
120.    0.461597
130.    0.353165
140.    0.26004
150.    0.184752
160.    0.126993
170.    0.0846691
180.    0.0548874
190.    0.0346726
200.    0.0213868
210.    0.0129048
220.    0.00762994
230.    0.00442702
240.    0.00252413





**
Table[{x, CDF[PoissonDistribution[1/10 * 120], x]}, {x, 0, 20}] //N 
//TableForm
0.    6.14421*10^-6
1.    0.0000798748
2.    0.000522258
3.    0.00229179
4.    0.00760039
5.    0.020341
6.    0.0458223
7.    0.0895045
8.    0.155028
9.    0.242392
10.    0.347229
11.    0.461597
12.    0.575965
13.    0.681536
14.    0.772025
15.    0.844416
16.    0.898709
17.    0.937034
18.    0.962584
19.    0.97872
20.    0.988402

Re: [bitcoin-dev] Mempool "Expected Byte Stay" policy

[Thomas Zander]

On Wednesday 15. July 2015 16.15.24 Tom Harding via bitcoin-dev wrote:
> On 7/15/2015 12:18 PM, Thomas Zander via bitcoin-dev wrote:
> > On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:
> >> Rule 2: A transaction and its dependents are evicted on its 2-hour
> >> anniversary, whether space is required or not
> > 
> > Instead of 2 hours, why not a number of blocks?
> 
> So users/wallets can know when they should rebroadcast and consider 
> increasing the fee.
> 
> 
> Using 12 blocks, there is a 5% chance he has to wait 3 hours.*
> 
> Using 120 minutes, there is only a .23% chance that fewer than 4 blocks 
> have occurred.**

Using the good old saying; results in the past are no indication of the 
future.
I see a logic error in your thinking.

Your assumption that time is a better indicator is false. Naturally time 
itself is universal, but blocks are known by wallets too. Its just as good.

This assumption of yours leans heavily on block mining times, and that is
not guaranteed in the future.  Imagine one day half the miners dropping and 
blocks take much longer for a week or so.  Your assumptions just broke the 
mempool.

-- 
Thomas Zander

Re: [bitcoin-dev] Mempool "Expected Byte Stay" policy

[Tom Harding]

On 7/16/2015 2:38 AM, Thomas Zander via bitcoin-dev wrote:
> On Wednesday 15. July 2015 16.15.24 Tom Harding via bitcoin-dev wrote:
>> On 7/15/2015 12:18 PM, Thomas Zander via bitcoin-dev wrote:
>>> On Tuesday 14. July 2015 17.24.23 Tom Harding via bitcoin-dev wrote:
>>>> Rule 2: A transaction and its dependents are evicted on its 2-hour
>>>> anniversary, whether space is required or not
>>> Instead of 2 hours, why not a number of blocks?
>> So users/wallets can know when they should rebroadcast and consider
>> increasing the fee.
>>
>>
>> Using 12 blocks, there is a 5% chance he has to wait 3 hours.*
>>
>> Using 120 minutes, there is only a .23% chance that fewer than 4 blocks
>> have occurred.**
> Using the good old saying; results in the past are no indication of the
> future.
> I see a logic error in your thinking.
>
> Your assumption that time is a better indicator is false. Naturally time
> itself is universal, but blocks are known by wallets too. Its just as good.
>
> This assumption of yours leans heavily on block mining times, and that is
> not guaranteed in the future.  Imagine one day half the miners dropping and
> blocks take much longer for a week or so.  Your assumptions just broke the
> mempool.
>

It's not a question of right vs. wrong.  Either method has consequences 
for user expectations and behavior.

With fixed-block mempool expiration, the expiration time is variable.  
User can get an alert, but at an unpredictable time.

With fixed-timeout, the likelihood of expiration is more variable 
(expiration occurrence is unpredictable regardless), but any expiration 
will occur at the timeout.