[bitcoin-dev] Do we really need a mempool? (for relay nodes)

[bitcoin-dev] Do we really need a mempool? (for relay nodes)

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1616 bytes --]

As in, do relay nodes need to keep a record of the transactions they've
relayed? Strictly speaking, the answer is no: one a tx is relayed modulo
DoS concerns the entire thing can be discarded by the node. (unconfirmed
txs spending other unconfirmed txs can be handled by creating packages
of transactions, evaluated as a whole)

To mitigate DoS concerns, we of course have to have some per-UTXO limit
on bandwidth relayed, but that task can be accomplished by simply
maintaining some kind of per-UTXO record of bandwidth used. For instance
if the weighted fee and fee/KB were recorded, and forced to - say -
double for each additional tx relayed that spent a given UTXO you would
have a clear and simple upper limit of lifetime bandwidth. Equally it's
easy to limit bandwidth moment to moment by asking peers for highest
fee/KB transactions they advertise first, stopping when our bandwidth
limit is reached.

You probably could even remove IsStandard() pretty much entirely with
the right increasingly expensive "replacement" policy, relying on it
alone to provide anti-DoS. Obviously this would simplify some of the
debates around mining policy! This could even be re-used for scalable a
general-purpose messaging network paid by coin ownership if the UTXO set
is split up, and some kind of expiration over time policy is
implemented.

Miners of course would still want to have a mempool, but that codebase
may prove simpler if it doesn't have to work double-duty for relaying as
well.

-- 
'peter'[:-1]@petertodd.org
00000000000000000b675c4d825a10c278b8d63ee4df90a19393f3b6498fd073

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 646 bytes --]

Re: [bitcoin-dev] Do we really need a mempool? (for relay nodes)

[Patrick Strateman]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

Relay nodes do not need a mempool, but do need some mechanism to avoid
DoS issues.

Wallet nodes can use the mempool for fee estimation (in addition to
looking at past blocks).

On 07/18/2015 11:52 AM, Peter Todd via bitcoin-dev wrote:
> As in, do relay nodes need to keep a record of the transactions they've
> relayed? Strictly speaking, the answer is no: one a tx is relayed modulo
> DoS concerns the entire thing can be discarded by the node. (unconfirmed
> txs spending other unconfirmed txs can be handled by creating packages
> of transactions, evaluated as a whole)
>
> To mitigate DoS concerns, we of course have to have some per-UTXO limit
> on bandwidth relayed, but that task can be accomplished by simply
> maintaining some kind of per-UTXO record of bandwidth used. For instance
> if the weighted fee and fee/KB were recorded, and forced to - say -
> double for each additional tx relayed that spent a given UTXO you would
> have a clear and simple upper limit of lifetime bandwidth. Equally it's
> easy to limit bandwidth moment to moment by asking peers for highest
> fee/KB transactions they advertise first, stopping when our bandwidth
> limit is reached.
>
> You probably could even remove IsStandard() pretty much entirely with
> the right increasingly expensive "replacement" policy, relying on it
> alone to provide anti-DoS. Obviously this would simplify some of the
> debates around mining policy! This could even be re-used for scalable a
> general-purpose messaging network paid by coin ownership if the UTXO set
> is split up, and some kind of expiration over time policy is
> implemented.
>
> Miners of course would still want to have a mempool, but that codebase
> may prove simpler if it doesn't have to work double-duty for relaying as
> well.
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[-- Attachment #2: Type: text/html, Size: 2588 bytes --]

Re: [bitcoin-dev] Do we really need a mempool? (for relay nodes)

[odinn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Some brief thoughts, adding here a suggestion for a dynamic approach
to the issue. (e.g. each additional tx relayed has some thing
associated with it, that is, a "doubling" for each additional tx
relayed that spends a given UTXO, doesn't sound like it would be the
most dynamic approach to the issue; considering that full nodes use
the (UTXOs) to establish if transactions are valid – all inputs to a
transaction must be in the UTXO database for it to be valid, but
rather, would end up ratcheting upward the fee/kB for each additional
tx relayed, as proposed).

A more dynamic fee approach would be a better one, imho, but how is
this to occur?

Quoting from Gavin Andresen's http://gavinandresen.ninja/utxo-uhoh,
"The entire UTXO set doesn’t have to be in RAM; it can be stored on an
SSD or spinning hard disk. The access pattern to the UTXO is not
random; outputs that were spent recently are more likely to be
re-spent than outputs that have not been spent in a long time. Bitcoin
Core already has a multi-level UTXO cache, thanks to the hard work of
Pieter Wuille."

The relay nodes would, in this scenario that is proposed here in this
message, be confirming and discarding; the wallet nodes, if I
understand properly, in this scenario, as proposed, should be
retaining (keeping a record of the transactions they've relayed and
using a mempool).

There are some questions here:

- - How is the mempool to be limited?
- - What is the mechanism by which the UTXO set is stored (or proposed
to be stored)?
- - How would dynamic fee determinations be calculated?
- - Please describe more the general purpose messaging network?

Thank you



On 07/18/2015 12:46 PM, Patrick Strateman via bitcoin-dev wrote:
> Relay nodes do not need a mempool, but do need some mechanism to
> avoid DoS issues.
> 
> Wallet nodes can use the mempool for fee estimation (in addition
> to looking at past blocks).
> 
> On 07/18/2015 11:52 AM, Peter Todd via bitcoin-dev wrote:
>> As in, do relay nodes need to keep a record of the transactions
>> they've relayed? Strictly speaking, the answer is no: one a tx is
>> relayed modulo DoS concerns the entire thing can be discarded by
>> the node. (unconfirmed txs spending other unconfirmed txs can be
>> handled by creating packages of transactions, evaluated as a
>> whole)
>> 
>> To mitigate DoS concerns, we of course have to have some per-UTXO
>> limit on bandwidth relayed, but that task can be accomplished by
>> simply maintaining some kind of per-UTXO record of bandwidth
>> used. For instance if the weighted fee and fee/KB were recorded,
>> and forced to - say - double for each additional tx relayed that
>> spent a given UTXO you would have a clear and simple upper limit
>> of lifetime bandwidth. Equally it's easy to limit bandwidth
>> moment to moment by asking peers for highest fee/KB transactions
>> they advertise first, stopping when our bandwidth limit is
>> reached.
>> 
>> You probably could even remove IsStandard() pretty much entirely
>> with the right increasingly expensive "replacement" policy,
>> relying on it alone to provide anti-DoS. Obviously this would
>> simplify some of the debates around mining policy! This could
>> even be re-used for scalable a general-purpose messaging network
>> paid by coin ownership if the UTXO set is split up, and some kind
>> of expiration over time policy is implemented.
>> 
>> Miners of course would still want to have a mempool, but that
>> codebase may prove simpler if it doesn't have to work double-duty
>> for relaying as well.
>> 
>> 
>> 
>> _______________________________________________ bitcoin-dev
>> mailing list bitcoin-dev@lists.linuxfoundation.org 
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> 
> 
> _______________________________________________ bitcoin-dev mailing
> list bitcoin-dev@lists.linuxfoundation.org 
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

- -- 
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVq2cFAAoJEGxwq/inSG8CIo4IAJAZ97NvW6Qdjd6HLN8q2074
sEUGdDeonARiQZXLfGyTJVg43Yb6LKPqkjWPQEgl9LLNni8t99iUqu3BJxedRDjd
8x+/F8n5VJrUrn1pXUcbC1aWss1y8JPTO2KpF/WL254IFc8iE8MJf4YF8PDSgy5j
9uW8NvWvdT4dz+rXu9vqfcplz8x7NGQ+CWN2N2JlChhKLMFprXPIx8a7NQwaKdrY
lTpgAJWGMyPGNCmYQprBjIjOfp8tdTLQFlsLUAsXDmEisJX9goRVGMsHOWLTREoA
l3kTgM0WMv6MIG7NOQQcWLD7cZdwWYR9e49hc27VcHt2R/FTepvnwPqo2GtE0tM=
=eRbR
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] Do we really need a mempool? (for relay nodes)

[Jorge Timón]

On Sat, Jul 18, 2015 at 9:46 PM, Patrick Strateman via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Relay nodes do not need a mempool, but do need some mechanism to avoid DoS
> issues.
>
> Wallet nodes can use the mempool for fee estimation (in addition to looking
> at past blocks).

Exactly, so an anti-DoS mechanism that would be sufficient for a
non-mempool node would be also useful for small values in -maxmempool.
I think a simple cache for transaction validations should be enough.
Please, review a draft for that in the newest #6448.

https://github.com/bitcoin/bitcoin/pull/6448

I would be happy to rebase it back to 0.11 and even 0.10.