[Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Nathan Wilcox]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

[I'm currently wading through bitcoin-development. I'm still about a month
behind, so I apologize in advance for any noisy redundancy in this post.]

While reading about blocksize, I've just finished Mike Hearn's blog post
describing expected systemic behavior as actual blocks approach the current
limit (with or without non-protocol-changing implementation improvements):

https://medium.com/@octskyward/crash-landing-f5cc19908e32


One detail Mike uses to argue against the "fee's will save us" line of
reasoning is that wallets have no good way to learn fee information.

So, here's a proposal to fix that: put fee and (and perhaps block size,
UTXO, etc...) statistics into the locally-verifiable data available to SPV
clients (ie: block headers).


It's easy to imagine a hard fork that places details like per-block total
fees, transaction count, fee variance, UTXO delta, etc... in a each block
header. This would allow SPV clients to rely on this data with the same
PoW-backed assurances as all other header data.

This mechanism seems valuable regardless of the outcome of blocksize
debate. So long as fees are interesting or important, SPV clients should
know about them. (Same for other stats such as UTXO count.)

Upgrading the protocol without a hard-fork may be possible and is left as
an exercise for the reader. ;-)

-- 
Nathan Wilcox
Least Authoritarian

email: nathan@leastauthority.com
twitter: @least_nathan
PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993

[-- Attachment #2: Type: text/html, Size: 1898 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Aaron Voisine]

[-- Attachment #1: Type: text/plain, Size: 2821 bytes --]

It could be done by agreeing on a data format and encoding it in an
op_return output in the coinbase transaction. If it catches on it could
later be enforced with a soft fork.

For real up-to-the-minute fee calculations you're also going to want to
look at the current mempool, how many transactions are waiting, what fees
they're paying, etc, but of course that information is susceptible to sybil
attack.

In practice what we're doing for now is using services like blockcypher
who's business is improving reliability of zero-conf to tell us what
fee-per-kb is needed, and then putting a hard coded range around it to
protect against the service being compromised. This is also the kind of
thing being done for exchange rate data which is probably the bigger
security risk until bitcoin becomes the standard unit of account for the
planet.

Aaron Voisine
co-founder and CEO
breadwallet.com

On Wed, Jun 10, 2015 at 10:37 AM, Nathan Wilcox <nathan@leastauthority.com>
wrote:

> [I'm currently wading through bitcoin-development. I'm still about a month
> behind, so I apologize in advance for any noisy redundancy in this post.]
>
> While reading about blocksize, I've just finished Mike Hearn's blog post
> describing expected systemic behavior as actual blocks approach the current
> limit (with or without non-protocol-changing implementation improvements):
>
> https://medium.com/@octskyward/crash-landing-f5cc19908e32
>
>
> One detail Mike uses to argue against the "fee's will save us" line of
> reasoning is that wallets have no good way to learn fee information.
>
> So, here's a proposal to fix that: put fee and (and perhaps block size,
> UTXO, etc...) statistics into the locally-verifiable data available to SPV
> clients (ie: block headers).
>
>
> It's easy to imagine a hard fork that places details like per-block total
> fees, transaction count, fee variance, UTXO delta, etc... in a each block
> header. This would allow SPV clients to rely on this data with the same
> PoW-backed assurances as all other header data.
>
> This mechanism seems valuable regardless of the outcome of blocksize
> debate. So long as fees are interesting or important, SPV clients should
> know about them. (Same for other stats such as UTXO count.)
>
> Upgrading the protocol without a hard-fork may be possible and is left as
> an exercise for the reader. ;-)
>
> --
> Nathan Wilcox
> Least Authoritarian
>
> email: nathan@leastauthority.com
> twitter: @least_nathan
> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 3987 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Nathan Wilcox]

[-- Attachment #1: Type: text/plain, Size: 4310 bytes --]

On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail.com> wrote:

> It could be done by agreeing on a data format and encoding it in an
> op_return output in the coinbase transaction. If it catches on it could
> later be enforced with a soft fork.
>
>
Sounds plausible, except SPV protocols would need to include this coinbase
txn if it's going to help SPV clients. (Until a softfork is activated, SPV
clients should not rely on this encoding, since until that time the results
can be fabricated by individual miners.)


> For real up-to-the-minute fee calculations you're also going to want to
> look at the current mempool, how many transactions are waiting, what fees
> they're paying, etc, but of course that information is susceptible to sybil
> attack.
>

Hm, when you mention Sybil attack, I don't quite follow.

When a client relies on any report of a mempool [*], this is already
outside the realm of locally-verifiable SPV information, so they are
already susceptible to the service making false claims. If that's
acceptable (and in many cases it may be) then this whole mechanism is moot,
because the client can ask the service for fee statistics for past blocks.


> In practice what we're doing for now is using services like blockcypher
> who's business is improving reliability of zero-conf to tell us what
> fee-per-kb is needed, and then putting a hard coded range around it to
> protect against the service being compromised.
>

This is interesting for me, because I had previously believed fees were
fairly static presently, and also because I like hearing about real life
wallet implementations.

So if this "SPV Fee Stats" feature were added, a wallet might rely on an
API for timely stats (aka "block height < 1") then verify that the API
isn't lying after doing SPV verification of fee stats for confirmed blocks.


This is also the kind of thing being done for exchange rate data which is
> probably the bigger security risk until bitcoin becomes the standard unit
> of account for the planet.
>
>
That makes sense, although there's no SPV equivalent for exchange data.


Aaron Voisine
> co-founder and CEO
> breadwallet.com
>
> On Wed, Jun 10, 2015 at 10:37 AM, Nathan Wilcox <nathan@leastauthority.com
> > wrote:
>
>> [I'm currently wading through bitcoin-development. I'm still about a
>> month behind, so I apologize in advance for any noisy redundancy in this
>> post.]
>>
>> While reading about blocksize, I've just finished Mike Hearn's blog post
>> describing expected systemic behavior as actual blocks approach the current
>> limit (with or without non-protocol-changing implementation improvements):
>>
>> https://medium.com/@octskyward/crash-landing-f5cc19908e32
>>
>>
>> One detail Mike uses to argue against the "fee's will save us" line of
>> reasoning is that wallets have no good way to learn fee information.
>>
>> So, here's a proposal to fix that: put fee and (and perhaps block size,
>> UTXO, etc...) statistics into the locally-verifiable data available to SPV
>> clients (ie: block headers).
>>
>>
>> It's easy to imagine a hard fork that places details like per-block total
>> fees, transaction count, fee variance, UTXO delta, etc... in a each block
>> header. This would allow SPV clients to rely on this data with the same
>> PoW-backed assurances as all other header data.
>>
>> This mechanism seems valuable regardless of the outcome of blocksize
>> debate. So long as fees are interesting or important, SPV clients should
>> know about them. (Same for other stats such as UTXO count.)
>>
>> Upgrading the protocol without a hard-fork may be possible and is left as
>> an exercise for the reader. ;-)
>>
>> --
>> Nathan Wilcox
>> Least Authoritarian
>>
>> email: nathan@leastauthority.com
>> twitter: @least_nathan
>> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>>
>


-- 
Nathan Wilcox
Least Authoritarian

email: nathan@leastauthority.com
twitter: @least_nathan
PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993

[-- Attachment #2: Type: text/html, Size: 6748 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote:
> On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail.com> wrote:
> 
> > It could be done by agreeing on a data format and encoding it in an
> > op_return output in the coinbase transaction. If it catches on it could
> > later be enforced with a soft fork.
> >
> >
> Sounds plausible, except SPV protocols would need to include this coinbase
> txn if it's going to help SPV clients. (Until a softfork is activated, SPV
> clients should not rely on this encoding, since until that time the results
> can be fabricated by individual miners.)

Fee stats can always be fabricated by individual miners because fees can
be paid out-of-band.

-- 
'peter'[:-1]@petertodd.org
00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 650 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Nathan Wilcox]

[-- Attachment #1: Type: text/plain, Size: 1859 bytes --]

On Wed, Jun 10, 2015 at 2:03 PM, Peter Todd <pete@petertodd.org> wrote:

> On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote:
> > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail.com>
> wrote:
> >
> > > It could be done by agreeing on a data format and encoding it in an
> > > op_return output in the coinbase transaction. If it catches on it could
> > > later be enforced with a soft fork.
> > >
> > >
> > Sounds plausible, except SPV protocols would need to include this
> coinbase
> > txn if it's going to help SPV clients. (Until a softfork is activated,
> SPV
> > clients should not rely on this encoding, since until that time the
> results
> > can be fabricated by individual miners.)
>
> Fee stats can always be fabricated by individual miners because fees can
> be paid out-of-band.
>
>
This is a point I hadn't considered carefully before. I don't understand
the marketplace here or why miners would want to move fees outside of
explicit inband fees. Implicit in this proposal is that the statistics only
cover in-band data, because that's the scope of consensus rules, and thus
the proposal is only as useful as the information of in-band fees is useful.

I've also noticed a detracting technical argument given a particular
tradeoff:

A Header-PoW-verifying client could still be given all transactions in a
recent block, from which it can see the in-band fees directly.  The
trade-off is the size of those transactions versus the need to alter any
consensus rules or do soft forks.

Notice how this trade-off's costs change with maximum block size.




> --
> 'peter'[:-1]@petertodd.org
> 00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f
>



-- 
Nathan Wilcox
Least Authoritarian

email: nathan@leastauthority.com
twitter: @least_nathan
PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993

[-- Attachment #2: Type: text/html, Size: 2839 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Aaron Voisine]

[-- Attachment #1: Type: text/plain, Size: 2553 bytes --]

> A Header-PoW-verifying client could still be given all transactions in a
recent block, from which it can see the in-band fees directly.

You don't know the fees paid by any given transaction unless you also have
all it's inputs. Transaction inputs do not include an amount. You could
however get the average fee-per-kb paid by all transactions in a block by
looking at the coinbase transaction, subtracting the block reward, and
dividing by the size of block minus the header.


Aaron Voisine
co-founder and CEO
breadwallet.com

On Thu, Jun 11, 2015 at 11:30 AM, Nathan Wilcox <nathan@leastauthority.com>
wrote:

> On Wed, Jun 10, 2015 at 2:03 PM, Peter Todd <pete@petertodd.org> wrote:
>
>> On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote:
>> > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail.com>
>> wrote:
>> >
>> > > It could be done by agreeing on a data format and encoding it in an
>> > > op_return output in the coinbase transaction. If it catches on it
>> could
>> > > later be enforced with a soft fork.
>> > >
>> > >
>> > Sounds plausible, except SPV protocols would need to include this
>> coinbase
>> > txn if it's going to help SPV clients. (Until a softfork is activated,
>> SPV
>> > clients should not rely on this encoding, since until that time the
>> results
>> > can be fabricated by individual miners.)
>>
>> Fee stats can always be fabricated by individual miners because fees can
>> be paid out-of-band.
>>
>>
> This is a point I hadn't considered carefully before. I don't understand
> the marketplace here or why miners would want to move fees outside of
> explicit inband fees. Implicit in this proposal is that the statistics only
> cover in-band data, because that's the scope of consensus rules, and thus
> the proposal is only as useful as the information of in-band fees is useful.
>
> I've also noticed a detracting technical argument given a particular
> tradeoff:
>
> A Header-PoW-verifying client could still be given all transactions in a
> recent block, from which it can see the in-band fees directly.  The
> trade-off is the size of those transactions versus the need to alter any
> consensus rules or do soft forks.
>
> Notice how this trade-off's costs change with maximum block size.
>
>
>
>
>> --
>> 'peter'[:-1]@petertodd.org
>> 00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f
>>
>
>
>
> --
> Nathan Wilcox
> Least Authoritarian
>
> email: nathan@leastauthority.com
> twitter: @least_nathan
> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993
>

[-- Attachment #2: Type: text/html, Size: 4100 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Nathan Wilcox]

[-- Attachment #1: Type: text/plain, Size: 3119 bytes --]

On Thu, Jun 11, 2015 at 12:55 PM, Aaron Voisine <voisine@gmail.com> wrote:

> > A Header-PoW-verifying client could still be given all transactions in
> a recent block, from which it can see the in-band fees directly.
>
> You don't know the fees paid by any given transaction unless you also have
> all it's inputs. Transaction inputs do not include an amount. You could
> however get the average fee-per-kb paid by all transactions in a block by
> looking at the coinbase transaction, subtracting the block reward, and
> dividing by the size of block minus the header.
>
>
Excellent point and alternative proposal. You're right: to get the specifi
fees, you'd need all transactions in a block, and all TxOuts with
membership proofs. Your alternative seems like a much leaner trade-off for
similar data.


>
> Aaron Voisine
> co-founder and CEO
> breadwallet.com
>
> On Thu, Jun 11, 2015 at 11:30 AM, Nathan Wilcox <nathan@leastauthority.com
> > wrote:
>
>> On Wed, Jun 10, 2015 at 2:03 PM, Peter Todd <pete@petertodd.org> wrote:
>>
>>> On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote:
>>> > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail.com>
>>> wrote:
>>> >
>>> > > It could be done by agreeing on a data format and encoding it in an
>>> > > op_return output in the coinbase transaction. If it catches on it
>>> could
>>> > > later be enforced with a soft fork.
>>> > >
>>> > >
>>> > Sounds plausible, except SPV protocols would need to include this
>>> coinbase
>>> > txn if it's going to help SPV clients. (Until a softfork is activated,
>>> SPV
>>> > clients should not rely on this encoding, since until that time the
>>> results
>>> > can be fabricated by individual miners.)
>>>
>>> Fee stats can always be fabricated by individual miners because fees can
>>> be paid out-of-band.
>>>
>>>
>> This is a point I hadn't considered carefully before. I don't understand
>> the marketplace here or why miners would want to move fees outside of
>> explicit inband fees. Implicit in this proposal is that the statistics only
>> cover in-band data, because that's the scope of consensus rules, and thus
>> the proposal is only as useful as the information of in-band fees is useful.
>>
>> I've also noticed a detracting technical argument given a particular
>> tradeoff:
>>
>> A Header-PoW-verifying client could still be given all transactions in a
>> recent block, from which it can see the in-band fees directly.  The
>> trade-off is the size of those transactions versus the need to alter any
>> consensus rules or do soft forks.
>>
>> Notice how this trade-off's costs change with maximum block size.
>>
>>
>>
>>
>>> --
>>> 'peter'[:-1]@petertodd.org
>>> 00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f
>>>
>>
>>
>>
>> --
>> Nathan Wilcox
>> Least Authoritarian
>>
>> email: nathan@leastauthority.com
>> twitter: @least_nathan
>> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993
>>
>
>


-- 
Nathan Wilcox
Least Authoritarian

email: nathan@leastauthority.com
twitter: @least_nathan
PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993

[-- Attachment #2: Type: text/html, Size: 5236 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Aaron Voisine]

[-- Attachment #1: Type: text/plain, Size: 5306 bytes --]

> Sounds plausible, except SPV protocols would need to include this
coinbase txn if it's going to help SPV clients.

Yes you'd either need a way to add those transactions to the bloom filter,
or add/modify a p2p message to request it specifically.

> when you mention Sybil attack, I don't quite follow.

I just mean that someone could spin up a bunch of malicious p2p nodes that
lied about mempool data. It's a bit worse for SPV clients since they can't
verify that unconfirmed transactions are valid.

> I had previously believed fees were fairly static presently,

I actually just added it the other day after getting blockcypher to include
it in their api. The current release is still using a hard coded fee rate.

Aaron Voisine
co-founder and CEO
breadwallet.com

On Wed, Jun 10, 2015 at 1:00 PM, Nathan Wilcox <nathan@leastauthority.com>
wrote:

> On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail.com> wrote:
>
>> It could be done by agreeing on a data format and encoding it in an
>> op_return output in the coinbase transaction. If it catches on it could
>> later be enforced with a soft fork.
>>
>>
> Sounds plausible, except SPV protocols would need to include this coinbase
> txn if it's going to help SPV clients. (Until a softfork is activated, SPV
> clients should not rely on this encoding, since until that time the results
> can be fabricated by individual miners.)
>
>
>> For real up-to-the-minute fee calculations you're also going to want to
>> look at the current mempool, how many transactions are waiting, what fees
>> they're paying, etc, but of course that information is susceptible to sybil
>> attack.
>>
>
> Hm, when you mention Sybil attack, I don't quite follow.
>
> When a client relies on any report of a mempool [*], this is already
> outside the realm of locally-verifiable SPV information, so they are
> already susceptible to the service making false claims. If that's
> acceptable (and in many cases it may be) then this whole mechanism is moot,
> because the client can ask the service for fee statistics for past blocks.
>
>
>> In practice what we're doing for now is using services like blockcypher
>> who's business is improving reliability of zero-conf to tell us what
>> fee-per-kb is needed, and then putting a hard coded range around it to
>> protect against the service being compromised.
>>
>
> This is interesting for me, because I had previously believed fees were
> fairly static presently, and also because I like hearing about real life
> wallet implementations.
>
> So if this "SPV Fee Stats" feature were added, a wallet might rely on an
> API for timely stats (aka "block height < 1") then verify that the API
> isn't lying after doing SPV verification of fee stats for confirmed blocks.
>
>
> This is also the kind of thing being done for exchange rate data which is
>> probably the bigger security risk until bitcoin becomes the standard unit
>> of account for the planet.
>>
>>
> That makes sense, although there's no SPV equivalent for exchange data.
>
>
> Aaron Voisine
>> co-founder and CEO
>> breadwallet.com
>>
>> On Wed, Jun 10, 2015 at 10:37 AM, Nathan Wilcox <
>> nathan@leastauthority.com> wrote:
>>
>>> [I'm currently wading through bitcoin-development. I'm still about a
>>> month behind, so I apologize in advance for any noisy redundancy in this
>>> post.]
>>>
>>> While reading about blocksize, I've just finished Mike Hearn's blog post
>>> describing expected systemic behavior as actual blocks approach the current
>>> limit (with or without non-protocol-changing implementation improvements):
>>>
>>> https://medium.com/@octskyward/crash-landing-f5cc19908e32
>>>
>>>
>>> One detail Mike uses to argue against the "fee's will save us" line of
>>> reasoning is that wallets have no good way to learn fee information.
>>>
>>> So, here's a proposal to fix that: put fee and (and perhaps block size,
>>> UTXO, etc...) statistics into the locally-verifiable data available to SPV
>>> clients (ie: block headers).
>>>
>>>
>>> It's easy to imagine a hard fork that places details like per-block
>>> total fees, transaction count, fee variance, UTXO delta, etc... in a each
>>> block header. This would allow SPV clients to rely on this data with the
>>> same PoW-backed assurances as all other header data.
>>>
>>> This mechanism seems valuable regardless of the outcome of blocksize
>>> debate. So long as fees are interesting or important, SPV clients should
>>> know about them. (Same for other stats such as UTXO count.)
>>>
>>> Upgrading the protocol without a hard-fork may be possible and is left
>>> as an exercise for the reader. ;-)
>>>
>>> --
>>> Nathan Wilcox
>>> Least Authoritarian
>>>
>>> email: nathan@leastauthority.com
>>> twitter: @least_nathan
>>> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Bitcoin-development mailing list
>>> Bitcoin-development@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>
>>>
>>
>
>
> --
> Nathan Wilcox
> Least Authoritarian
>
> email: nathan@leastauthority.com
> twitter: @least_nathan
> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993
>

[-- Attachment #2: Type: text/html, Size: 8651 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

I described an alternative way for SPV wallets to learn about fees some
time ago. It requires a new transaction version that embeds output values
into the signed data. Then an upgrade to the P2P protocol to send UTXO data
along with transactions when they are relayed.

The idea is that the wallet sets a Bloom filter with an FP rate that
ensures it will see some random subset of all transactions being broadcast
on the network, and with the extra data, it can calculate the fee paid.
Once a transaction broadcast is observed the wallet includes that tx hash
in its next Bloom filter, thus it can see which block the tx confirmed in.
By measuring the amount of time that passed between a broadcast and it
appearing in a block, it can calculate its own tables of fee paid:time
taken.

This has the advantage that you don't have to trust miners to publish data
accurately. However it requires some protocol upgrades and of course, a lot
of new code in SPV wallets.

The way Bitcoin Wallet for Android handles fees currently is to just update
a hard coded value every so often.

[-- Attachment #2: Type: text/html, Size: 1174 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Aaron Voisine]

[-- Attachment #1: Type: text/plain, Size: 1932 bytes --]

The other complication is that this will tend to be a lagging indicator
based on network congestion from the last time you connected. If we assume
that transactions are being dropped in an unpredictable way when blocks are
full, knowing the network congestion *right now* is critical, and even then
you just have to hope that someone who wants that space more than you do
doesn't show up after you disconnect.


Aaron Voisine
co-founder and CEO
breadwallet.com

On Wed, Jun 10, 2015 at 1:26 PM, Mike Hearn <mike@plan99.net> wrote:

> I described an alternative way for SPV wallets to learn about fees some
> time ago. It requires a new transaction version that embeds output values
> into the signed data. Then an upgrade to the P2P protocol to send UTXO data
> along with transactions when they are relayed.
>
> The idea is that the wallet sets a Bloom filter with an FP rate that
> ensures it will see some random subset of all transactions being broadcast
> on the network, and with the extra data, it can calculate the fee paid.
> Once a transaction broadcast is observed the wallet includes that tx hash
> in its next Bloom filter, thus it can see which block the tx confirmed in.
> By measuring the amount of time that passed between a broadcast and it
> appearing in a block, it can calculate its own tables of fee paid:time
> taken.
>
> This has the advantage that you don't have to trust miners to publish data
> accurately. However it requires some protocol upgrades and of course, a lot
> of new code in SPV wallets.
>
> The way Bitcoin Wallet for Android handles fees currently is to just
> update a hard coded value every so often.
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 2657 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

>
> If we assume that transactions are being dropped in an unpredictable way
> when blocks are full, knowing the network congestion *right now* is
> critical, and even then you just have to hope that someone who wants that
> space more than you do doesn't show up after you disconnect.
>

Yeah, my proposal is not intended to function correctly with full blocks,
as Bitcoin cannot work at all in such a state. It assumes that fees only
change slowly and that transactions are being cleared normally.

[-- Attachment #2: Type: text/html, Size: 736 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

On Wed, Jun 10, 2015 at 02:18:30PM -0700, Aaron Voisine wrote:
> The other complication is that this will tend to be a lagging indicator
> based on network congestion from the last time you connected. If we assume
> that transactions are being dropped in an unpredictable way when blocks are
> full, knowing the network congestion *right now* is critical, and even then
> you just have to hope that someone who wants that space more than you do
> doesn't show up after you disconnect.

Hence the need for ways to increase fees on transactions after initial
broadcast like replace-by-fee and child-pays-for-parent.

Re: "dropped in an unpredictable way" - transactions would be dropped
lowest fee/KB first, a completely predictable way.

-- 
'peter'[:-1]@petertodd.org
0000000000000000124bae79afdcee9267b4e6f8137758b8b4135455cd8e3bfd

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 650 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Martin Lie]

Peter Todd wrote:
> Re: "dropped in an unpredictable way" - transactions would be dropped
> lowest fee/KB first, a completely predictable way.

It would be 'completely predictable' for whoever knew the state and 
policies of a miner's mempool, but from an end user's perspective that 
wouldn't matter much: The end users wouldn't know if their 
transaction(s) would make the cut or not, somewhere in the network, and 
by what time. They (obviously) won't know what miners will find the next 
block(s), they won't know the miners' mempool sizes, potential custom 
eviction policies, etc.

I agree that this can be somewhat remedied by FSSRBF/CPFP, though, 
provided wallets give users a good (semi-automated?) interface for such 
transaction replacements/chains.


-- 
Martin Lie

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Tom Harding]

On 6/11/2015 6:10 AM, Peter Todd wrote:
> On Wed, Jun 10, 2015 at 02:18:30PM -0700, Aaron Voisine wrote:
>> The other complication is that this will tend to be a lagging indicator
>> based on network congestion from the last time you connected. If we assume
>> that transactions are being dropped in an unpredictable way when blocks are
>> full, knowing the network congestion *right now* is critical, and even then
>> you just have to hope that someone who wants that space more than you do
>> doesn't show up after you disconnect.
> Hence the need for ways to increase fees on transactions after initial
> broadcast like replace-by-fee and child-pays-for-parent.
>
> Re: "dropped in an unpredictable way" - transactions would be dropped
> lowest fee/KB first, a completely predictable way.

Quite agreed.  Also, transactions with unconfirmed inputs should be 
among the first to get dropped, as discussed in the "Dropped-transaction 
spam" thread.  Like all policy rules, either of these works in 
proportion to its deployment.

Be advised that pull request #6068 emphasizes the view that the network 
will never have consistent mempool/relay policies, and on the contrary 
needs a framework that supports and encourages pluggable, generally 
parameterized policies that could (some might say should) conflict 
wildly with each other.

It probably doesn't matter that much.  Deploying a new policy still 
wouldn't be much easier than deploying a patched version.  I mean, 
nobody has proposed a policy rule engine yet (oops).

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 315 bytes --]

>
> > Re: "dropped in an unpredictable way" - transactions would be dropped
> > lowest fee/KB first, a completely predictable way.
>
> Quite agreed.


No, Aaron is correct. It's unpredictable from the perspective of the user
sending the transaction, and as they are the ones picking the fees, that is
what matters.

[-- Attachment #2: Type: text/html, Size: 588 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Aaron Voisine]

[-- Attachment #1: Type: text/plain, Size: 1209 bytes --]

One possible solution that wallets could adopt when blocks fill up, would
be to abandon the p2p network for transaction propagation altogether, and
instead work directly with a handful of the largest miners and pools to get
transactions into blocks. The miners could auction off space in their
blocks with the guarantee that they will be included in the order accepted.
We'd lose the peer-to-peer nature of sending transactions for a federated
service operator style model, but avoid the problem of unpredictable
transaction failure. Also, unlike replace-by-fee, this would allow for
a send-and-forget usage pattern, as well as known up-front fees. Something
like this is certainly what I imagine the large hosted wallet companies
will end up moving to.

Aaron

On Thursday, June 11, 2015, Mike Hearn <mike@plan99.net> wrote:

> > Re: "dropped in an unpredictable way" - transactions would be dropped
>> > lowest fee/KB first, a completely predictable way.
>>
>> Quite agreed.
>
>
> No, Aaron is correct. It's unpredictable from the perspective of the user
> sending the transaction, and as they are the ones picking the fees, that is
> what matters.
>


-- 

Aaron Voisine
co-founder and CEO
breadwallet.com

[-- Attachment #2: Type: text/html, Size: 1834 bytes --]

Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism

[Nathan Wilcox]

[-- Attachment #1: Type: text/plain, Size: 2812 bytes --]

On Thu, Jun 11, 2015 at 7:10 AM, Peter Todd <pete@petertodd.org> wrote:

> On Wed, Jun 10, 2015 at 02:18:30PM -0700, Aaron Voisine wrote:
> > The other complication is that this will tend to be a lagging indicator
> > based on network congestion from the last time you connected. If we
> assume
> > that transactions are being dropped in an unpredictable way when blocks
> are
> > full, knowing the network congestion *right now* is critical, and even
> then
> > you just have to hope that someone who wants that space more than you do
> > doesn't show up after you disconnect.
>
> Hence the need for ways to increase fees on transactions after initial
> broadcast like replace-by-fee and child-pays-for-parent.
>
>
I haven't looked closely at replace-by-fee yet, but I assume this is a
non-consensus change to mempool mechanics. To me, this seems like the
"actuator" side of fee mechanics: it provides a transaction sender a way to
influence the system.  By contrast, learning about fees is the "sensor"
side of fee mechanics.

Consider how a replace-by-fee wallet makes fee decisions. When does it
replace by fee?  It needs feedback in one of two forms:

a. Direct feedback from a trnasaction relay service, or:

b. Information in the blockchain, which is verified by all verifying nodes
and refined by all miners.

The first kind of information is quite acceptable and practical for many
use cases, but leave the wallet vulnerable to fabrications by that service.
This vulnerability is precisely what SPV security intends to mitigate,
right?

With only information type a, a (non-SPV) wallet can "shop around" to find
competing services, and this should work pretty well, provided the wallet
can discover those competing services. If, OTOH, the wallet has access to
information type b, it now has "perfect competition" across all such
services, even when it can't discover the low-priced services directly.

This actual-fees-in-the-actual-block-chain information of type b seem like
a powerful source of pricing information, and if SPV security is already
valuable for other reasons, then it seems natural to leverage that value.


Re: "dropped in an unpredictable way" - transactions would be dropped
> lowest fee/KB first, a completely predictable way.
>
> --
> 'peter'[:-1]@petertodd.org
> 0000000000000000124bae79afdcee9267b4e6f8137758b8b4135455cd8e3bfd
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>


-- 
Nathan Wilcox
Least Authoritarian

email: nathan@leastauthority.com
twitter: @least_nathan
PGP: 11169993 / AAAC 5675 E3F7 514C 67ED  E9C9 3BFE 5263 1116 9993

[-- Attachment #2: Type: text/html, Size: 3956 bytes --]