[Bitcoin-development] deterministic transaction expiration

[Bitcoin-development] deterministic transaction expiration

[Kaz Wesley]

There is currently little in place for managing transaction lifetime
in the network's mempools (see discussion in github in #3722 "mempool
transaction expiration", and it seems to be a major factor blocking
some mempool exchange, see #1833/1918, #3721). Expiry per-node a
certain amount of wall time after receipt has been proposed, but
that's a fragile mechanism -- a single node could keep all relayable
transactions alive forever by remembering transactions until most
nodes have dropped them and then releasing them back into the wild.

I have a proposal for a way to add finite and predictable lifespans to
transactions in mempools: we d̶e̶s̶t̶r̶o̶y̶ ̶t̶h̶e̶
̶r̶e̶s̶u̶r̶r̶e̶c̶t̶i̶o̶n̶ ̶h̶u̶b̶ use nLockTime and a new standardness
rule. It could be done in stages, would not necessarily require even a
soft fork, and does not cause problems with reorgs like the proposal
in #3509:
1. start setting nLockTime to the current height by default in newly
created transactions (or slightly below the current height, for
reorg-friendliness)
2. once users have had some time to upgrade to clients that set
nLockTime, start discouraging transactions without nLockTime --
possibly with a slightly higher fee required for relay
3. start rate-limiting relay of transactions without an nLockTime
(maybe this alone could be used to achieve [2])
4. add a new IsStandard rule rejecting transactions with an nLockTime
more than N blocks behind the current tip (for some fixed value N, to
be determined)

Transactions would stop being relayed and drop out of mempools a fixed
number of blocks from their creation; once that window had passed, the
sender's wallet could begin to expect the transaction would not be
confirmed. In case a reorg displaces a transaction until after its
expiry height, a miner can still put it back in the blockchain; the
expiry height is just a relay rule. Also, a user who needed to get
their original "expired" transaction confirmed could still do so by
submitting it directly to a miner with suitable policies.

Re: [Bitcoin-development] deterministic transaction expiration

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

On Thu, Jul 31, 2014 at 05:58:23PM -0700, Kaz Wesley wrote:
> I have a proposal for a way to add finite and predictable lifespans to
> transactions in mempools: we d̶e̶s̶t̶r̶o̶y̶ ̶t̶h̶e̶
> ̶r̶e̶s̶u̶r̶r̶e̶c̶t̶i̶o̶n̶ ̶h̶u̶b̶ use nLockTime and a new standardness
> rule. It could be done in stages, would not necessarily require even a
> soft fork, and does not cause problems with reorgs like the proposal
> in #3509:

Anything that changes the semantics of nLockTime will do harm to
existing and future applications that make use of nLockTime for things
like refund transactions.

In any case, why do transactions need finite lifespans in mempools? If
you want to double-spend them with higher fees, then implement
replace-by-fee. In any case, lifetimes will never be deterministic as
not everyone runs the same software.

> Transactions would stop being relayed and drop out of mempools a fixed
> number of blocks from their creation; once that window had passed, the
> sender's wallet could begin to expect the transaction would not be
> confirmed. In case a reorg displaces a transaction until after its
> expiry height, a miner can still put it back in the blockchain; the
> expiry height is just a relay rule. Also, a user who needed to get
> their original "expired" transaction confirmed could still do so by
> submitting it directly to a miner with suitable policies.

...in which case someone will circumvent this IsStandard() rule by
submitting "expired" transactions directly to miners with suitable
policies.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Kaz Wesley]

On Thu, Jul 31, 2014 at 6:06 PM, Peter Todd <pete@petertodd.org> wrote:
> Anything that changes the semantics of nLockTime will do harm to
> existing and future applications that make use of nLockTime for things
> like refund transactions.

I think this would be compatible with most uses of nLockTime -- e.g.,
at the time a refund transaction becomes broadcastable, its
beneficiary would usually have no reason to wait for a long time
before broadcasting it; if they did so (probably because they weren't
online to redeem the refund), they'd need to use the
submit-directly-to-miner option, but they wouldn't lose their refund.

> In any case, why do transactions need finite lifespans in mempools? If
> you want to double-spend them with higher fees, then implement
> replace-by-fee.

Perpetuating transactions that have been in mempools for a long time
and are not being confirmed has been cited as a reason for nodes not
to exchange mempools (#3721, #1833, #3722); it's been implied that it
would be desirable for wallets to wait until a transaction had had a
chance to be accepted before double-spending with a higher fee
(#3722); and an unconfirmed transaction-age-based policy for
preventing mempool accumulation has been advocated (#3753, #3722) [I
hope my summarization is not misrepresenting anyone's opinions here;
please see the arguments made in the actual comments on the bugs].
These discussions are mostly fairly old, but I don't know of any
changes that have been made that provide alternative answers to these
concerns mentioned by at least three different developers.

> In any case, lifetimes will never be deterministic as not everyone runs
> the same software.

That's true, but none of the benefits of these changes require the
policy to be unanimous; most of the effect could be provided by most
of the network following these rules.

>> Transactions would stop being relayed and drop out of mempools a fixed
>> number of blocks from their creation; once that window had passed, the
>> sender's wallet could begin to expect the transaction would not be
>> confirmed. In case a reorg displaces a transaction until after its
>> expiry height, a miner can still put it back in the blockchain; the
>> expiry height is just a relay rule. Also, a user who needed to get
>> their original "expired" transaction confirmed could still do so by
>> submitting it directly to a miner with suitable policies.
>
> ...in which case someone will circumvent this IsStandard() rule by
> submitting "expired" transactions directly to miners with suitable
> policies.

Yes, that is a feature. None of the benefits of transaction expiration
rely on expiration being final, and any possible downsides of
expiration are largely mitigated by the option still being available
to get expired transactions mined.

Re: [Bitcoin-development] deterministic transaction expiration

[Matt Whitlock]

It would make more sense to introduce a new script opcode that pushes the current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes.


On Thursday, 31 July 2014, at 5:58 pm, Kaz Wesley wrote:
> There is currently little in place for managing transaction lifetime
> in the network's mempools (see discussion in github in #3722 "mempool
> transaction expiration", and it seems to be a major factor blocking
> some mempool exchange, see #1833/1918, #3721). Expiry per-node a
> certain amount of wall time after receipt has been proposed, but
> that's a fragile mechanism -- a single node could keep all relayable
> transactions alive forever by remembering transactions until most
> nodes have dropped them and then releasing them back into the wild.
> 
> I have a proposal for a way to add finite and predictable lifespans to
> transactions in mempools: we d̶e̶s̶t̶r̶o̶y̶ ̶t̶h̶e̶
> ̶r̶e̶s̶u̶r̶r̶e̶c̶t̶i̶o̶n̶ ̶h̶u̶b̶ use nLockTime and a new standardness
> rule. It could be done in stages, would not necessarily require even a
> soft fork, and does not cause problems with reorgs like the proposal
> in #3509:
> 1. start setting nLockTime to the current height by default in newly
> created transactions (or slightly below the current height, for
> reorg-friendliness)
> 2. once users have had some time to upgrade to clients that set
> nLockTime, start discouraging transactions without nLockTime --
> possibly with a slightly higher fee required for relay
> 3. start rate-limiting relay of transactions without an nLockTime
> (maybe this alone could be used to achieve [2])
> 4. add a new IsStandard rule rejecting transactions with an nLockTime
> more than N blocks behind the current tip (for some fixed value N, to
> be determined)
> 
> Transactions would stop being relayed and drop out of mempools a fixed
> number of blocks from their creation; once that window had passed, the
> sender's wallet could begin to expect the transaction would not be
> confirmed. In case a reorg displaces a transaction until after its
> expiry height, a miner can still put it back in the blockchain; the
> expiry height is just a relay rule. Also, a user who needed to get
> their original "expired" transaction confirmed could still do so by
> submitting it directly to a miner with suitable policies.

Re: [Bitcoin-development] deterministic transaction expiration

[Gregory Maxwell]

On Thu, Jul 31, 2014 at 6:38 PM, Matt Whitlock <bip@mattwhitlock.name> wrote:
> It would make more sense to introduce a new script opcode that pushes the current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes.

Transactions that become invalid later are have pretty severe
consequences because they might mean that completely in an absence of
fraud transactions are forever precluded due to a otherwise harmless
reorg.

While there may be uses for that, the resulting outputs should be
considered differently fungible— like coinbases which are immature—
and should probably be only used with great caution... not as a
mechanism for ordinary transactions.

Re: [Bitcoin-development] deterministic transaction expiration

[Matt Whitlock]

On Thursday, 31 July 2014, at 7:28 pm, Gregory Maxwell wrote:
> On Thu, Jul 31, 2014 at 6:38 PM, Matt Whitlock <bip@mattwhitlock.name> wrote:
> > It would make more sense to introduce a new script opcode that pushes the current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes.
> 
> Transactions that become invalid later are have pretty severe
> consequences because they might mean that completely in an absence of
> fraud transactions are forever precluded due to a otherwise harmless
> reorg.

I understand what you're saying, but I don't understand why it's a problem. Transactions shouldn't be considered "final" until a reasonable number of confirmations anyway, so the possibility that an "accepted" transaction could become invalid due to a chain reorganization is not a new danger. Ordinary transactions can similarly become invalid due to chain reorganizations, due to inputs already having been spent in the new branch.

Re: [Bitcoin-development] deterministic transaction expiration

[Gregory Maxwell]

On Thu, Jul 31, 2014 at 8:26 PM, Matt Whitlock <bip@mattwhitlock.name> wrote:
> I understand what you're saying, but I don't understand why it's a problem. Transactions shouldn't be considered "final" until a reasonable number of confirmations anyway, so the possibility that an "accepted" transaction could become invalid due to a chain reorganization is not a new danger. Ordinary transactions can similarly become invalid due to chain reorganizations, due to inputs already having been spent in the new branch.

A distinction there is that they can only become invalid via a
conflict— replaced by another transaction authored by the prior
signers. If no other transaction could be created (e.g. you're a
multisigner and won't sign it again) then there is no such risk.  It
now introduces chance events ("act of god") into the mix where they
they didn't exist before.  Basically it takes was what is a very loose
one way coupling and makes it much tighter. I'm sure if you spend a
bit thinking you can come up with some more corner cases that it would
expose— e.g. flooding the network with unrelated high fee transactions
in order to push a transaction out to where it can never be mined at
all.

Re: [Bitcoin-development] deterministic transaction expiration

[Tom Harding]

On 7/31/2014 5:58 PM, Kaz Wesley wrote:
> 1. start setting nLockTime to the current height by default in newly
> created transactions (or slightly below the current height, for
> reorg-friendliness)

Reorg-frendliness is the opposite of the rationale behind #2340, which 
proposes setting nLockTime at current-height + 1 to prevent 
"fee-sniping" reorgs...


> 2. once users have had some time to upgrade to clients that set
> nLockTime, start discouraging transactions without nLockTime --
> possibly with a slightly higher fee required for relay
> 3. start rate-limiting relay of transactions without an nLockTime
> (maybe this alone could be used to achieve [2])
> 4. add a new IsStandard rule rejecting transactions with an nLockTime
> more than N blocks behind the current tip (for some fixed value N, to
> be determined)
>

One way to proceed is implement #3753 (mempool janitor) in such a way 
that transactions with nLockTime are allowed to live a bit longer in the 
mempool (say 500 blocks) than those without (72 hours).  In other words, 
as a first step, just actually start expiring things from the mempool in 
bitcoin core, and leave any relay fee adjustments or rate limiting for 
later.  The isStandard change would be a good complement to #3753, to 
avoid relaying a tx that will soon expire by the nLockTime rule anyway.

Re: [Bitcoin-development] deterministic transaction expiration

[Flavien Charlon]

[-- Attachment #1: Type: text/plain, Size: 2962 bytes --]

> It would make more sense to introduce a new script opcode that pushes the
current block height onto the operand stack. Then you could implement
arbitrary logic about which blocks the transaction can be valid in. This
would require that the client revalidate all transactions in its mempool
(really, only those making use of this opcode) whenever the chain tip
changes.

I have to say I like this idea, this would allow someone to prove they
can't spend funds before a given date, and vice versa, prove that the funds
can't ever be spent after a given date (and this is provably prunable,
isn't it?). Of course, there are some risks associated with that, but
nobody is forced to use it.

> flooding the network with unrelated high fee transactions
> in order to push a transaction out to where it can never be mined at
> all.

This becomes increasingly expensive as the deadline is further away, so not
very hard to mitigate.


On Sat, Aug 2, 2014 at 1:36 AM, Tom Harding <tomh@thinlink.com> wrote:

> On 7/31/2014 5:58 PM, Kaz Wesley wrote:
> > 1. start setting nLockTime to the current height by default in newly
> > created transactions (or slightly below the current height, for
> > reorg-friendliness)
>
> Reorg-frendliness is the opposite of the rationale behind #2340, which
> proposes setting nLockTime at current-height + 1 to prevent
> "fee-sniping" reorgs...
>
>
> > 2. once users have had some time to upgrade to clients that set
> > nLockTime, start discouraging transactions without nLockTime --
> > possibly with a slightly higher fee required for relay
> > 3. start rate-limiting relay of transactions without an nLockTime
> > (maybe this alone could be used to achieve [2])
> > 4. add a new IsStandard rule rejecting transactions with an nLockTime
> > more than N blocks behind the current tip (for some fixed value N, to
> > be determined)
> >
>
> One way to proceed is implement #3753 (mempool janitor) in such a way
> that transactions with nLockTime are allowed to live a bit longer in the
> mempool (say 500 blocks) than those without (72 hours).  In other words,
> as a first step, just actually start expiring things from the mempool in
> bitcoin core, and leave any relay fee adjustments or rate limiting for
> later.  The isStandard change would be a good complement to #3753, to
> avoid relaying a tx that will soon expire by the nLockTime rule anyway.
>
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 3845 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 1402 bytes --]

Glad this was brought up.

Transaction expiration is something that I have wanted to see happen in
bitcoin for a long, long time.  The user experience of unconfirming
transactions setting around in limbo is just horrible.  Bitcoin software by
necessity has gotten better about attaching fees so this sort of behavior
is uncommon, but that does not eliminate the problem.

Of course, we cannot presume that a transaction will truly disappear -- The
Internet Never Forgets -- but given a bit of mempool adjusting, we can
achieve the next best thing:  the majority of the network "forgets" the
transaction and becomes willing to relay a respend of some or all of the
inputs.  This uses existing client logic where the client must rebroadcast
a transaction until it is confirmed.

In general, if a transaction has not made it into a block within 144*X
blocks, there is _some_ reason it is getting rejected by the miners.

The mempool janitor is a garbage collector design.  This is inferior to the
"superblock" model described at
https://github.com/bitcoin/bitcoin/issues/3723   Other models can also
achieve similar results.

There are a lot of issues tied together here:  transaction expiration, the
desire to cap the mempool ram usage, scalability, DoS prevention, ...
mempool ties a lot together.

-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 1762 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Alex Mizrahi]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

>
> A distinction there is that they can only become invalid via a
> conflict— replaced by another transaction authored by the prior
> signers. If no other transaction could be created (e.g. you're a
> multisigner and won't sign it again) then there is no such risk.


You need to check transaction's dependencies up to a certain depth to know
whether it is safe:
 If one of inputs depends on transaction which is signed by parties with
unknown trustworthiness, then it isn't safe.


>  It now introduces chance events ("act of god") into the mix where they
> they didn't exist before.


You need to check transaction's dependencies up to a certain depth to know
whether it is safe:
  If one of inputs depends on transaction time-locked script (or other
unrecognized script), then it isn't safe.

Situation is identical, you might need several extra lines of code.

I think it would matter only if we had deterministic, reliable mempool and
reorganization behavior. But it's not something we can depend on.

[-- Attachment #2: Type: text/html, Size: 1527 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

>
> The user experience of unconfirming transactions setting around in limbo
> is just horrible.  Bitcoin software by necessity has gotten better about
> attaching fees so this sort of behavior is uncommon, but that does not
> eliminate the problem.
>

Yes, indeed. I suspect there's a quick hack that could make this problem a
lot better though.

I think I brought up this idea before, but can't quite remember. Anyway I'm
willing to bet that if we analysed the data some more, we'd discover that
most "legitimate" i.e. non-DoS unconfirmed transactions that sit around for
ages are linked back to the block chain within two hops and not more. That
is people send a transaction that uses up their coin age, and then
immediately those coins are immediately respent again, but then those final
new coins are not spent.

On the other hand DoS attacks look like bouncing your coins around over and
over forever, i.e. more than two or three hops back to the chain.

So I wonder if making priority look back two or three transactions but not
more would help real users a lot, whilst not opening up any significant new
potential for DoS.

[-- Attachment #2: Type: text/html, Size: 1507 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

I feel like a lot of this will be driven by implementation, and costs of
changing the implementation.  Additional look-backs are of course doable,
but they incur some disk I/O costs.  The fields available in memory for
each mempool TX are

    uint256 tx_hash; // hash of next field
    CTransaction tx;
    int64_t nFee; // Cached to avoid expensive parent-transaction lookups
    size_t nTxSize; // ... and avoid recomputing tx size
    int64_t nTime; // Local time when entering the mempool
    double dPriority; // Priority when entering the mempool
    unsigned int nHeight; // Chain height when entering the mempool

As a first pass, we may prune the mempool without additional db lookups
quite easily based on time criteria.  Or, additional in-memory indexes may
be constructed to maintain hashes ordered by priority/fees.

Those techniques seem likely to be attempted before resorting to looking
back two or three TXs deep at coin age -- which I admit is an interesting
metric.

-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 1378 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Kaz Wesley]

> In general, if a transaction has not made it into a block within 144*X blocks, there is _some_ reason it is getting rejected by the miners.

This is the crux of my concern: relay policy and miner priorities will
not necessarily always be in sync, and node resource management
shouldn't rely on them being compatible. There are other solutions
than transaction expiration; I think Gavin's idea from the
block-squashing thread, in which miners explicitly provide information
about their policies, would go a long way to address this. But even
when mechanisms for reconciling nodes' expectations about miners'
behavior with miners' actual behavior are available, it may be
desirable to keep an expiry mechanism in place in case of glitches
between node understanding of policy and actual miner policy.

Any approach based on beginning a transaction expiry countdown when a
transaction is received (as in mempool janitor) seems unviable to me:
once a node has forgotten a transaction, it must be susceptible to
reaccepting it; all the functionality of such an expiry mechanism
depends on the network not containing any nodes with slightly
different relay behavior from bitcoind. I could accidentally
debilitate mempool janitors across the entire network if I set up two
nodes to exchange mempools whenever they reconnected to each other,
and restarted each frequently.

That's why I think including information in the transaction itself, as
with my nLockTime/IsStandard proposal, is necessary for transactions
to reliably eventually die off from mempools.
There's a modification I've been thinking about: allow a transaction's
lifetime to be refreshed (even after expiry) by a child transaction,
along the lines of child-pays-for-parent fee policy. This would
eliminate the need to reuse a key to make a replacement for an expired
transaction (when submitting the tx directly to a miner is not an
option), as well as alleviating the potential inconvenience in cases
like Peter brought up, where nLockTime is used for exchanged locked
transactions as part of a multi-transaction contract. With
child-refreshes-parent, a transaction's receivers and senders would
have the ability to keep trying to get their payment confirmed, but
anyone on the network can't just keep all transactions alive.


On Tue, Aug 5, 2014 at 10:48 AM, Jeff Garzik <jgarzik@bitpay.com> wrote:
> Glad this was brought up.
>
> Transaction expiration is something that I have wanted to see happen in
> bitcoin for a long, long time.  The user experience of unconfirming
> transactions setting around in limbo is just horrible.  Bitcoin software by
> necessity has gotten better about attaching fees so this sort of behavior is
> uncommon, but that does not eliminate the problem.
>
> Of course, we cannot presume that a transaction will truly disappear -- The
> Internet Never Forgets -- but given a bit of mempool adjusting, we can
> achieve the next best thing:  the majority of the network "forgets" the
> transaction and becomes willing to relay a respend of some or all of the
> inputs.  This uses existing client logic where the client must rebroadcast a
> transaction until it is confirmed.
>
> In general, if a transaction has not made it into a block within 144*X
> blocks, there is _some_ reason it is getting rejected by the miners.
>
> The mempool janitor is a garbage collector design.  This is inferior to the
> "superblock" model described at
> https://github.com/bitcoin/bitcoin/issues/3723   Other models can also
> achieve similar results.
>
> There are a lot of issues tied together here:  transaction expiration, the
> desire to cap the mempool ram usage, scalability, DoS prevention, ...
> mempool ties a lot together.
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

On Tue, Aug 5, 2014 at 3:10 PM, Kaz Wesley <keziahw@gmail.com> wrote:

> Any approach based on beginning a transaction expiry countdown when a
> transaction is received (as in mempool janitor) seems unviable to me:
>
...

> That's why I think including information in the transaction itself, as
> with my nLockTime/IsStandard proposal, is necessary for transactions
> to reliably eventually die off from mempools.
>

"reliably die off from mempools" leads into the land of "tightly
synchronizing memory pools across the network" which is a problem of...
large scope and much debate.  :)

For the moment, simply capping the mempool's size at each local node is a
much more reachable goal.  Capping, then, implies some culling policy.  In
general, bitcoind Tx mempool size is rather open ended, and that needs
sorting out.

-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 1621 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Tom Harding]

On 8/5/2014 12:10 PM, Kaz Wesley wrote:
> Any approach based on beginning a transaction expiry countdown when a 
> transaction is received (as in mempool janitor) seems unviable to me: 
> once a node has forgotten a transaction, it must be susceptible to 
> reaccepting it;

It's hard to argue with that logic.

If nLockTime is used for expiration, transaction creator can't lie to 
help tx live longer without pushing initial confirmation eligibility 
into the future.  Very pretty.  It would also enable "fill or kill" 
transactions with a backdated nLockTime, which must be confirmed in a 
few blocks, or start vanishing from mempools.

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

 ...and existing users and uses of nLockTime suddenly become worthless,
breaking payment channel refunds and other active uses of nLockTime.

You cannot assume the user is around to rewrite their nLockTime, if it
fails to be confirmed before some arbitrary deadline being set.



On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink.com> wrote:

> On 8/5/2014 12:10 PM, Kaz Wesley wrote:
> > Any approach based on beginning a transaction expiry countdown when a
> > transaction is received (as in mempool janitor) seems unviable to me:
> > once a node has forgotten a transaction, it must be susceptible to
> > reaccepting it;
>
> It's hard to argue with that logic.
>
> If nLockTime is used for expiration, transaction creator can't lie to
> help tx live longer without pushing initial confirmation eligibility
> into the future.  Very pretty.  It would also enable "fill or kill"
> transactions with a backdated nLockTime, which must be confirmed in a
> few blocks, or start vanishing from mempools.
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>



-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 2509 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 2460 bytes --]

We could however introduce a new field in a new tx version. We know we need
to rev the format at some point anyway.


On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay.com> wrote:

>  ...and existing users and uses of nLockTime suddenly become worthless,
> breaking payment channel refunds and other active uses of nLockTime.
>
> You cannot assume the user is around to rewrite their nLockTime, if it
> fails to be confirmed before some arbitrary deadline being set.
>
>
>
> On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink.com> wrote:
>
>> On 8/5/2014 12:10 PM, Kaz Wesley wrote:
>> > Any approach based on beginning a transaction expiry countdown when a
>> > transaction is received (as in mempool janitor) seems unviable to me:
>> > once a node has forgotten a transaction, it must be susceptible to
>> > reaccepting it;
>>
>> It's hard to argue with that logic.
>>
>> If nLockTime is used for expiration, transaction creator can't lie to
>> help tx live longer without pushing initial confirmation eligibility
>> into the future.  Very pretty.  It would also enable "fill or kill"
>> transactions with a backdated nLockTime, which must be confirmed in a
>> few blocks, or start vanishing from mempools.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Infragistics Professional
>> Build stunning WinForms apps today!
>> Reboot your WinForms applications with our WinForms controls.
>> Build a bridge from your legacy apps to the future.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 3887 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Tom Harding]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]


How is eventual expiration of a tx that started life with an nLockTime 
in the future "breaking", any more than any other tx expiring?


On 8/6/2014 6:54 AM, Mike Hearn wrote:
> We could however introduce a new field in a new tx version. We know we 
> need to rev the format at some point anyway.
>
>
> On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay.com 
> <mailto:jgarzik@bitpay.com>> wrote:
>
>      ...and existing users and uses of nLockTime suddenly become
>     worthless, breaking payment channel refunds and other active uses
>     of nLockTime.
>
>     You cannot assume the user is around to rewrite their nLockTime,
>     if it fails to be confirmed before some arbitrary deadline being set.
>
>
>
>     On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink.com
>     <mailto:tomh@thinlink.com>> wrote:
>
>         ...
>

>         If nLockTime is used for expiration, transaction creator can't
>         lie to
>         help tx live longer without pushing initial confirmation
>         eligibility
>         into the future.  Very pretty.  It would also enable "fill or
>         kill"
>         transactions with a backdated nLockTime, which must be
>         confirmed in a
>         few blocks, or start vanishing from mempools.
>


[-- Attachment #2: Type: text/html, Size: 3663 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

 ...because nLockTime is the exact opposite of expiration.  A locked TX
begins life invalid, and becomes valid (not-expired) after nLockTime passes.

A new field containing expiration time would work.



On Wed, Aug 6, 2014 at 10:44 AM, Tom Harding <tomh@thinlink.com> wrote:

>
> How is eventual expiration of a tx that started life with an nLockTime in
> the future "breaking", any more than any other tx expiring?
>
>
>
> On 8/6/2014 6:54 AM, Mike Hearn wrote:
>
> We could however introduce a new field in a new tx version. We know we
> need to rev the format at some point anyway.
>
>
> On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay.com> wrote:
>
>>  ...and existing users and uses of nLockTime suddenly become worthless,
>> breaking payment channel refunds and other active uses of nLockTime.
>>
>> You cannot assume the user is around to rewrite their nLockTime, if it
>> fails to be confirmed before some arbitrary deadline being set.
>>
>>
>>
>> On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink.com> wrote:
>>
>>> ...
>>>
>>
>      If nLockTime is used for expiration, transaction creator can't lie to
>>> help tx live longer without pushing initial confirmation eligibility
>>> into the future.  Very pretty.  It would also enable "fill or kill"
>>> transactions with a backdated nLockTime, which must be confirmed in a
>>> few blocks, or start vanishing from mempools.
>>>
>>
>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 4194 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Christian Decker]

[-- Attachment #1: Type: text/plain, Size: 2444 bytes --]

+1 for the new field, overloading fields with new meaning is definitely not
a good idea.

Something like nExpireAt with a block height sounds reasonable to me, but
we need to document that the usual caveats with blockchain reorgs apply.


On Wed, Aug 6, 2014 at 4:08 PM, Jeff Garzik <jgarzik@bitpay.com> wrote:

>  ...because nLockTime is the exact opposite of expiration.  A locked TX
> begins life invalid, and becomes valid (not-expired) after nLockTime passes.
>
> A new field containing expiration time would work.
>
>
>
> On Wed, Aug 6, 2014 at 10:44 AM, Tom Harding <tomh@thinlink.com> wrote:
>
>>
>> How is eventual expiration of a tx that started life with an nLockTime in
>> the future "breaking", any more than any other tx expiring?
>>
>>
>>
>> On 8/6/2014 6:54 AM, Mike Hearn wrote:
>>
>> We could however introduce a new field in a new tx version. We know we
>> need to rev the format at some point anyway.
>>
>>
>> On Wed, Aug 6, 2014 at 2:55 PM, Jeff Garzik <jgarzik@bitpay.com> wrote:
>>
>>>  ...and existing users and uses of nLockTime suddenly become worthless,
>>> breaking payment channel refunds and other active uses of nLockTime.
>>>
>>> You cannot assume the user is around to rewrite their nLockTime, if it
>>> fails to be confirmed before some arbitrary deadline being set.
>>>
>>>
>>>
>>> On Wed, Aug 6, 2014 at 12:01 AM, Tom Harding <tomh@thinlink.com> wrote:
>>>
>>>> ...
>>>>
>>>
>>      If nLockTime is used for expiration, transaction creator can't lie
>>>> to
>>>> help tx live longer without pushing initial confirmation eligibility
>>>> into the future.  Very pretty.  It would also enable "fill or kill"
>>>> transactions with a backdated nLockTime, which must be confirmed in a
>>>> few blocks, or start vanishing from mempools.
>>>>
>>>
>>
>
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 5705 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 6 August 2014 08:17:02 GMT-07:00, Christian Decker <decker.christian@gmail.com> wrote:
>+1 for the new field, overloading fields with new meaning is definitely
>not
>a good idea.

To add a new field the best way to do it is create a new, parallel, tx format where fields are committed by merkle radix tree in an extensible and provable way. You'd then commit to that tree with a mandatory OP_RETURN output in the last txout, or with a new merkle root.

Changing the tx format itself in a hard-fork is needlessly disruptive, and in this case, wastes opportunities for improvement.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCAA6BQJT4kzQMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhamzCAC+zRaXRodP63+ke3K+
Viapiepvk4uIOlqxqtMB2O0zWcyu2+xCJDiRPykK/6HLDBeFDEC9/dGK8++Lovl6
//qZ340LOPFlgT2kYy9E5h/yX469fhtsWhBCv2K47fWwkMS0S/0r4SQnCkbt2R2c
4dQjkoldhw6rNMBTUmwvhSlL30KsT/msWTZiX7DW/YjfOzezEJzy+mYyKp9Sk7ba
1fOiBXORk7mNOs7sTYTvje3sqEGpGTOLP08cY/RCEvl6bG8mHkPqwiojq+3biHFP
RsoBVu1f5cbnU7Wq0gPNdVnQssnEQDadyTX8gT0Wze7PuVyaZT2mXFZBKzSHuLy2
sJKN
=oPSo
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

A fork is not necessarily required, if you are talking about information
that deals primarily with pre-consensus mempool behavior.  You can make a
"network TX" with some information that is digitally signed, yet discarded
before it reaches miners.


On Wed, Aug 6, 2014 at 11:42 AM, Peter Todd <pete@petertodd.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
>
>
> On 6 August 2014 08:17:02 GMT-07:00, Christian Decker <
> decker.christian@gmail.com> wrote:
> >+1 for the new field, overloading fields with new meaning is definitely
> >not
> >a good idea.
>
> To add a new field the best way to do it is create a new, parallel, tx
> format where fields are committed by merkle radix tree in an extensible and
> provable way. You'd then commit to that tree with a mandatory OP_RETURN
> output in the last txout, or with a new merkle root.
>
> Changing the tx format itself in a hard-fork is needlessly disruptive, and
> in this case, wastes opportunities for improvement.
> -----BEGIN PGP SIGNATURE-----
> Version: APG v1.1.1
>
> iQFQBAEBCAA6BQJT4kzQMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
> cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhamzCAC+zRaXRodP63+ke3K+
> Viapiepvk4uIOlqxqtMB2O0zWcyu2+xCJDiRPykK/6HLDBeFDEC9/dGK8++Lovl6
> //qZ340LOPFlgT2kYy9E5h/yX469fhtsWhBCv2K47fWwkMS0S/0r4SQnCkbt2R2c
> 4dQjkoldhw6rNMBTUmwvhSlL30KsT/msWTZiX7DW/YjfOzezEJzy+mYyKp9Sk7ba
> 1fOiBXORk7mNOs7sTYTvje3sqEGpGTOLP08cY/RCEvl6bG8mHkPqwiojq+3biHFP
> RsoBVu1f5cbnU7Wq0gPNdVnQssnEQDadyTX8gT0Wze7PuVyaZT2mXFZBKzSHuLy2
> sJKN
> =oPSo
> -----END PGP SIGNATURE-----
>
>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 2235 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Mark Friedenbach]

On 08/06/2014 11:42 AM, Peter Todd wrote:
> On 6 August 2014 08:17:02 GMT-07:00, Christian Decker
> <decker.christian@gmail.com> wrote:
>> +1 for the new field, overloading fields with new meaning is
>> definitely not a good idea.
> 
> To add a new field the best way to do it is create a new, parallel,
> tx format where fields are committed by merkle radix tree in an
> extensible and provable way. You'd then commit to that tree with a
> mandatory OP_RETURN output in the last txout, or with a new merkle
> root.
> 
> Changing the tx format itself in a hard-fork is needlessly
> disruptive, and in this case, wastes opportunities for improvement.

I highly doubt that is the best approach.

If this nExpiry field is a consensus rule, then the Merkle tree or the
appropriate paths through needs to be included with the transaction as
part of the network and on-disk data structures, so that proper
validation can be done. This would be both more disruptive and less
efficient than simply adding an nExpiry field to the transaction format,
as we do in Freimarkets.

If the field is pre-consensus (a mempool gentleman's agreement), then it
has no business in the transaction structure at all and should be
packaged in some sort of envelope container.

Re: [Bitcoin-development] deterministic transaction expiration

[Tom Harding]

[-- Attachment #1: Type: text/plain, Size: 2814 bytes --]


Today we have first-eligible-height (nLockTime), and mempool expiration 
measured from this height would work for the goals being discussed, no 
fork or protocol rev.

With first-eligible-height and last-eligible-height, creator could 
choose a lifetime shorter than the max,  and in addition, lock the whole 
thing until some point in the future.


On 8/6/2014 9:15 AM, Jeff Garzik wrote:
> A fork is not necessarily required, if you are talking about 
> information that deals primarily with pre-consensus mempool behavior.  
> You can make a "network TX" with some information that is digitally 
> signed, yet discarded before it reaches miners.
>
>
> On Wed, Aug 6, 2014 at 11:42 AM, Peter Todd <pete@petertodd.org 
> <mailto:pete@petertodd.org>> wrote:
>
>     -----BEGIN PGP SIGNED MESSAGE-----
>     Hash: SHA256
>
>
>
>     On 6 August 2014 08:17:02 GMT-07:00, Christian Decker
>     <decker.christian@gmail.com <mailto:decker.christian@gmail.com>>
>     wrote:
>     >+1 for the new field, overloading fields with new meaning is
>     definitely
>     >not
>     >a good idea.
>
>     To add a new field the best way to do it is create a new,
>     parallel, tx format where fields are committed by merkle radix
>     tree in an extensible and provable way. You'd then commit to that
>     tree with a mandatory OP_RETURN output in the last txout, or with
>     a new merkle root.
>
>     Changing the tx format itself in a hard-fork is needlessly
>     disruptive, and in this case, wastes opportunities for improvement.
>     -----BEGIN PGP SIGNATURE-----
>     Version: APG v1.1.1
>
>     iQFQBAEBCAA6BQJT4kzQMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
>     cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhamzCAC+zRaXRodP63+ke3K+
>     Viapiepvk4uIOlqxqtMB2O0zWcyu2+xCJDiRPykK/6HLDBeFDEC9/dGK8++Lovl6
>     //qZ340LOPFlgT2kYy9E5h/yX469fhtsWhBCv2K47fWwkMS0S/0r4SQnCkbt2R2c
>     4dQjkoldhw6rNMBTUmwvhSlL30KsT/msWTZiX7DW/YjfOzezEJzy+mYyKp9Sk7ba
>     1fOiBXORk7mNOs7sTYTvje3sqEGpGTOLP08cY/RCEvl6bG8mHkPqwiojq+3biHFP
>     RsoBVu1f5cbnU7Wq0gPNdVnQssnEQDadyTX8gT0Wze7PuVyaZT2mXFZBKzSHuLy2
>     sJKN
>     =oPSo
>     -----END PGP SIGNATURE-----
>
>
>
>
> -- 
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc. https://bitpay.com/
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #2: Type: text/html, Size: 4766 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 6 August 2014 09:31:24 GMT-07:00, Mark Friedenbach <mark@monetize.io> wrote:
>I highly doubt that is the best approach.
>
>If this nExpiry field is a consensus rule, then the Merkle tree or the
>appropriate paths through needs to be included with the transaction as
>part of the network and on-disk data structures, so that proper
>validation can be done. This would be both more disruptive and less
>efficient than simply adding an nExpiry field to the transaction
>format,
>as we do in Freimarkets.

The general case doesn't require transmission of any merkle data; it is derived from the tx data. Equally changing a data format is certainly: note how Freimarkets has no third-party library support because you've made it incompatible with the standard Bitcoin data structures. Merkle radix tree formatting OTOH is just a cryptographically committed extension of the tag-value concept seen in protobuf, among others.

re: efficiency, we need fundamental improvements in efficiency, not little micro-optimisations everywhere done at high cost to maintainability.

re: validation, note how the merkle radix tree meets that need by allowing the absence of data to be proven.

>If the field is pre-consensus (a mempool gentleman's agreement), then
>it
>has no business in the transaction structure at all and should be
>packaged in some sort of envelope container.

It's also rather useless without consensus. Expiry is only useful if it is a guarantee, if not you might as well just implement tx replacement directly.

-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCAA6BQJT4mPZMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhcbKCACz/Qh3wm7ud9iwbvm3
Hzib36/fixk2++z6xlxh8G8afUaAe7ADCoz/TLK7RNIhUnr2hlsPO+Id2XvVBSm1
gXavj4iDxq8TpWsC8zPs5vyyY/dVwQ0RbidQFSpncmdW6EYVpIQp9nP3sSnBv1M8
c7BVidg708tc44uYiM9jeTzh6amP5yD0+G9FYYmy36BAQj8+4iD1ZCkiye1y5WUL
9MSN8LOxRFEwWQJmySXmJ1I9V81l1NSRQcQtfLVCzEIWLJXrZ0xwOq0SryEObg73
72NZKc2u8la3CPDoCG773ENbGHl+mGJW9M5ypV0s2RdkdZMgaFNnl/SBrWAcPd43
FkLr
=OMOy
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] deterministic transaction expiration

[Mark Friedenbach]

On 08/06/2014 01:02 PM, Tom Harding wrote:
> With first-eligible-height and last-eligible-height, creator could
> choose a lifetime shorter than the max,  and in addition, lock the whole
> thing until some point in the future.

Note that this would be a massive, *massive* change that would
completely break bitcoin output frangibility. Merchants would have to
start demanding input history back to a certain depth in order to ensure
they are not exposing themselves to undue reorg-expiry risk.

There are useful applications of a consensus-enforced expiry,
particularly within a private (signed block) side chain, and for that
reason it is useful to have a discussion about the merits of an nExpiry
field or BLOCK_HEIGHT / BLOCK_TIME opcode, and methods for achieving
either. However I don't see this ever becoming part of the public
bitcoin network.

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 3557 bytes --]

That won't necessarily work through large reorgs.  You don't want to create
a situation where a miner cannot mine a previously mined transactions.



On Wed, Aug 6, 2014 at 1:02 PM, Tom Harding <tomh@thinlink.com> wrote:

>
> Today we have first-eligible-height (nLockTime), and mempool expiration
> measured from this height would work for the goals being discussed, no fork
> or protocol rev.
>
> With first-eligible-height and last-eligible-height, creator could choose
> a lifetime shorter than the max,  and in addition, lock the whole thing
> until some point in the future.
>
>
>
> On 8/6/2014 9:15 AM, Jeff Garzik wrote:
>
> A fork is not necessarily required, if you are talking about information
> that deals primarily with pre-consensus mempool behavior.  You can make a
> "network TX" with some information that is digitally signed, yet discarded
> before it reaches miners.
>
>
> On Wed, Aug 6, 2014 at 11:42 AM, Peter Todd <pete@petertodd.org> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>>
>>
>> On 6 August 2014 08:17:02 GMT-07:00, Christian Decker <
>> decker.christian@gmail.com> wrote:
>> >+1 for the new field, overloading fields with new meaning is definitely
>> >not
>> >a good idea.
>>
>>  To add a new field the best way to do it is create a new, parallel, tx
>> format where fields are committed by merkle radix tree in an extensible and
>> provable way. You'd then commit to that tree with a mandatory OP_RETURN
>> output in the last txout, or with a new merkle root.
>>
>> Changing the tx format itself in a hard-fork is needlessly disruptive,
>> and in this case, wastes opportunities for improvement.
>> -----BEGIN PGP SIGNATURE-----
>> Version: APG v1.1.1
>>
>> iQFQBAEBCAA6BQJT4kzQMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
>> cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhamzCAC+zRaXRodP63+ke3K+
>> Viapiepvk4uIOlqxqtMB2O0zWcyu2+xCJDiRPykK/6HLDBeFDEC9/dGK8++Lovl6
>> //qZ340LOPFlgT2kYy9E5h/yX469fhtsWhBCv2K47fWwkMS0S/0r4SQnCkbt2R2c
>> 4dQjkoldhw6rNMBTUmwvhSlL30KsT/msWTZiX7DW/YjfOzezEJzy+mYyKp9Sk7ba
>> 1fOiBXORk7mNOs7sTYTvje3sqEGpGTOLP08cY/RCEvl6bG8mHkPqwiojq+3biHFP
>> RsoBVu1f5cbnU7Wq0gPNdVnQssnEQDadyTX8gT0Wze7PuVyaZT2mXFZBKzSHuLy2
>> sJKN
>> =oPSo
>> -----END PGP SIGNATURE-----
>>
>>
>
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
>
>
>
> _______________________________________________
> Bitcoin-development mailing listBitcoin-development@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 6061 bytes --]

Re: [Bitcoin-development] deterministic transaction expiration

[Mark Friedenbach]

On 08/06/2014 01:20 PM, Peter Todd wrote:
> The general case doesn't require transmission of any merkle data; it
> is derived from the tx data.

How can that possibly be the case? The information is hidden behind the
Merkle root in the transaction. The validator needs to know whether
there is an expiry and what it is. What's it supposed to do, guess?

Re: [Bitcoin-development] deterministic transaction expiration

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 6 August 2014 10:21:56 GMT-07:00, Mark Friedenbach <mark@monetize.io> wrote:
>On 08/06/2014 01:02 PM, Tom Harding wrote:
>> With first-eligible-height and last-eligible-height, creator could
>> choose a lifetime shorter than the max,  and in addition, lock the
>whole
>> thing until some point in the future.
>
>Note that this would be a massive, *massive* change that would
>completely break bitcoin output frangibility. Merchants would have to
>start demanding input history back to a certain depth in order to
>ensure
>they are not exposing themselves to undue reorg-expiry risk.

Bitcoin is already "broken" in that regard due to malleability, and more fundamentally, the existence of anyone-can-spend outputs, known private keys, SIGHASH_ANYONECANPAY, etc.

In any case, reorg-doublespend risk is no different than reorg-expiry risk.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCAA6BQJT4mcdMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhSdiB/9no/fXR50Zej4l6Hyt
gDvM9GWosGxZydQfplrUYzS9nLWTJgkjNYkrJk1OXPlkiNoHhlpCK6TuEL3DXBo8
txDBhp/xls7aFHELpPhP5iKrEj0J6fyMp9wKRVtUu0J+RhHY22v+iEQf//dGUX4v
hQPwATubmnyeVd71TAKyW6zCPjoEh0IG19tRVvw/v7/qNTXHdSZTkSVzQa4GP2gr
2xVqXTeOycPKqIU+GaNI4aRAL2DUm1kW3jG/+h3BwnJNd5q+0ELpC6xDmkA6hkNz
N6BFCtoghhKNH+FNsZKAzE9w8dYngZQbaA9vVdaR6SXzz9KuG526EymOF7e55IBJ
FMu+
=ii2+
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] deterministic transaction expiration

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 6 August 2014 10:30:11 GMT-07:00, Mark Friedenbach <mark@monetize.io> wrote:
>On 08/06/2014 01:20 PM, Peter Todd wrote:
>> The general case doesn't require transmission of any merkle data; it
>> is derived from the tx data.
>
>How can that possibly be the case? The information is hidden behind the
>Merkle root in the transaction. The validator needs to know whether
>there is an expiry and what it is. What's it supposed to do, guess?

The general case is all committed information is included in the transaction; the merkle tree is a compatibility path, as well as an optimisation for lite clients and applications.

You should read more about soft-forks; see the BIP. Remember that Bitcoin protocol development and deployment is not a centrally controlled activity.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCAA6BQJT4mgPMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhSYlCAC1ncAGQt53HKS+8/rq
OG0RGrqE2l2/qCM/ybd9M8TkwxaI3NB5bqfIus8dB5MnyiTBFS3ooN54kNNOHtSX
2rEzPJphtOj46tk3nqe1QO3cbFJPjBCtxZff51DWZckhCiO2Iy1Br3fK3v55iscp
1jxyZnpfgUG/Ivfx+h6vkisucBXgXJ82d5vzvMIMxixh4v2+4/SAcSY6HCLIpxmV
Z3l0NcGllnmWe5B6eftpWYUAREuoCNk/671jHmwu0cqk2u/Egrp776zxkEO1xivH
d0EWjJmlDLmQ2hEhkpBq46ji/2m4EWPLqTW/EXf3RzwU8uCEldbxEe2tyZ0d6oBt
NnTE
=AhV7
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] deterministic transaction expiration

[Tom Harding]

Having explored more drastic approaches, it looks like Kaz' basic idea 
stands well.  His #1...

> 1. start setting nLockTime to the current height by default in newly
> created transactions (or slightly below the current height, for
> reorg-friendliness)

is already implemented in bitcoin-qt #2340, and a "final call" on 
merging it was already sent to this list.  After some thought I agree 
with its policy of eventually setting nLockTime at current-height + 1 by 
default.  This is the "best reasonably expected height" of any tx 
created right now.  It discourages fee-sniping, and if a reorg happens 
anyway, it won't actually delay inclusion of tx beyond the reasonable 
expectation sans reorg.

However right now, #2340 takes a very cautious approach and sets to 
current-height - 10 by default, with randomness to mitigate worries 
about loss of privacy.

Kaz' #2, #3 and #4 are future actions.  #4 only goes most of the way ...

> 4. add a new IsStandard rule rejecting transactions with an nLockTime
> more than N blocks behind the current tip (for some fixed value N, to
> be determined)

... a janitor mechanism is desirable to purge mempool of txes more than 
N behind current-height.

Nodes dropping a tx N blocks after they became eligible to be mined (the 
meaning of nLockTime) makes sense.  It is not an overloading or new use 
for nLockTime, but a logical extension of it.  As Kaz pointed out, this 
solves a big problem with expiring by locally measured age: 
unintentional resurrection.

Re: [Bitcoin-development] deterministic transaction expiration

[Jeff Garzik]

On Fri, Aug 8, 2014 at 1:38 PM, Tom Harding <tomh@thinlink.com> wrote:
>> 4. add a new IsStandard rule rejecting transactions with an nLockTime
>> more than N blocks behind the current tip (for some fixed value N, to
>> be determined)

It cannot be assumed that transaction creation time and transaction
publish-to-outside-world time are the same, even though they often
are.

-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

Re: [Bitcoin-development] deterministic transaction expiration

[Kaz Wesley]

[-- Attachment #1: Type: text/plain, Size: 1363 bytes --]

A new network tx field would have the same problem, right?

With a child-refreshes-parent policy, someone wishing to redeem a
transaction that has passed its relay window without being confirmed could
still do so.
On Aug 8, 2014 11:16 AM, "Jeff Garzik" <jgarzik@bitpay.com> wrote:

> On Fri, Aug 8, 2014 at 1:38 PM, Tom Harding <tomh@thinlink.com> wrote:
> >> 4. add a new IsStandard rule rejecting transactions with an nLockTime
> >> more than N blocks behind the current tip (for some fixed value N, to
> >> be determined)
>
> It cannot be assumed that transaction creation time and transaction
> publish-to-outside-world time are the same, even though they often
> are.
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 2028 bytes --]