[Bitcoin-development] Build your own nHashType

[Bitcoin-development] Build your own nHashType

[Stephen Morse]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

Seeking feedback on a proposal that will allow a transaction signer to
explicitly specify what is to be serialized for the signature hash. The
basic idea is to make the nHashType general enough that we won't need a new
sighash flag every time a new use case comes up.

If implemented into bitcoin (via a soft fork), this would make malleability
almost a non-issue (the TXID referenced by inputs just need to be updated
previous TX changes) and would enable hardware wallets to securely sign
without needing to download/process each transaction it spends from.

Please let me know your thoughts.

https://github.com/scmorse/bitcoin-misc/blob/master/sighash_proposal.md

[-- Attachment #2: Type: text/html, Size: 860 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

Hi Stephen,

It's an interesting idea. I'm not sure that all the combinations make
sense. Excluding the connected output script or value but still signing the
prev tx hash appears pointless: the script cannot change anyway, and you
still need to know what it is to actually calculate the inputs to it, so
what is the point of this?

I also worry that quite a few of these combinations could be unexpectedly
dangerous. If you don't sign the prevout hash or value and combine it with
a regular pay-to-address output then you've effectively written a blank
cheque that can be used by anyone, to claim any money ever sent to that
address ... no? And then any p2p node or miner could do so, making the
transaction pretty useless.

That isn't inherently a problem as long as people understand which
combinations have what effects or cannot be used for various reasons. But
it would need good documentation and careful thought to explore each
possibility people might use.

I'll leave the soft fork business to one side for now. I think any change
in CHECKSIG or new version of it would likely be ready around the same time
as the hard fork we need for changing the block size limit anyway, and it's
much cleaner to do it that way.

The most important change that we need in sighash calculation, IMO, is
ensuring that you don't have to hash data over and over again without a
good reason. The current sighash definition is unfortunate because it's
possible to make small transactions that involve hashing huge amounts of
data. It's not clear to me that your proposal fixes that: ideally there
would be one exactly one sighash for one transaction no matter how many
checksigs are involved in verifying it.

thanks,
-mike

[-- Attachment #2: Type: text/html, Size: 1944 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Stephen Morse]

[-- Attachment #1: Type: text/plain, Size: 3953 bytes --]

Hi Mike,

Hi Stephen,
>
> It's an interesting idea. I'm not sure that all the combinations make
> sense. Excluding the connected output script or value but still signing the
> prev tx hash appears pointless: the script cannot change anyway, and you
> still need to know what it is to actually calculate the inputs to it, so
> what is the point of this?
>

That's a good point, maybe SIGHASH_WITHOUT_PREV_SCRIPTPUBKEY and
SIGHASH_WITHOUT_PREV_VALUE should be assumed false, since you need the data
anyway. That gets the total number of flags down to 17. If we eliminate
SIGHASH_WITHOUT_TX_VERSION (I can't think of any good reason for this one),
then we're down to a 2-byte nHashType. SIGHASH_SIGN_STACK_ELEMENT could
also be removed, I'm not convinced of the usefulness of that one either.


>
> I also worry that quite a few of these combinations could be unexpectedly
> dangerous. If you don't sign the prevout hash or value and combine it with
> a regular pay-to-address output then you've effectively written a blank
> cheque that can be used by anyone, to claim any money ever sent to that
> address ... no? And then any p2p
>
node or miner could do so, making the transaction pretty useless.
>
> That isn't inherently a problem as long as people understand which
> combinations have what effects or cannot be used for various reasons. But
> it would need good documentation and careful thought to explore each
> possibility people might use.
>

I don't think it's quite a blank check, but it would enable replay attacks
in the form of sending the money to the same place it was sent before if an
address ever receives coins again. Almost like auto-forwarding addresses.
If, in addition, you signed with just that input and no outputs as well,
then you're basically forfeiting your rights to any coins sent to that
address.

It allows for some dangerous combinations, but we already have some
dangerous nHashTypes. e.g. SIGHASH_NONE | SIGHASH_ANYONECANPAY. Good
documentation and careful developers shouldn't have any issues if they use
a standard set of sighash flag combinations for their standard use cases.
But developers that need special combinations can now use them, so long as
they are careful and think things through.


>
> I'll leave the soft fork business to one side for now. I think any change
> in CHECKSIG or new version of it would likely be ready around the same time
> as the hard fork we need for changing the block size limit anyway, and it's
> much cleaner to do it that way.
>
> The most important change that we need in sighash calculation, IMO, is
> ensuring that you don't have to hash data over and over again without a
> good reason. The current sighash definition is unfortunate because it's
> possible to make small transactions that involve hashing huge amounts of
> data. It's not clear to me that your proposal fixes that: ideally there
> would be one exactly one sighash for one transaction no matter how many
> checksigs are involved in verifying it.
>
>
It's hard, though, because there is different data needs to be signed for
each input. Although, I suppose if you signed your input with
SIGHASH_WITHOUT_PREV_SCRIPTPUBKEY, SIGHASH_WITHOUT_PREV_VALUE, and the
equivalent of SIGHASH_ALL, then the hash that needs to be signed would be
the same for all of your inputs. Strangely enough, I think we might have
just found use cases for the flags that we had nearly dismissed.

Another possibility would be to put the previous scriptPubKey and previous
output value at the END of the serialized transaction, so that you could
make use of some sort of a signature hash midstate. But that feels a little
messy. It sort of makes sense to have a base serialization for a
transaction and then append it with whatever input/output specific
information you have, but still, messy.

Is hashing transaction data once for each input really a huge bottleneck,
though? Do mobile devices have an issue with this?

Best,
Stephen

[-- Attachment #2: Type: text/html, Size: 5189 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

On Thu, Apr 9, 2015 at 7:10 AM, Stephen Morse <stephencalebmorse@gmail.com>
wrote:

> Is hashing transaction data once for each input really a huge bottleneck,
> though? Do mobile devices have an issue with this?
>


Think about what slow transaction verification speed means.  Slower
propagation across the network.  More work per node.  Greater opportunity
for algorithmic attacks, races and other shenanigans by attackers.

-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 1100 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Thu, Apr 09, 2015 at 07:22:52AM -0700, Jeff Garzik wrote:
> On Thu, Apr 9, 2015 at 7:10 AM, Stephen Morse <stephencalebmorse@gmail.com>
> wrote:
> 
> > Is hashing transaction data once for each input really a huge bottleneck,
> > though? Do mobile devices have an issue with this?
> >
> 
> 
> Think about what slow transaction verification speed means.  Slower
> propagation across the network.  More work per node.  Greater opportunity
> for algorithmic attacks, races and other shenanigans by attackers.

Keep in mind though we can always make part of the soft-fork be to make
the hash operations in the new CHECKSIG mechanism consume sigops.

For the OP: Have you looked at how CODESEPARATOR allows the signature to
sign code to run as part of verifying the signature? E.g. my signature
can say "valid if you run these additional opcodes and they return true"
where those additional opcodes take the transaction, hash it in the
defined way, and verify that the ECC signature correctly signs that
hash and the hash of the additional opcodes. For instance in this case
making a signature that's only valid if the tx fee is less than the
defined amount would be a matter of GET_FEE <max fee + 1> LESSTHAN VERIFY

This can be a much more general mechanism with easy to test modular
opcodes; for the consensus-critical codebase this can result in a much
easier and simpler to test CHECKSIG facility than a dozen new flags.

-- 
'peter'[:-1]@petertodd.org
000000000000000006975f442f50caa4fcc18e165746b3c77b641b75635afecb

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 650 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Stephen Morse]

[-- Attachment #1: Type: text/plain, Size: 3302 bytes --]

Regarding the re-hashing the transaction data once per input being a
bottleneck, I was mistakenly only thinking about this from the point of
view of the signer. Full nodes have to check all transactions' inputs,
which is much more costly, as the link Gavin posted shows.

On Thu, Apr 9, 2015 at 10:45 AM, Mike Hearn <mike@plan99.net> wrote:
>
> Right, good point. I wonder if this sort of auto forwarding could even be
> a useful feature. I can't think of one right now.
>

I can think of a few convoluted use cases, but not any good ones. People
have definitely looked for this feature before, though, just look at this
Bitcoin SE post
<http://bitcoin.stackexchange.com/questions/1495/is-there-a-way-to-automatically-send-bitcoins-from-one-wallet-to-another>.
I think there are better ways to handle key management than
auto-forwarding, though. Anyone looking for this feature probably just
wasn't aware that there are better solutions.

On Thu, Apr 9, 2015 at 10:45 AM, Mike Hearn <mike@plan99.net> wrote:
>
> Yes but is that fundamental or is there a way to avoid it? That's what I'm
> getting at.
>

In the bitcointalk article referenced, Sergio actually gave us the answer:

> Hash(Tx,previn-index) = Hash ( Hash(outputs) || Hash
(Inputs-with-script-cleared) || <previn-index> )
>   (for SIGHASH_ALL)
>   This way the values "Hash(outputs)" and
"Hash(Inputs-with-script-cleared)" can be cached and reused.

Basically, just re-order the way stuff is serialized. Put the stuff that is
nearly always signed at the beginning, and vice versa. I'll see if I can
update the proposal to make this optimization possible. What I suspect,
though, is that with all the new controls, blocks with ordinary
transactions will verify faster, but an attacker could still create a very
CPU intensive block by signing inputs with a wide variety of nHashTypes and
then signing the last one with the equivalent of SIGHASH_ALL. I don't think
that's a big limitation, though, the attack is already somewhat possible,
and would be very hard to do, and doesn't really gain the attacker anything
(other than infamy).

On Thu, Apr 9, 2015 at 1:28 PM, Peter Todd <pete@petertodd.org> wrote:

> For the OP: Have you looked at how CODESEPARATOR allows the signature to
> sign code to run as part of verifying the signature? E.g. my signature
> can say "valid if you run these additional opcodes and they return true"
> where those additional opcodes take the transaction, hash it in the
> defined way, and verify that the ECC signature correctly signs that
> hash and the hash of the additional opcodes. For instance in this case
> making a signature that's only valid if the tx fee is less than the
> defined amount would be a matter of GET_FEE <max fee + 1> LESSTHAN VERIFY
>

I've never been able to really see a good use case for OP_CODESEPARATOR,
and I'm not sure I completely have my head wrapped around what you're
proposing. From this
<http://bitcoin.stackexchange.com/questions/34013/what-is-op-codeseparator-used-for>
 and this
<https://bitcointalk.org/index.php?topic=52949.msg631255#msg631255>,
though, it seems like OP_CODESEPARATOR cannot really be made useful unless
you already have a way to sign without hashing the TXIDs referenced by your
input, in which case you need to modify the nHashType.

Best,
Stephen

[-- Attachment #2: Type: text/html, Size: 4707 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 4513 bytes --]

On Thu, Apr 09, 2015 at 10:56:20PM -0400, Stephen Morse wrote:
> On Thu, Apr 9, 2015 at 1:28 PM, Peter Todd <pete@petertodd.org> wrote:
> 
> > For the OP: Have you looked at how CODESEPARATOR allows the signature to
> > sign code to run as part of verifying the signature? E.g. my signature
> > can say "valid if you run these additional opcodes and they return true"
> > where those additional opcodes take the transaction, hash it in the
> > defined way, and verify that the ECC signature correctly signs that
> > hash and the hash of the additional opcodes. For instance in this case
> > making a signature that's only valid if the tx fee is less than the
> > defined amount would be a matter of GET_FEE <max fee + 1> LESSTHAN VERIFY
> >
> 
> I've never been able to really see a good use case for OP_CODESEPARATOR,
> and I'm not sure I completely have my head wrapped around what you're
> proposing. From this
> <http://bitcoin.stackexchange.com/questions/34013/what-is-op-codeseparator-used-for>
>  and this
> <https://bitcointalk.org/index.php?topic=52949.msg631255#msg631255>,
> though, it seems like OP_CODESEPARATOR cannot really be made useful unless
> you already have a way to sign without hashing the TXIDs referenced by your
> input, in which case you need to modify the nHashType.

I wrote up how to do this on #bitcoin-wizards, Dec 9th 2014:

17:13 < petertodd> hearn: even now you can use OP_CODESEPARATOR to
implement efficient payword schemes
17:14 < petertodd> hearn: early on you could have used it to do some
really useful after-the-fact signing delegation by wrapping a IF ENDIF
around the CODESEPARATOR introduced into the middle of the
scriptSig/scriptPubKey pair - shame we got rid of that without thinking
the design through
17:15 < petertodd> hearn: e.g. "create a signature that delegates
signing authority to another pubkey"
17:15 < petertodd> probably all 100% accidental... but a nice accident
17:16 < hearn> it's probably for the best. i can imagine such things
being a surprise for implementations not expecting them. a script 2.0
effort that incorporates lots of neat features but still lets script 1.0
work would be nice to have, one day
17:17 < petertodd> satoshi belived in 1 implementation, and by putting
CODESEPARATOR into the scriptSig/scriptPubKey concatenation you had to
opt-in to making that feature possible to use in any particular
scriptPubKey
17:17 < petertodd> w/o the mis-matched ENDIF you can't pull off that
trick because you can't turn CODESEPARATOR off
17:19 < petertodd> to be explicit: scriptPubKey: ENDIF <pubkey>
CHECKSIG, then the normal case is scriptSig: <signature> 1 IF
17:19 < petertodd> they concatenate to  <signature> 1 IF ENDIF <pubkey>
CHECKSIG, CODESEPARATOR is evaluated, and the signature is evaluated on
the script ENDIF <pubkey> CHECKSIG
17:20 < petertodd> to delegate signing authority after the fact sign a
signature on the script <pubkey2> 0 IF ENDIF <pubkey> CHECKSIG
17:21 < petertodd> (remember that CODESEPARATORS are removed by
SignatureHash())
17:22 < petertodd> oops, I mean: <pubkey2> CHECKSIGVERIFY 0 IF ENDIF
<pubkey> CHECKSIG
17:22 < petertodd> anyway, to finally spend it, create another signature
with pubkey2 signing the script <pubkey2> CHECKSIGVERIFY 0 IF ENDIF
<pubkey> CHECKSIG again, and finally spend it with the scriptSig:
<pubkey-sig> <pubkey2-sig> CODESEPARATOR <pubkey2> 0 IF
17:24 < petertodd> after concatenation the script: <pubkey-sig>
<pubkey2-sig> CODESEPARATOR <pubkey2> 0 IF CODESEPARATOR ENDIF <pubkey>
CHECKSIG is evaluated, the inner signature satisfies, and the outer
signature is satisfied only if the scriptPubKey was essentially changed
after the fact to also require the inner, second, pubkey2 to be
satisfied
17:26 < petertodd> a nice use-case would, forinstance, have been to have
a signing robot be able to create signatures offline for a given txout
with SIGHASH_SINGLE such that you had a spending limit enforced, and
exactly who was then allowed to spend the funds - say a department of a
company - could be picked after the fact without re-spending the txout
17:33 < petertodd> gmaxwell: re: script validation state, a good model
would be to have the tx input to EvalScript() essentially be a
CMerkleTx, and the prevout scriptPubKey be the prevout CTxOut (*maybe*
the prevout tx itself... bit dubious there...)

-- 
'peter'[:-1]@petertodd.org
00000000000000000e7980aab9c096c46e7f34c43a661c5cb2ea71525ebb8af7

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 650 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

>
> I don't think it's quite a blank check, but it would enable replay attacks
> in the form of sending the money to the same place it was sent before if an
> address ever receives coins again.
>

Right, good point. I wonder if this sort of auto forwarding could even be a
useful feature. I can't think of one right now.


> It's hard, though, because there is different data needs to be signed for
> each input.
>

Yes but is that fundamental or is there a way to avoid it? That's what I'm
getting at.


> Another possibility would be to put the previous scriptPubKey and previous
> output value at the END of the serialized transaction, so that you could
> make use of some sort of a signature hash midstate.
>

Interesting idea! I don't agree it's messy. If anything it should be
simpler than what we have today - the need to edit a transaction *in the
middle* means that sighash computation involves constantly reserializing a
transaction before it even gets to be hashed.


> Is hashing transaction data once for each input really a huge bottleneck,
> though? Do mobile devices have an issue with this?
>

Consider what happens with very large transactions, like a big assurance
contract that might have thousands of inputs and be multiple megabytes in
size. Obviously such large transactions cannot happen today, but there is
user demand for giant contracts (or at least, users tell me there is,
whether they'd actually do it for real is a bit unclear).

[-- Attachment #2: Type: text/html, Size: 2521 bytes --]

Re: [Bitcoin-development] Build your own nHashType

[Gregory Maxwell]

On Wed, Apr 8, 2015 at 7:50 PM, Stephen Morse
<stephencalebmorse@gmail.com> wrote:
> Seeking feedback on a proposal that will allow a transaction signer to
> explicitly specify what is to be serialized for the signature hash. The
> basic idea is to make the nHashType general enough that we won't need a new
> sighash flag every time a new use case comes up.
>
> If implemented into bitcoin (via a soft fork), this would make malleability
> almost a non-issue (the TXID referenced by inputs just need to be updated
> previous TX changes) and would enable hardware wallets to securely sign
> without needing to download/process each transaction it spends from.

I'm not sure if I'm super fond of that particular non-programmatic but
many options approach; It sort of has the problem that there are
relatively few useful options that don't rapidly extend into a choose
your own adventure with too many options to count; so you take a
complexity penalty perhaps without a matching functionality payoff.

but thats not why I'm commenting...

I wonder if anyone noticed that any sighash masking that eliminates
the txin txid enables covenants?

Covenants are payments which constrain their future payments (like
deed covenants), I've written about them in the past
https://bitcointalk.org/index.php?topic=278122.0  ... they can
sometimes be pretty useful but are also potentially a irritating hit
to fungibility, at least if used stupidly.

the approach here is that you make the scriptpubkey contain "[push:
0x30, 0x06, 0x02, 0x01, 0x04, 0x02, 0x01, 0x04, flags] [push pubkey
resulting from pubkey recovery] OP_CHECKSIG"  and set the flags to
match only the things you want to enforce in the spending transaction
hash them up and recover the EC public point.   You can think of that
construct as giving a you a OP_MASKED_TRANSACTION_HASH_EQUALS  ... the
recovered pubkey is just a kind of message hash, though a weird and
expensive to compute one.

I don't currently see how to get a perpetual covenant out of it-- e.g.
a coin that anyone can spend, but only to its same scriptpubkey, (the
obvious way requires the ability to be able to checksig stuff on the
stack) though I wouldn't be shocked if it were possible with a
sufficiently complex sighash flag and nothing else.