[bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

As the title suggests, I would like to formally request the assignment of a 
BIP number for my FT spec.

Thank you!


Source; 

https://github.com/zander/bips/blob/FlexTrans/bip-9999.mediawiki

<pre>
  BIP: ??
  Title: Flexible Transactions
  Author: Tom Zander <tomz@freedommail.ch>
  Status: Draft
  Type: Standards Track
  Created: 2016-07-27
</pre>

==Abstract==

This BIP describes the next step in making Bitcoin's most basic element,
the transaction, more flexible and easier to extend. At the same time this
fixes all known cases of malleability and resolves significant amounts of
technical debt.

==Summary==

Flexible Transactions uses the fact that the first 4 bytes in a transaction
determine the version and that the majority of the clients use a
non-consensus rule (a policy) to not accept transaction version numbers
other than those specifically defined by Bitcoin.
This BIP chooses a new version number, 4, and defines that the data
following the bytes for the version is in a format called Compact Message
Format (CMF). CMF is a flexible, token based format where each token is a
combination of a name, a format and a value. Because the name is added we
can skip unused tokens and we can freely add new tokens in a simple manner
in future. Soft fork upgrades will become much easier and cleaner this
way.

This protocol upgrade cleans up past soft fork changes like BIP68 which
reuse existing fields and do them in a much better to maintain and easier
to parse system. It creates the building blocks to allow new features to be
added much cleaner in the future.

It also shows to be possible to remove signatures from transactions with
minimal upgrades of software and still maintain a coherent transaction
history. Tests show that this can reduce space usage to about 75%.

==Motivation==

Token based file-formats are not new, systems like XML and HTMl use a
similar system to allow future growth and they have been quite successful
for decades in part because of this property.

Bitcoin needs a similar way of making the transaction future-proof because
re-purposing not used fields for new features is not good for creating
maintainable code.

Next to that this protocol upgrade will re-order the data-fields which
allows us to cleanly fix the malleability issue which means that future
technologies like Lightning Network will depend on this BIP being deployed.

At the same time, due to this re-ordering of data fields, it becomes very
easy to remove signatures from a transaction without breaking its tx-id,
which is great for future pruning features.


=== Tokens ===

In the compact message format we define tokens and in this specification we
define how these tokens are named, where they can be placed and which are
optional.  To refer to XML, this specification would be the schema of
a transaction.

CMF tokens are triplets of name, format (like PositiveInteger) and value.
Names in this scope are defined much like an enumeration where the actual
integer value (id, below) is equally important to the written name.
If any token found that is not covered in the next table will make the
transaction that contains it invalid.

{| class="wikitable"
|-
! Name !! id !! Format !! Default Value !! Description
|-
|TxEnd         ||  0 ||BoolTrue ||  Required    ||A marker that is the last 
byte in the txid calculation
|-
|TxInPrevHash  ||  1 ||ByteArray||  Required    ||TxId we are spending
|-
|TxPrevIndex   ||  2 ||Integer  ||      0       ||Index in prev tx we are 
spending (applied to previous TxInPrevHash)
|-
|TxInScript    ||  3 ||ByteArray||  Required    ||The 'input' part of the 
script
|-
|TxOutValue    ||  4 ||Integer  ||  Required    ||Amount of satoshi to 
transfer
|-
|TxOutScript   ||  5 ||ByteArray||  Required    ||The 'output' part of the 
script
|-
|LockByBlock   ||  6 ||Integer  ||  Optional    ||BIP68 replacement
|-
|LockByTime    ||  7 ||Integer  ||  Optional    ||BIP68 replacement
|-
|ScriptVersion ||  8 ||Integer  ||      2       ||Defines script version for 
outputs following
|-
|NOP_1x        || 1x || . ||  Optional    ||Values that will be ignored by 
anyone parsing the transaction
|}


=== Scripting changes ===

In the current version of Bitcoin-script, version 1, there are various
opcodes that are used to validate the cryptographic proofs that users have
to provide in order to spend outputs.

The OP_CHECKSIG is the most well known and, as its name implies, it
validates a signature.
In the new version of 'script' (version 2) the data that is signed is
changed to be equivalent to the transaction-id. This is a massive
simplification and also the only change between version 1 and version 2 of
script.

=== Serialization order===

The tokens defined above have to be serialized in a certain order for the
transaction to be well-formatted.  Not serializing transactions in the
order specified would allow multiple interpretations of the data which
can't be allowed.
There is still some flexibility and for that reason it is important for
implementors to remember that the actual serialized data is used for the
calculation of the transaction-id. Reading and writing it may give you a
different output and when the txid changes, the signatures will break.

At a macro-level the transaction has these segments. The order of the
segments can not be changed, but you can skip segments.

{| class="wikitable"
!Segment !! Description
|-
|   Inputs   || Details about inputs.
|-
|  Outputs   || Details and scripts for outputs
|-
| Additional || For future expansion
|-
| Signatures || The scripts for the inputs
|-
|   TxEnd    || End of the transaction
|}

The TxId is calculated by taking the serialized transaction without the
Signatures and the TxEnd and hashing that.


{| class="wikitable"
!Segment !! Tags !! Description
|-
|Inputs||TxInPrevHash and TxInPrevIndex||Index can be skipped, but in any 
input the PrevHash always has to come first
|-
|Outputs||TxOutScript, TxOutValue||Order is not relevant
|-
|Additional||LockByBlock  LockByTime NOP_1x
|-
|Signatures||TxInScript||Exactly the same amount as there are inputs
|-
|TxEnd||TxEnd
|}

TxEnd is there to allow a parser to know when one transaction in a stream
has ended, allowing the next to be parsed.

Notice that the token ScriptVersion is currently not allowed because we
don't have any valid value to give it. But if we introduce a new script
version it would be placed in the outputs segment.

=== Script v2 ===

The default value of ScriptVersion is number 2, as opposed to the version 1
of script that the is in use today.  The version 2 is mostly identical
to version one, including upgrades made to it over the years and in the 
future. The only exception is that the OP_CHECKSIG is made dramatically
simpler.  The input-type for OP_CHECKSIG is now no longer configurable, it is
always '1' and the content that will be signed is the txid.

TODO: does check-multisig need its own mention?


=== Block-malleability ===

The effect of leaving the signatures out of the calculation of the
transaction-id implies that the signatures are also not used for the
calculation of the merkle tree.  This means that changes in signatures
would not be detectable. Except naturally by the fact that missing or
broken signatures breaks full validation. But it is important to detect
modifications to such signatures outside of validating all transactions.

For this reason the merkle tree is extended to include (append) the hash of
the v4 transactions (and those alone) where the hash is taken over a
data-blob that is build up from:

1. the tx-id
2. the CMF-tokens 'TxInScript'


=== Future extensibility ===

The NOP_1x wildcard used in the table explaining tokens is actually a list
of 10 values that currently are specified as NOP (no-operation) tags.

Any implementation that supports the v4 transaction format should ignore
this field in a transaction. Interpreting and using the transaction as if
that field was not present at all.

Future software may use these fields to decorate a transaction with
additional data or features. Transaction generating software should not
trivially use these tokens for their own usage without cooperation and
communication with the rest of the Bitcoin ecosystem as miners certainly
have the option to reject transactions that use unknown-to-them tokens.


==Reference Implementation==

Bitcoin Classic includes this in its beta releases and a reference
implementation can be found at;

https://github.com/bitcoinclassic/bitcoinclassic/pull/186


==Deployment==

To be determined

==References==

[https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md] 
CMF

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Luke Dashjr]

On Tuesday, September 20, 2016 5:15:45 PM Tom via bitcoin-dev wrote:
> As the title suggests, I would like to formally request the assignment of a
> BIP number for my FT spec.

Please open a pull request on the bitcoin/bips repo after this has been 
discussed a bit on the ML.

Note that at least a basic backwards compatibility section is required for 
assignment, especially since this appears to be a hard-fork proposal (if it is 
meant as a soft-fork, please explain how that would work).

> ==Motivation==
> 
> Token based file-formats are not new, systems like XML and HTMl use a
> similar system to allow future growth and they have been quite successful
> for decades in part because of this property.

There is already a binary-safe format called EBML. Why not use that as the 
basis for this BIP?

> Next to that this protocol upgrade will re-order the data-fields which
> allows us to cleanly fix the malleability issue which means that future
> technologies like Lightning Network will depend on this BIP being deployed.

Note this won't fix malleability entirely, only third-party malleability (much 
like segwit). The sender can always modify the transaction with more 
inputs/outputs.

> |TxEnd         ||  0 ||BoolTrue ||  Required    ||A marker that is the last
> byte in the txid calculation

The last byte in the txid calculation, or the last byte of the entire 
transaction?

It seems from the later comments, that it is the end of the transaction as a 
whole. Yet a separator between the txid and non-txid data would probably be 
valuable, rather than hard-coding txid to skip signature types (which may be 
unknown to old nodes, when extended).

> The OP_CHECKSIG is the most well known and, as its name implies, it
> validates a signature.
> In the new version of 'script' (version 2) the data that is signed is
> changed to be equivalent to the transaction-id. This is a massive
> simplification and also the only change between version 1 and version 2 of
> script.

This seems to be a major regression. What is the replacement for 
SIGHASH_SINGLE and SIGHASH_ANYONECANPAY?

When revising OP_CHECKSIG, it would also be nice to add the ability to use 
*only* a hash of the prevout's scriptPubKey in the input, so that *when* the 
prevtx is malleated, the spending one remains valid. (This use case is 
currently not supported.)

> === Serialization order===
> 
> The tokens defined above have to be serialized in a certain order for the
> transaction to be well-formatted.  Not serializing transactions in the
> order specified would allow multiple interpretations of the data which
> can't be allowed.

What happens if nodes encounter a different order in a block?

> Notice that the token ScriptVersion is currently not allowed because we
> don't have any valid value to give it. But if we introduce a new script
> version it would be placed in the outputs segment.

What happens if I put ScriptVersion=1 here?

> === Script v2 ===
> 
> TODO: does check-multisig need its own mention?

Does multisig still require a dummy item on the stack?

> === Block-malleability ===
> 
> For this reason the merkle tree is extended to include (append) the hash of
> the v4 transactions (and those alone) where the hash is taken over a
> data-blob that is build up from:

How should nodes know where in the merkle-tree the txids end, and the v4hashes 
begin?

Luke

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On Tue, Sep 20, 2016 at 07:15:45PM +0200, Tom via bitcoin-dev wrote:
> === Serialization order===
> 
> The tokens defined above have to be serialized in a certain order for the
> transaction to be well-formatted.  Not serializing transactions in the
> order specified would allow multiple interpretations of the data which
> can't be allowed.

If the order of the tokens is fixed, the tokens themselves are redundant
information when tokens are required; when tokens may be omitted, a simple
"Some/None" flag to mark whether or not the optional data has been omitted is
appropriate.


Also, if you're going to break compatibility with all existing software, it
makes sense to use a format that extends the merkle tree down into the
transaction inputs and outputs.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Tuesday 20 Sep 2016 21:31:47 Luke Dashjr wrote:
> On Tuesday, September 20, 2016 5:15:45 PM Tom via bitcoin-dev wrote:
> > As the title suggests, I would like to formally request the assignment of
> > a
> > BIP number for my FT spec.
> 
> Please open a pull request on the bitcoin/bips repo after this has been
> discussed a bit on the ML.

> It seems from the later comments, that it is the end of the transaction as a
> whole. Yet a separator between the txid and non-txid data would probably be
> valuable, rather than hard-coding txid to skip signature types (which may
> be unknown to old nodes, when extended).
> 
> > The OP_CHECKSIG is the most well known and, as its name implies, it
> > validates a signature.
> > In the new version of 'script' (version 2) the data that is signed is
> > changed to be equivalent to the transaction-id. This is a massive
> > simplification and also the only change between version 1 and version 2 of
> > script.
> 
> This seems to be a major regression. What is the replacement for
> SIGHASH_SINGLE and SIGHASH_ANYONECANPAY?

How is this a regression? Can you explain what functionality is lost please?

> When revising OP_CHECKSIG, it would also be nice to add the ability to use
> *only* a hash of the prevout's scriptPubKey in the input, so that *when* the
> prevtx is malleated, the spending one remains valid. (This use case is
> currently not supported.)

Maybe for the next version of script :)
 
> > Notice that the token ScriptVersion is currently not allowed
> What happens if I put ScriptVersion=1 here?

The transaction is invalid...
 
> > === Block-malleability ===
> > 
> > For this reason the merkle tree is extended to include (append) the hash
> > of
> > the v4 transactions (and those alone) where the hash is taken over a
> 
> > data-blob that is build up from:
>
> How should nodes know where in the merkle-tree the txids end, and the
> v4hashes begin?

Because the txid based ones are not going away. So the number of transactions 
in the block can be used to determine when the pure tx-id segment stops and 
when the v4 hashes begin.  Then its up to the client to rebuild the tree from 
that list based on the larger input set to get the same root-node.

I clarified many little things on my clone of the bips, check there if you 
want to see the details.

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]

Thanks for your email Peter!

On Tuesday 20 Sep 2016 17:56:44 Peter Todd wrote:
> On Tue, Sep 20, 2016 at 07:15:45PM +0200, Tom via bitcoin-dev wrote:
> > === Serialization order===
> > 
> > The tokens defined above have to be serialized in a certain order for the
> > transaction to be well-formatted.  Not serializing transactions in the
> > order specified would allow multiple interpretations of the data which
> > can't be allowed.
> 
> If the order of the tokens is fixed, the tokens themselves are redundant
> information when tokens are required; when tokens may be omitted, a simple
> "Some/None" flag to mark whether or not the optional data has been omitted
> is appropriate.

This is addressed in the spec; 
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md

«The way towards that flexibility is to use a generic concept made popular
various decades ago with the XML format. The idea is that we give each
field a name and this means that new fields can be added or optional fields
can be omitted from individual transactions»


> Also, if you're going to break compatibility with all existing software, it
> makes sense to use a format that extends the merkle tree down into the
> transaction inputs and outputs.

Please argue your case.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Andreas Schildbach]

Just glancing over your BIP, I wonder if we should use Protobuf. It uses
this "flexible" format already and is quite compact/binary. We use
Protobuf already for the payment protocol, and there is very good tool
support.

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Wednesday 21 Sep 2016 14:00:23 Andreas Schildbach via bitcoin-dev wrote:
> Just glancing over your BIP, I wonder if we should use Protobuf. It uses
> this "flexible" format already and is quite compact/binary. We use
> Protobuf already for the payment protocol, and there is very good tool
> support.

There is a lot of overlap between different binary formats. Looking through 
the on-the-wire protocol you'll see that my spec is very similar. Practically 
all the advantages of protobuf are present in CMF. I can write you a java 
parser if you want, it should be easy to port from Qt/C++ code :)
https://github.com/bitcoinclassic/transactions

CMF: 
https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md

There is no tool support needed, just one or two classes. Which personally I 
think is an advantage.


Some advantages of CMF over protobuf from the top of my head;

* It reuses the var-int parsing that Bitcoin uses (which is itself slightly 
different from others).

* zero-copy support (not relevant for this bip, though).

* Additional values addition (i.e. adding new data) is .. tricky in protobuf.
https://developers.google.com/protocol-buffers/docs/proto#updating

* In my experience parsing a message manually (like a SOX parser) is much 
better in reporting errors and detecting wrong usages than auto-generated code 
(but personally I'm not much a fan of auto-generated APIs) at all...

* Generated parsing/writing code will not be as fast as we can make it.

* CMF is more compact (uses less bytes) for its messages.


Protobuf is something I've used before and I think we can do better. I think 
that CMF together with some support classes can do this better.

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Gregory Maxwell]

On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> BIP number for my FT spec.

This document does not appear to be concretely specified enough to
review or implement from it.

For example, it does not specify the serialization of "integer" (is it
a 32 bit word in network byte order or?) nor does it specify how the
presence of the optional fields are signaled nor the cardinality of
the inputs or outputs. For clearly variable length elements
('bytearray') no mention is made of their length encoding. etc.

Without information like this, I don't see how someone could
realistically begin reviewing this proposal.

The motivation seems unclear to me as well: The scheme is described as
'flexible' but it appears to remove flexibility from the existing
system. The "schema" appears to be hardcoded and never communicated.
If the goal is to simply have a more compact on the wire
representation, this could be done without changing the serialization
used for hashing or the serialization used for costing.

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:
> On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
> 
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
> > BIP number for my FT spec.
> 
> This document does not appear to be concretely specified enough to
> review or implement from it.
> 
> For example, it does not specify the serialization of "integer"

It refers to the external specification which is linked at the bottom.
In that spec you'll see that "Integer" is the standard var-int that Bitcoin 
has used for years.

> nor does it specify how the
> presence of the optional fields are signaled 

How does one signals an optional field except of in the spec? Thats the job of 
a specification.

> nor the cardinality of
> the inputs or outputs. 

Did you miss this in the 3rd table ?  I suggest clicking on the github bips 
repo link as tables are not easy to read in mediawiki plain format that the 
email contained.

> For clearly variable length elements
> ('bytearray') no mention is made of their length encoding. etc.

Also in the external CMF spec.
 
> Without information like this, I don't see how someone could
> realistically begin reviewing this proposal.

I agree, that would be bad. Luckily that you just missed the link :)
Here it is;
https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md

> The motivation seems unclear to me as well: The scheme is described as
> 'flexible' but it appears to remove flexibility from the existing
> system. The "schema" appears to be hardcoded and never communicated.

Being hardcoded and never communicated is what the current format does to. How 
does that "remove flexibility".

Also read my reply to Peter Todd on why this is flexible.

> If the goal is to simply have {snip}

It is not.

Thanks for asking, I understand that the CMF spec is useful to see as well. 
Hopefully you can now review it properly since I linked to it above.

Cheers!

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Christian Decker]

On Thu, Sep 22, 2016 at 10:56:31AM +0200, Tom via bitcoin-dev wrote:
> On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:
> > On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
> > 
> > <bitcoin-dev@lists.linuxfoundation.org> wrote:
> > > BIP number for my FT spec.
> > 
> > This document does not appear to be concretely specified enough to
> > review or implement from it.
> > 
> > For example, it does not specify the serialization of "integer"
> 
> It refers to the external specification which is linked at the bottom.
> In that spec you'll see that "Integer" is the standard var-int that Bitcoin 
> has used for years.

I think BIPs should be self-contained, or rely on previous BIPs,
whenever possible. Referencing an external formatting document should
be avoided and requiring readers to reverse engineer a reference
implementation doesn't seem too user friendly either. Publishing a BIP
with CMF would certainly help, and completing this spec with the
details that are missing, or only "defined" in the implementation,
would be better.

> > nor does it specify how the
> > presence of the optional fields are signaled 
> 
> How does one signals an optional field except of in the spec? Thats the job of 
> a specification.

So the presence is signaled by encountering the tag, which contains
both token type and name-reference. The encoder and decoder operations
could be described better.

> > nor the cardinality of
> > the inputs or outputs. 
> 
> Did you miss this in the 3rd table ?  I suggest clicking on the github bips 
> repo link as tables are not easy to read in mediawiki plain format that the 
> email contained.

Minor nit: that table is not well-formed. As was pointed out in the
normalized transaction ID BIP, your proposal only addresses
third-party malleability, since signers can simply change the
transaction and re-sign it. This is evident from the fact that inputs
and outputs do not have a canonical order and it would appear that
tokens can be re-ordered in segments. Dependencies of tokens inside a
segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
TxOutScript <-> TxOutValue).

Finally, allowing miners to reject transactions with unknown fields
makes the OP_NOPs unusable since they'd result in forks: non-upgraded
nodes would reject blocks from upgraded nodes.

Regards,
Christian

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Thursday 22 Sep 2016 13:10:49 Christian Decker via bitcoin-dev wrote:
> On Thu, Sep 22, 2016 at 10:56:31AM +0200, Tom via bitcoin-dev wrote:
> > On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:
> > > On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
> > > 
> > > <bitcoin-dev@lists.linuxfoundation.org> wrote:
> > > > BIP number for my FT spec.
> > > 
> > > This document does not appear to be concretely specified enough to
> > > review or implement from it.
> > > 
> > > For example, it does not specify the serialization of "integer"
> > 
> > It refers to the external specification which is linked at the bottom.
> > In that spec you'll see that "Integer" is the standard var-int that
> > Bitcoin
> > has used for years.
> 
> I think BIPs should be self-contained, or rely on previous BIPs,
> whenever possible. Referencing an external formatting document should
> be avoided 

If luke-jr thinks I should submit CMF as a BIP, I can certainly do that.
Luke, what do you think?

I don't have a preference either way.

> > > nor does it specify how the
> > > presence of the optional fields are signaled
> > 
> > How does one signals an optional field except of in the spec? Thats the
> > job of a specification.
> 
> So the presence is signaled by encountering the tag, which contains
> both token type and name-reference. The encoder and decoder operations
> could be described better.

I'm sorry, I'm not following you here. Is there a question?


> > > nor the cardinality of
> > > the inputs or outputs.
> > 
> > Did you miss this in the 3rd table ?  I suggest clicking on the github
> > bips
> > repo link as tables are not easy to read in mediawiki plain format that
> > the
> > email contained.
> 
> Minor nit: that table is not well-formed.

I am not very well versed in mediawiki tables, and I found github has some 
incompatibilities too.
The markdown one looks better;
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md

> As was pointed out in the
> normalized transaction ID BIP, your proposal only addresses
> third-party malleability, since signers can simply change the
> transaction and re-sign it.

I have to disagree. That is not malleability. Creating a new document and re-
signing it is not changing anything. Its re-creating. Something that the owner 
of the coin has every right to do.

> This is evident from the fact that inputs
> and outputs do not have a canonical order and it would appear that
> tokens can be re-ordered in segments. 

Sorry, what is evident? You seem to imply that it is uncommon that you can 
create two transactions of similar intent but using different bytes.
You would be wrong with this implication as this is very common. You can just 
alter the order of the inputs, for instance.

I am unable to see what the point is you are trying to make. Is there a 
question or a suggestion for improvement here?

> Dependencies of tokens inside a
> segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
> TxOutScript <-> TxOutValue).

Maybe you missed this line; 
  «TxInPrevHash and TxInPrevIndex
   Index can be skipped, but in any input the PrevHash always has
   to come first»

If you still see something alarming, let me know.
You can look at the code in Bitcoin Classic and notice that it really isn't 
anything complicated or worrying.


> Finally, allowing miners to reject transactions with unknown fields
> makes the OP_NOPs unusable 

Hmm, it looks like you are mixing terminology and abstraction-levels.  OP_NOP 
is a field from script and there is no discussion about any rejection based on 
script in this BIP at all.

Rejection of transactions is done on there being unrecognised tokens in the 
transaction formatting itself.

Thank you for your email to my BIP, I hope you got the answers you were 
looking for :)

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On Wed, Sep 21, 2016 at 11:32:33AM +0200, Tom wrote:
> Thanks for your email Peter!
> 
> On Tuesday 20 Sep 2016 17:56:44 Peter Todd wrote:
> > On Tue, Sep 20, 2016 at 07:15:45PM +0200, Tom via bitcoin-dev wrote:
> > > === Serialization order===
> > > 
> > > The tokens defined above have to be serialized in a certain order for the
> > > transaction to be well-formatted.  Not serializing transactions in the
> > > order specified would allow multiple interpretations of the data which
> > > can't be allowed.
> > 
> > If the order of the tokens is fixed, the tokens themselves are redundant
> > information when tokens are required; when tokens may be omitted, a simple
> > "Some/None" flag to mark whether or not the optional data has been omitted
> > is appropriate.
> 
> This is addressed in the spec; 
> https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md
> 
> «The way towards that flexibility is to use a generic concept made popular
> various decades ago with the XML format. The idea is that we give each
> field a name and this means that new fields can be added or optional fields
> can be omitted from individual transactions»

That argument is not applicable to required fields: the code to get the fields
from the extensible format is every bit as complex as the very simple code
required to deserialize/serialize objects in the current system.

In any case your BIP needs to give some explicit examples of hypothetical
upgrades in the future, how they'd take advantage of this, and what the code to
do so would look like.

> > Also, if you're going to break compatibility with all existing software, it
> > makes sense to use a format that extends the merkle tree down into the
> > transaction inputs and outputs.
> 
> Please argue your case.

See my arguments re: segwit a few months ago, e.g. the hardware wallet txin
proof use-case.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Thursday, 22 September 2016 14:26:18 CEST Peter Todd wrote:
> > «The way towards that flexibility is to use a generic concept made
> > popular various decades ago with the XML format. The idea is that we
> > give each field a name and this means that new fields can be added or
> > optional fields can be omitted from individual transactions»
> 
> That argument is not applicable to required fields: 

The argument that optional fields can be omitted is not applicable to 
required fields, you are correct. That should be rather obvious because 
required fields are not optional fields.

> the code to get the
> fields from the extensible format is every bit as complex as the very
> simple code required to deserialize/serialize objects in the current
> system.

Probably a tiny bit more complex as the current format assumes a lot more.

You may have misread my email because there was no argument made towards 
complexity. The argument was towards flexibility.

> In any case your BIP needs to give some explicit examples of hypothetical
> upgrades in the future, how they'd take advantage of this, and what the
> code to do so would look like.

Why?

> > > Also, if you're going to break compatibility with all existing
> > > software, it makes sense to use a format that extends the merkle
> > > tree down into the transaction inputs and outputs.
> > 
> > Please argue your case.
> 
> See my arguments re: segwit a few months ago, e.g. the hardware wallet
> txin proof use-case.

Please consider that I'm not going to search for something based on a vague 
reference like that, if you want to convince me you could you at least 
provide a URL?
You want me to see the value of your idea, I think you should at least 
provide the argument. Isn't that fair?

Thanks for your email Peter, would love you to put a bit more time into 
understanding flexible transactions and we can have a proper discussion 
about it.

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Christian Decker]

On Thu, Sep 22, 2016 at 02:09:38PM +0200, Tom via bitcoin-dev wrote:
> On Thursday 22 Sep 2016 13:10:49 Christian Decker via bitcoin-dev wrote:
> > 
> > I think BIPs should be self-contained, or rely on previous BIPs,
> > whenever possible. Referencing an external formatting document should
> > be avoided 
> 
> If luke-jr thinks I should submit CMF as a BIP, I can certainly do that.
> Luke, what do you think?
> 
> I don't have a preference either way.
> 
> > 
> > So the presence is signaled by encountering the tag, which contains
> > both token type and name-reference. The encoder and decoder operations
> > could be described better.
> 
> I'm sorry, I'm not following you here. Is there a question?

Nope, just clarifying how presence or absence is indicated :-)

> > 
> > Minor nit: that table is not well-formed.
> 
> I am not very well versed in mediawiki tables, and I found github has some 
> incompatibilities too.
> The markdown one looks better;
> https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md

It's just some rows have 3 columns, others have 2. It's a minor nit
really.

> > As was pointed out in the
> > normalized transaction ID BIP, your proposal only addresses
> > third-party malleability, since signers can simply change the
> > transaction and re-sign it.
> 
> I have to disagree. That is not malleability. Creating a new document and re-
> signing it is not changing anything. Its re-creating. Something that the owner 
> of the coin has every right to do.

Same thing I was arguing back then, however Luke pointed out that
malleability just refers to the possibility of modifying a transaction
after the fact. Always referring to "third-party malleability" avoids
this ambiguity.

> > This is evident from the fact that inputs
> > and outputs do not have a canonical order and it would appear that
> > tokens can be re-ordered in segments. 
> 
> Sorry, what is evident? You seem to imply that it is uncommon that you can 
> create two transactions of similar intent but using different bytes.
> You would be wrong with this implication as this is very common. You can just 
> alter the order of the inputs, for instance.
> 
> I am unable to see what the point is you are trying to make. Is there a 
> question or a suggestion for improvement here?
> 
> > Dependencies of tokens inside a
> > segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
> > TxOutScript <-> TxOutValue).
> 
> Maybe you missed this line; 
>   «TxInPrevHash and TxInPrevIndex
>    Index can be skipped, but in any input the PrevHash always has
>    to come first»

Nope, that is exactly the kind of dependency I was talking
about. Instead of nesting a construct like the current transactions
do, you rely on the order of tokens to imply that they belong
together.

> If you still see something alarming, let me know.
> You can look at the code in Bitcoin Classic and notice that it really isn't 
> anything complicated or worrying.
> 
> 
> > Finally, allowing miners to reject transactions with unknown fields
> > makes the OP_NOPs unusable 
> 
> Hmm, it looks like you are mixing terminology and abstraction-levels.  OP_NOP 
> is a field from script and there is no discussion about any rejection based on 
> script in this BIP at all.
> 
> Rejection of transactions is done on there being unrecognised tokens in the 
> transaction formatting itself.

Ah, thanks for clearing that up. However, the problem persists, if we
add new fields that a non-upgraded node doesn't know about and it
rejects transactions containing it, we'll have a hard-fork. It should
probably not reject transactions with unknown fields if the
transaction is included in a block.

> Thank you for your email to my BIP, I hope you got the answers you were 
> looking for :)

Cheers,
Christian

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Friday, 23 September 2016 13:42:36 CEST Christian Decker via bitcoin-dev wrote:
> > I have to disagree. That is not malleability. Creating a new document
> > and re- signing it is not changing anything. Its re-creating.
> > Something that the owner of the coin has every right to do.
> Same thing I was arguing back then, however Luke pointed out that
> malleability just refers to the possibility of modifying a transaction
> after the fact.

I am not a fan of redefining dictionary words. I'll stick to the 
universally excepted one, thanks.

> Nope, that is exactly the kind of dependency I was talking
> about. Instead of nesting a construct like the current transactions
> do, you rely on the order of tokens to imply that they belong
> together.


> if we
> add new fields that a non-upgraded node doesn't know about and it
> rejects transactions containing it, we'll have a hard-fork. It should
> probably not reject transactions with unknown fields if the
> transaction is included in a block.

This is addressed here;
https://github.com/bitcoin/bips/blob/master/bip-0134.mediawiki#future-extensibility

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Friday, 23 September 2016 13:55:50 CEST Christian Decker via bitcoin-dev 
wrote: 
> Not sure if the comparison to XML and HTML holds: the lack of closing
> tags makes the meaning of individual tokens ambiguous, like I pointed
> out before. The use of segments gives at most two levels of nesting,
> so any relationship among tokens in the same segment has to rely on
> their relative position, which could result in ambiguities, like
> whether a tag refers to a single input or the transaction as a whole.


Practically all tagged formats make ordering a requirement, so indeed this 
is relevant, and not unique.

For instance if you write;
  <div> Some line </br>Another line</br>3rd line</div>
you can get a good idea of how ordering is relevant. You can reuse any item 
many times.

Whenever there is a possible confusion the specification specifically 
explains which order to use.

I'm not sure what you mean with the idea this;

>  The use of segments gives at most two levels of nesting

It looks like you assume there is some opening and closing tags, since 
otherwise there would be no nesting.
Such tags are not intended, nor documented.

> so any relationship among tokens in the same segment has to rely on
> their relative position, which could result in ambiguities, like
> whether a tag refers to a single input or the transaction as a whole.

I quoted parts of the spec in your previous email stating the same thing, 
but I'll repeat here.
Any place that has any sort of possibility to be ambiguous is specified 
specifically to have an order.  This makes writing and parsing easier.

Since you wrote two emails now with the same issue, and I addressed it 
twice, I would urge you to write out some examples which may be confusing 
and if you find that the spec is indeed missing requirements then please 
share it with us.  I did this some time ago and it helps understanding the 
ideas by having actual explicit examples.  I am not aware of any sort of 
ambiguities that the spec allows.

Cheers!

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Christian Decker]

On Thu, Sep 22, 2016 at 08:37:29PM +0200, Tom via bitcoin-dev wrote:
> On Thursday, 22 September 2016 14:27:29 CEST Peter Todd wrote:
> > CSV uses per-input sequence numbers; you only have a per-tx equivalent.
> 
> I think you misunderstand tagged systems at a very basic level.  You think 
> that html can only use a bold tag <b> once in a document? Thats equivalent 
> to what you are saying.
> 
> Your comment is rather embarrassing, I have to point out. You may want to 
> read a bit more before you comment more.

Not sure if the comparison to XML and HTML holds: the lack of closing
tags makes the meaning of individual tokens ambiguous, like I pointed
out before. The use of segments gives at most two levels of nesting,
so any relationship among tokens in the same segment has to rely on
their relative position, which could result in ambiguities, like
whether a tag refers to a single input or the transaction as a whole.

Cheers,
Christian

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Thursday, 22 September 2016 21:59:12 CEST Jonas Schnelli via bitcoin-dev 
wrote:
> Hi Tom
> 
> > I think you misunderstand tagged systems at a very basic level.  You
> > think that html can only use a bold tag <b> once in a document? Thats
> > equivalent to what you are saying.
> 
> Would the "additional" segment contain the same amount of
> nSequence-equivalent token as the number of inputs in the "inputs"
> segment?

At this point I don't know what it should look like, I have not had time to 
look deeply into BIP68.  Is this what you would suggest it to look like?
I rather figured spending limitations would be assigned to an output, not 
an input.

> However, I think that should be mentioned/specified in the BIP.

It can be, and likely should be.  This BIP doesn't pretend to be finished 
yet.

I welcome any and all discussion about this, it only serves to make the end 
result stronger!

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Jonas Schnelli]

[-- Attachment #1.1: Type: text/plain, Size: 599 bytes --]

Hi Tom

> I think you misunderstand tagged systems at a very basic level.  You think 
> that html can only use a bold tag <b> once in a document? Thats equivalent 
> to what you are saying.

Would the "additional" segment contain the same amount of
nSequence-equivalent token as the number of inputs in the "inputs" segment?
What if you only want to add a per-input-token in the additional segment
for a certain input (assume last input)?
I guess the fundamental difference to html is the possible nesting.

However, I think that should be mentioned/specified in the BIP.

</jonas>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Thursday, 22 September 2016 14:27:29 CEST Peter Todd wrote:
> CSV uses per-input sequence numbers; you only have a per-tx equivalent.

I think you misunderstand tagged systems at a very basic level.  You think 
that html can only use a bold tag <b> once in a document? Thats equivalent 
to what you are saying.

Your comment is rather embarrassing, I have to point out. You may want to 
read a bit more before you comment more.

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

On Thu, Sep 22, 2016 at 10:47:03AM +0200, Tom via bitcoin-dev wrote:
> On Wednesday 21 Sep 2016 18:45:55 adiabat via bitcoin-dev wrote:
> > Hi-
> > 
> > One concern is that this doesn't seem compatible with Lightning as
> > currently written.  Most relevant is that non-cooperative channel close
> > transactions in Lightning use OP_CHECKSEQUENCEVERIFY, which references the
> > sequence field of the txin; if the txin doesn't have a sequence number,
> > OP_CHECKSEQUENCEVERIFY can't work.
> > 
> > LockByBlock and LockByTime aren't described and there doesn't seem to be
> > code for them in the PR (186).  If there's a way to make OP_CLTV and OP_CSV
> > work with this new format, please let us know, thanks!
> 
> LockByBlock and LockByTime are still TODOs because I didn't have time to go
> in-dept into how BIP68 does the encoding.
> The intent is that these tags, while loading, will set the sequence integer in 
> the txin as the old version does. And while saving we do the reverse.
> 
> In other words; the lack of sequence number in the saved format doesn't affect 
> the in-memory format of the transaction. The in-memory version is the one that 
> script will operate on.
> 
> This means that there is no change in how CSV will work before and after on 
> any level other than serialisation.

CSV uses per-input sequence numbers; you only have a per-tx equivalent.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[Tom]

On Wednesday 21 Sep 2016 18:45:55 adiabat via bitcoin-dev wrote:
> Hi-
> 
> One concern is that this doesn't seem compatible with Lightning as
> currently written.  Most relevant is that non-cooperative channel close
> transactions in Lightning use OP_CHECKSEQUENCEVERIFY, which references the
> sequence field of the txin; if the txin doesn't have a sequence number,
> OP_CHECKSEQUENCEVERIFY can't work.
> 
> LockByBlock and LockByTime aren't described and there doesn't seem to be
> code for them in the PR (186).  If there's a way to make OP_CLTV and OP_CSV
> work with this new format, please let us know, thanks!

LockByBlock and LockByTime are still TODOs because I didn't have time to go
in-dept into how BIP68 does the encoding.
The intent is that these tags, while loading, will set the sequence integer in 
the txin as the old version does. And while saving we do the reverse.

In other words; the lack of sequence number in the saved format doesn't affect 
the in-memory format of the transaction. The in-memory version is the one that 
script will operate on.

This means that there is no change in how CSV will work before and after on 
any level other than serialisation.

Flexible Transactions is definitely meant to support the Lightning Network, so 
any problems you find is something we should solve before it ships.

Thanks for your email!

[bitcoin-dev] Requesting BIP assignment; Flexible Transactions.

[adiabat]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

Hi-

One concern is that this doesn't seem compatible with Lightning as
currently written.  Most relevant is that non-cooperative channel close
transactions in Lightning use OP_CHECKSEQUENCEVERIFY, which references the
sequence field of the txin; if the txin doesn't have a sequence number,
OP_CHECKSEQUENCEVERIFY can't work.

LockByBlock and LockByTime aren't described and there doesn't seem to be
code for them in the PR (186).  If there's a way to make OP_CLTV and OP_CSV
work with this new format, please let us know, thanks!

-Tadge

[-- Attachment #2: Type: text/html, Size: 622 bytes --]