Thanks for this work Jeremy.
I know we've discussed this before, but I'll restate my concerns with adding a new "global" state variable to the Script interpreter for tracking whether the previous opcode was a push-data operation or not. While it isn't so hard to implement this in Bitcoin Core's Script interpreter, adding a new global state variable adds that much more complexity to anyone trying to formally model Script semantics. Perhaps one can argue that there is already (non-stack) state in Script, e.g. to deal with CODESEPARATOR, so why not add more? But I'd argue that we should avoid making bad problems worse.
If we instead make the CHECKTEMPLATEVERIFY operation fail if it isn't preceded by (or alternatively followed by) an appropriate sized (canonical?) PUSHDATA constant, even in an unexecuted IF branch, then we can model the Script semantics by considering the PUSHDATA-CHECKTEMPLATEVERIFY pair as a single operation. This allows implementations to consider improper use of CHECKTEMPLATEVERIFY as a parsing error (just as today unbalanced IF-ENDIF pairs can be modeled as a parsing error, even though that isn't how it is implemented in Bitcoin Core).
I admit we would lose your soft-fork upgrade path to reading values off the stack; however, in my opinion, this is a reasonable tradeoff. When we are ready to add programmable covenants to Script, we'll do so by adding CAT and operations to push transaction data right onto the stack, rather than posting a preimage to this template hash.
Pleased to announce refinements to the BIP draft for OP_CHECKTEMPLATEVERIFY (replaces previous OP_SECURETHEBAG BIP). Primarily:
1) Changed the name to something more fitting and acceptable to the community
2) Changed the opcode specification to use the argument off of the stack with a primitive constexpr/literal tracker rather than script lookahead
3) Permits future soft-fork updates to loosen or remove "constexpr" restrictions
4) More detailed comparison to alternatives in the BIP, and why OP_CHECKTEMPLATEVERIFY should be favored even if a future technique may make it semi-redundant.
Please see:
I believe this addresses all outstanding feedback on the design of this opcode, unless there are any new concerns with these changes.
I'm also planning to host a review workshop in Q1 2020, most likely in San Francisco. Please fill out the form here
https://forms.gle/pkevHNj2pXH9MGee9 if you're interested in participating (even if you can't physically attend).
And as a "but wait, there's more":
1) RPC functions are under preliminary development, to aid in testing and evaluation of OP_CHECKTEMPLATEVERIFY. The new command `sendmanycompacted` shows one way to use OP_CHECKTEMPLATEVERIFY. See:
https://github.com/JeremyRubin/bitcoin/tree/checktemplateverify-rpcs. `sendmanycompacted` is still under early design. Standard practices for using OP_CHECKTEMPLATEVERIFY & wallet behaviors may be codified into a separate BIP. This work generalizes even if an alternative strategy is used to achieve the scalability techniques of OP_CHECKTEMPLATEVERIFY.
2) Also under development are improvements to the mempool which will, in conjunction with improvements like package relay, help make it safe to lift some of the mempool's restrictions on longchains specifically for OP_CHECKTEMPLATEVERIFY output trees. See:
https://github.com/bitcoin/bitcoin/pull/17268 This work offers an improvement irrespective of OP_CHECKTEMPLATEVERIFY's fate.
Neither of these are blockers for proceeding with the BIP, as they are ergonomics and usability improvements needed once/if the BIP is activated.
See prior mailing list discussions here:
Thanks to the many developers who have provided feedback on iterations of this design.
Best,
Jeremy