[bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Anthony Towns]

Subhead: "Nobody expects a Bitcoin Inquistion? C'mon man, *everyone*
expects a Bitcoin Inquisition."

As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
in the year [0], the question of "how to successfully get soft fork
ideas from concept to deployment" doesn't really have a good answer today.

Obviously, a centralised solution to this problem exists: we could
establish a trusted group, perhaps one containing devs, industry
representatives, investors, etc, have them review proposals and their
implementations, and follow their lead when they decide that a proposal
has met their standards and should be widely deployed. Some might even
say "sipa is precisely that group". The problem with having a group of
that nature isn't one of effectiveness, but rather that they are then
vulnerable to pressure and corruption, which isn't desirable if we want
everyone using Bitcoin to truly be peers, and often isn't desirable for
the prospective members of the group either. So that's not something we
should want people to volunteer for, nor is it a duty we should thrust
on people. Or at least, that's my opinion, anyway.

I think any alternative approach to doing consensus changes (while
avoiding a chain split) has to look something like this:

 * propose an idea (research phase)
 * implement the idea (development phase)
 * demonstrate the idea is worthwhile (evaluation phase)
 * once everyone is convinced, activate (deployment phase)

Without an evaluation phase that is thorough enough to convince (almost)
everyone, I think deployment becomes controversial and perhaps effectively
impossible (at least without some trusted leadership group). But with an
evaluation phase that demonstrates to everyone who's interested that the
proposal has actual value, minimal cost and no risk, I think activation
could be fairly easy and straightforward.

I contend that the most significant problem we have is in the "evaluation
phase". How do you convince enough people that a change is sufficiently
beneficial to justify the risk of messing with their money? If you're
only trying to convince a few experts, then perhaps you can do that with
papers and talks; but limiting the evaluation to only a few experts is
effectively just falling back to the centralised approach.

So I think that means that part of the "evaluation phase" should involve
implementing real systems on top of the proposed change, so that you
can demonstrate real value from the change. It's easy to say that
"CTV can enable vaults" or "CTV can make opening a lightning channel
non-interactive" -- but it's harder to go from saying something
is possible to actually making it happen, so, at least to me, it
seems reasonable to be skeptical of people claiming benefits without
demonstrating they're achievable in practice.

I contend the easiest way we could make it easy to demonstrate a soft
fork working as designed is to deploy it on the default global signet,
essentially as soon as it has a fully specified proposal and a reasonably
high-quality implementation.

The problem with that idea is that creates a conundrum: you can't activate
a soft fork on the default signet without first merging the code into
bitcoin core, you can't merge the code into bitcoin core until it's been
fully evaluated, and the way you evaluate it is by activating it on the
default signet?

I think the weakest link in that loop is the first one: what if we did
activate soft forks on the default signet prior to the code being merged
into core? To that end, I'm proposing a fork of core that I'm calling
"bitcoin-inquisition", with the idea that it branches from stable
releases of core and adds support for proposed changes to consensus
(CTV, ANYPREVOUT, TLUV, OP_CAT, etc...) and potentially also relay
policy (relay changes are often implied by consensus changes, but also
potentially things like package relay).

  https://github.com/bitcoin-inquisition/bitcoin/wiki
  https://github.com/bitcoin-inquisition/bitcoin/pulls

The idea being that if you're trying to work on "upgrading lightning
to support eltoo", you can iterate through changes needed to consensus
(via bitcoin-inquisition) and client apps (cln, lnd, eclair etc), while
testing them in public (on signet) and having any/all the pre-existing
signet infrastructure available (faucets, explorers etc) without having
to redeploy it yourself. Having multiple consensus changes deployed in
one place also seems like it might make it easier to compare alternative
approaches (eg CTV vs ANYPREVOUT vs OP_TXHASH vs OP_TX, etc).

So that's the concept. For practical purposes, I haven't yet merged
either CTV or APO support into the bitcoin-inquisition 23.0 branch yet,
and before actually mining blocks I want to make the signet miner able
to automatically detect/recover if the bitcoin-inquisition node either
crashes or starts producing incompatible blocks.

Anyway, I wanted to post the idea publicly, both to give folks an
opportunity to poke holes in the idea, or to suggest any further
improvements or otherwise do any review before the CTV and APO patches
get merged.

Some other details that may be of interest.

The biggest challenge with soft forks and the idea of "iterating
through changes" is that making improvements can create a hard fork,
which then forces everyone running old software to update, which can be
pretty inconvenient, especially if you don't actually care about that
change. Since signet (and regtest) mining is effectively permissioned,
we can avoid that problem by having all these proposed soft forks
come with a pre-baked ability to abandon the soft fork (much as David
Harding described in [1]). Once a soft fork is abandoned, it can either
be ignored forever (and later versions of the software can not include
the code to enforce it at all), or it can be replaced by a new version
of the soft fork.

Another benefit that comes from signet chains being permissioned is
that miners can be expected to coordinate upgrading out of band, so
there is no need for a 90% signalling threshold. Instead, activation
(and abandonment) of a soft fork can be triggered by a single block
signalling. That further means there is no need for any individual
block to signal for multiple forks, and instead of having 29 different
signals, we can instead easily have up to 2**29. I've chosen to make
the standard signal have 16 bits for specifying a bip number (0-65535)
and 8 bits for specifying a version of that bip, which seems like it
should be more than enough at least for a while. More details at [2].

I'm basing bitcoin-inquisition solely off stable releases. This is partly
because it can be annoying to constantly rebase consensus changes aginst
bitcoin core's master branch, but also I think it might help consensus
changes be easily backported once they pass the "evaluation phase"
and move into the "deployment phase".

I'm not sure what level of code quality PRs should have before being
merged into bitcoin-inquisition. I think CTV is plenty good enough,
but I'm not sure about APO, particularly its test coverage. If you want
to influence what becomes the tradition here, contributing a review,
or posting patches against the upsteam branch might be a good start?

Does this make the global default signet miners, or perhaps the
bitcoin-inquisition maintainers the "trusted group" that we want to
avoid? Hopefully not -- anyone can run their own fork or do their own
fork of bitcoin core, so if the miners/maintainers start trying to
arbitrarily block proposals they can be worked around without too much
hassle. And since they're clearly separate from any of the actions that
need to be taken for actual deployment once activation is complete,
they shouldn't have any ability to unduly promote fork proposals that
people aren't fully satisfied are ready for deployment.

Cheers,
aj

[0] https://bitcoinops.org/en/newsletters/2022/04/27/#discussion-about-activating-ctv

[1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-April/020242.html

[2] https://github.com/bitcoin-inquisition/bitcoin/wiki/Heretical-Deployments

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Matt Corallo]

Apologies for any typos, somewhat jet-lagged atm.

On 9/16/22 3:15 AM, Anthony Towns via bitcoin-dev wrote:
> Subhead: "Nobody expects a Bitcoin Inquistion? C'mon man, *everyone*
> expects a Bitcoin Inquisition."
> 
> As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
> in the year [0], the question of "how to successfully get soft fork
> ideas from concept to deployment" doesn't really have a good answer today.

I strongly disagree with this. Going back many, many years we've had many discussions about fork 
process, and the parts people (historically) agreed with tend to be:

(1) come up with an idea
(2) socialize the idea in the technical community, see if anyone comes up with any major issues or 
can suggest better ideas which solve the same use-cases in cleaner ways
(3) propose the concrete idea with a more well-defined strawman, socialize that, get some kind of 
rough consensus in the loosely-defined, subjective, "technical community" (ie just ask people and 
adapt to feedback until you have found some kind of average of the opinions of people you, the 
fork-champion, think are reasonably well-informed!).
(4) okay, admittedly beyond this is a bit less defined, but we can deal with it when we get there.

Turns out, the issue today is a lack of champions following steps 1-3, we can debate what the 
correct answer is to step (4) once we actually have people who want to be champions who are willing 
to (humbly) push an idea forward towards rough agreement of the world of technical bitcoiners 
(without which I highly doubt you'd ever see broader-community consensus).

Matt

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Anthony Towns]

On Fri, Sep 16, 2022 at 12:46:53PM -0400, Matt Corallo via bitcoin-dev wrote:
> On 9/16/22 3:15 AM, Anthony Towns via bitcoin-dev wrote:
> > As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
> > in the year [0], the question of "how to successfully get soft fork
> > ideas from concept to deployment" doesn't really have a good answer today.
> I strongly disagree with this.

Okay? "X is good" is obviously just a statement of opinion, so if you
want to disagree, that's obviously allowed. 

I also kind of feel like that's the *least* interesting paragraph in the
entire email to talk further about; if you think the current answer's
already good, then the rest of the mail's just about (hopefully) making
it better, which would be worthwhile anyway?

> Going back many, many years we've had many
> discussions about fork process, and the parts people (historically) agreed
> with tend to be:
> (1) come up with an idea
> (2) socialize the idea in the technical community, see if anyone comes up
> with any major issues or can suggest better ideas which solve the same
> use-cases in cleaner ways
> (3) propose the concrete idea with a more well-defined strawman, socialize
> that, get some kind of rough consensus in the loosely-defined, subjective,
> "technical community" (ie just ask people and adapt to feedback until you
> have found some kind of average of the opinions of people you, the
> fork-champion, think are reasonably well-informed!).
> (4) okay, admittedly beyond this is a bit less defined, but we can deal with it when we get there.
> Turns out, the issue today is a lack of champions following steps 1-3, we
> can debate what the correct answer is to step (4) once we actually have
> people who want to be champions who are willing to (humbly) push an idea
> forward towards rough agreement of the world of technical bitcoiners
> (without which I highly doubt you'd ever see broader-community consensus).

Personally, I think this is easily refuted by contradiction.

1) If we did have a good answer for how to progress a soft-fork, then
the great consensus cleanup [0] would have made more progress over the
past 3.5 years. Maybe not all of the ideas in it were unambiguously good
[1], but personally, I'm convinced at least some of them are, and I
don't think I'm alone in thinking that. Even if the excuse is that its
original champion wasn't humble enough, there's something wrong with
the process if there doesn't exist some other potential champion with
the right balance of humility, confidence, interest and time who could
have taken it over in that timeframe.

2) Many will argue that CTV has already done steps (1) through (3) above:
certainly there's been an idea, it's been socialised through giving talks,
having discussion forums, having research workshops [2], documenting use
cases use cases; there's been a concrete implementation for years now,
with a test network that supports the proposed feature, and new tools
that demonstrate some of the proposed use cases, and while alternative
approaches have been suggested [3], none of them have even really made
it to step (2), let alone step (3). So that leaves a few possibilities
to my mind:

 * CTV should be in step (4), and its lack of definition is a problem,
   and trying the "deal with it when we get there" approach is precisely
   what didn't work back in April.

 * The evaluation process is too inconclusive: it should either be
   saying "CTV is not good enough, fix these problems", or "CTV hasn't
   sufficiently demonstrated its value/cost, work on X next", but it
   isn't.

 * Parts (2) to (3) are too hard, and that's preventing alternatives
   from making progress, which in turn is preventing people from
   being able to decide whether CTV is the superior approach, or some
   alternative is.

But each of those possibilities indicates a significant problem with
our answer for developing soft forks.

I guess my belief is that it's mostly (2) and (3) being too hard (which
taproot overcame because many were excited about it, and CTV overcame
because Jeremy's put a lot of effort into it; but consensus cleanup,
APO, simplicity, TXHASH, etc have not similarly overcome at this point),
which leads to the evaluation process being inconclusive when plausible
alternatives exist. 

(In particular, I think having the process be massively difficult is
going to naturally cause any "humble" champion to decide that they're
not up to the task of following the process through to completion)

Anyway, that's some additional reasons why I believe what I said above,
in case that's interesting. But like I said at the start, if you still
disagree, that's fine of course.

Cheers,
aj

[0] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-March/016714.html
    https://github.com/bitcoin/bitcoin/pull/15481
    https://github.com/bitcoin/bitcoin/pull/15482

[1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-March/016765.html
    https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-March/016763.html
    https://github.com/bitcoin/bitcoin/pull/15482#issuecomment-469822630

[2] https://utxos.org/workshops/

[3] TXHASH https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-January/019813.html
    TX https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020450.html
    Elements-style opcodes https://twitter.com/rusty_twit/status/1518007923896578048
      cf https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-February/019851.html
    ANYPREVOUT? https://twitter.com/darosior/status/1474375301262151684
    Simplicity? https://twitter.com/Mario_Gibney/status/1403890965903859718
    Lisp? https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-March/020036.html

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Matt Corallo]

On 9/17/22 2:14 AM, Anthony Towns wrote:
> On Fri, Sep 16, 2022 at 12:46:53PM -0400, Matt Corallo via bitcoin-dev wrote:
>> On 9/16/22 3:15 AM, Anthony Towns via bitcoin-dev wrote:
>>> As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
>>> in the year [0], the question of "how to successfully get soft fork
>>> ideas from concept to deployment" doesn't really have a good answer today.
>> I strongly disagree with this.
> 
> Okay? "X is good" is obviously just a statement of opinion, so if you
> want to disagree, that's obviously allowed.
> 
> I also kind of feel like that's the *least* interesting paragraph in the
> entire email to talk further about; if you think the current answer's
> already good, then the rest of the mail's just about (hopefully) making
> it better, which would be worthwhile anyway?

No, I think its at least a good chunk of the "statement of problem". Yes, more testing is good, and 
this project is a way to get that. Cool. But implying that lack of test frameworks is in any 
material way part of the lack of movement on forks in Bitcoin I think is very wrong, so its worth 
pointing out, whether the particular project is useful or not is separate.

>> Going back many, many years we've had many
>> discussions about fork process, and the parts people (historically) agreed
>> with tend to be:
>> (1) come up with an idea
>> (2) socialize the idea in the technical community, see if anyone comes up
>> with any major issues or can suggest better ideas which solve the same
>> use-cases in cleaner ways
>> (3) propose the concrete idea with a more well-defined strawman, socialize
>> that, get some kind of rough consensus in the loosely-defined, subjective,
>> "technical community" (ie just ask people and adapt to feedback until you
>> have found some kind of average of the opinions of people you, the
>> fork-champion, think are reasonably well-informed!).
>> (4) okay, admittedly beyond this is a bit less defined, but we can deal with it when we get there.
>> Turns out, the issue today is a lack of champions following steps 1-3, we
>> can debate what the correct answer is to step (4) once we actually have
>> people who want to be champions who are willing to (humbly) push an idea
>> forward towards rough agreement of the world of technical bitcoiners
>> (without which I highly doubt you'd ever see broader-community consensus).
> 
> Personally, I think this is easily refuted by contradiction.
> 
> 1) If we did have a good answer for how to progress a soft-fork, then
> the great consensus cleanup [0] would have made more progress over the
> past 3.5 years

No? Who is the champion for it? I haven't been. No one else is obliged to take up the reins and run 
with it, that's not how open-source works. And no one has emerged who has strong interest in doing 
so, and that's totally fine. It means it hasn't made any progress, but that's an indication that no 
one feels strongly enough about it that its risen to the top of their personal priority list so 
clearly doesn't *need* to make progress.

> Maybe not all of the ideas in it were unambiguously good
> [1], but personally, I'm convinced at least some of them are, and I
> don't think I'm alone in thinking that. Even if the excuse is that its
> original champion wasn't humble enough, there's something wrong with
> the process if there doesn't exist some other potential champion with
> the right balance of humility, confidence, interest and time who could
> have taken it over in that timeframe.

No? Its not up to the community to find a champion for someone who wants a fork to happen. Either 
someone thinks its a good enough idea that they step up, or no one does. If no one does, then so be 
it. If the original proper (me, in this case) thought it was that important then its *their* 
responsibility to be the champion, no one else's.

> 2) Many will argue that CTV has already done steps (1) through (3) above:
> certainly there's been an idea, it's been socialised through giving talks,
> having discussion forums, having research workshops [2], documenting use
> cases use cases; there's been a concrete implementation for years now,
> with a test network that supports the proposed feature, and new tools
> that demonstrate some of the proposed use cases, and while alternative
> approaches have been suggested [3], none of them have even really made
> it to step (2), let alone step (3).

I don't really see how you can make this argument seriously. Honestly, if a soft-fork BIP only has 
one author on the list, then I'm not sure one can argue that step (3) has really been completed, and 
maybe not even step (2).

> So that leaves a few possibilities
> to my mind:

>   * CTV should be in step (4), and its lack of definition is a problem,
>     and trying the "deal with it when we get there" approach is precisely
>     what didn't work back in April.
> 
>   * The evaluation process is too inconclusive: it should either be
>     saying "CTV is not good enough, fix these problems", or "CTV hasn't
>     sufficiently demonstrated its value/cost, work on X next", but it
>     isn't.
> 
>   * Parts (2) to (3) are too hard, and that's preventing alternatives
>     from making progress, which in turn is preventing people from
>     being able to decide whether CTV is the superior approach, or some
>     alternative is.

I think this is most of it, but its not that they're too hard, its that people are *too busy*. There 
seemed to be more positive feedback, for example, to Rusty's proposal, but being the champion for a 
soft-fork is a full-time job for months on end, and last I checked Rusty has a lightning 
implementation to maintain, which tends to be a more-than-full-time job already.

To my knowledge, no one but Jeremy has made any serious attempt at being the champion for a 
soft-fork since Taproot, and before that Segwit (if someone reading this who contributes to Core 
already wants to, and isn't sure how to, there's lots of people who would happily mentor you! I'm 
sure you can figure out who to reach out to!).

Matt

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Michael Folkson]

I agree with Matt. The less said about the "Aw shucks Jeremy didn't know that CTV didn't have community consensus at the time" [0] and "it was the lack of process that was the problem" the better. If people don't care about lack of community consensus there is no process in a permissionless, open source community that can stop them or direct them down a more patient, productive path (I tried). I think it is a shame because I think (maybe I'm wrong) at least in the technical community there is an understanding that long term Bitcoin is far from finished in exhausting its potential and we do need people who will work on changes that we'll need in the long term.

There are a few interesting things in here though. I'm not convinced by the name (bitcoin-inquisition, shedpaint, shedpaint, let's park that for the moment) but I do like the idea of signet having soft fork proposals enabled on it [1] whether that be CTV, APO etc and if that requires more of the signet code to be moved out of the Core repo so be it. I'm surprised more isn't being done on Liquid already with what possible future functionality is (and could be) enabled [2] there but maybe there is more happening than I'm aware of. Protocols or use cases built out and demonstrated on signet (and/or Liquid/sidechains) seem an obvious stepping stone to me for convincing the community that it is potentially worth taking the chain split risk for a particular upgrade. It is a long slog to get community consensus on an upgrade (Taproot certainly was a slog) but I think the vast majority of us think Taproot was worth that slog and Bitcoin would be poorer today without it.

The Great Consensus Cleanup is an interesting example in that I get Matt doesn't have time to champion it or focus on it with his LDK commitments but I have no idea where it would rank on his long term priority list if he wasn't working on LDK. Similarly I have no idea what people who understand this evolving system much better than I do are thinking with regards to say adding new opcodes, sighash flags versus say waiting on Simplicity and possibly adding new functionality within that potential upgrade. For people like me who are extremely unlikely to propose their own consensus change(s) getting some signal on what to spend time digging into would be useful rather than second guessing what people are thinking. There is a danger that you flirt with perceived public roadmaps when possible authority figures stick their necks out and effectively say "I'm not in charge but in an imaginary world where I was this is my current thinking of the ordering in which we could improve this system long term. But this could change depending on x, y and z and possible upgrades are only ready when they're ready and they have community consensus." There is no way people don't play these exercises in their minds. I do, I just have very few answers :) I personally think APO is in prime position to improve Lightning channel state management with eltoo and if it enables some covenant schemes too that seems like an added bonus. On APO versus waiting for APO like functionality in Simplicity I have no idea. Work on APO/eltoo and Simplicity both seem to be progressing in parallel so the APO versus Simplicity discussion perhaps rests on whether people think certain covenants should only really be enabled once we have the security guarantees of Simplicity [3] (if at all).

Antoine's covenant R&D effort [4] seems really promising and I hope the shenanigans from earlier in the year don't put people off from engaging with that. Hopefully we can see more exploration, development and research in covenants (e.g. this excellent research paper "Bitcoin Covenants: Three Ways to Control the Future" [5]) and we can foster a community which has very high standards, is open to new ideas and new research but can avoid these months long resisting chain split fights. I think the discussion would be much more interesting and much more productive if people didn't have to think "If I express a view now it will be used to attack me personally later" or worse "If I express a view now it will be used to justify an upcoming chain split". 

[0]: https://gist.github.com/michaelfolkson/352a503f4f9fc5de89af528d86a1b718
[1]: https://bitcoin.stackexchange.com/questions/98642/can-we-experiment-on-signet-with-multiple-proposed-soft-forks-whilst-maintaining
[2]: https://bitcoin.stackexchange.com/questions/109764/what-opcodes-are-supported-on-liquid-but-not-yet-on-bitcoin
[3]: https://bitcoinops.org/en/topics/simplicity/
[4]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-September/020912.html
[5]: https://arxiv.org/pdf/2006.16714.pdf

--
Michael Folkson
Email: michaelfolkson at protonmail.com
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3


------- Original Message -------
On Saturday, September 17th, 2022 at 09:39, Matt Corallo via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:


> 
> On 9/17/22 2:14 AM, Anthony Towns wrote:
> 
> > On Fri, Sep 16, 2022 at 12:46:53PM -0400, Matt Corallo via bitcoin-dev wrote:
> > 
> > > On 9/16/22 3:15 AM, Anthony Towns via bitcoin-dev wrote:
> > > 
> > > > As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
> > > > in the year [0], the question of "how to successfully get soft fork
> > > > ideas from concept to deployment" doesn't really have a good answer today.
> > > > I strongly disagree with this.
> > 
> > Okay? "X is good" is obviously just a statement of opinion, so if you
> > want to disagree, that's obviously allowed.
> > 
> > I also kind of feel like that's the least interesting paragraph in the
> > entire email to talk further about; if you think the current answer's
> > already good, then the rest of the mail's just about (hopefully) making
> > it better, which would be worthwhile anyway?
> 
> 
> No, I think its at least a good chunk of the "statement of problem". Yes, more testing is good, and
> this project is a way to get that. Cool. But implying that lack of test frameworks is in any
> material way part of the lack of movement on forks in Bitcoin I think is very wrong, so its worth
> pointing out, whether the particular project is useful or not is separate.
> 
> > > Going back many, many years we've had many
> > > discussions about fork process, and the parts people (historically) agreed
> > > with tend to be:
> > > (1) come up with an idea
> > > (2) socialize the idea in the technical community, see if anyone comes up
> > > with any major issues or can suggest better ideas which solve the same
> > > use-cases in cleaner ways
> > > (3) propose the concrete idea with a more well-defined strawman, socialize
> > > that, get some kind of rough consensus in the loosely-defined, subjective,
> > > "technical community" (ie just ask people and adapt to feedback until you
> > > have found some kind of average of the opinions of people you, the
> > > fork-champion, think are reasonably well-informed!).
> > > (4) okay, admittedly beyond this is a bit less defined, but we can deal with it when we get there.
> > > Turns out, the issue today is a lack of champions following steps 1-3, we
> > > can debate what the correct answer is to step (4) once we actually have
> > > people who want to be champions who are willing to (humbly) push an idea
> > > forward towards rough agreement of the world of technical bitcoiners
> > > (without which I highly doubt you'd ever see broader-community consensus).
> > 
> > Personally, I think this is easily refuted by contradiction.
> > 
> > 1) If we did have a good answer for how to progress a soft-fork, then
> > the great consensus cleanup [0] would have made more progress over the
> > past 3.5 years
> 
> 
> No? Who is the champion for it? I haven't been. No one else is obliged to take up the reins and run
> with it, that's not how open-source works. And no one has emerged who has strong interest in doing
> so, and that's totally fine. It means it hasn't made any progress, but that's an indication that no
> one feels strongly enough about it that its risen to the top of their personal priority list so
> clearly doesn't need to make progress.
> 
> > Maybe not all of the ideas in it were unambiguously good
> > [1], but personally, I'm convinced at least some of them are, and I
> > don't think I'm alone in thinking that. Even if the excuse is that its
> > original champion wasn't humble enough, there's something wrong with
> > the process if there doesn't exist some other potential champion with
> > the right balance of humility, confidence, interest and time who could
> > have taken it over in that timeframe.
> 
> 
> No? Its not up to the community to find a champion for someone who wants a fork to happen. Either
> someone thinks its a good enough idea that they step up, or no one does. If no one does, then so be
> it. If the original proper (me, in this case) thought it was that important then its their
> responsibility to be the champion, no one else's.
> 
> > 2) Many will argue that CTV has already done steps (1) through (3) above:
> > certainly there's been an idea, it's been socialised through giving talks,
> > having discussion forums, having research workshops [2], documenting use
> > cases use cases; there's been a concrete implementation for years now,
> > with a test network that supports the proposed feature, and new tools
> > that demonstrate some of the proposed use cases, and while alternative
> > approaches have been suggested [3], none of them have even really made
> > it to step (2), let alone step (3).
> 
> 
> I don't really see how you can make this argument seriously. Honestly, if a soft-fork BIP only has
> one author on the list, then I'm not sure one can argue that step (3) has really been completed, and
> maybe not even step (2).
> 
> > So that leaves a few possibilities
> > to my mind:
> 
> > * CTV should be in step (4), and its lack of definition is a problem,
> > and trying the "deal with it when we get there" approach is precisely
> > what didn't work back in April.
> > 
> > * The evaluation process is too inconclusive: it should either be
> > saying "CTV is not good enough, fix these problems", or "CTV hasn't
> > sufficiently demonstrated its value/cost, work on X next", but it
> > isn't.
> > 
> > * Parts (2) to (3) are too hard, and that's preventing alternatives
> > from making progress, which in turn is preventing people from
> > being able to decide whether CTV is the superior approach, or some
> > alternative is.
> 
> 
> I think this is most of it, but its not that they're too hard, its that people are too busy. There
> seemed to be more positive feedback, for example, to Rusty's proposal, but being the champion for a
> soft-fork is a full-time job for months on end, and last I checked Rusty has a lightning
> implementation to maintain, which tends to be a more-than-full-time job already.
> 
> To my knowledge, no one but Jeremy has made any serious attempt at being the champion for a
> soft-fork since Taproot, and before that Segwit (if someone reading this who contributes to Core
> already wants to, and isn't sure how to, there's lots of people who would happily mentor you! I'm
> sure you can figure out who to reach out to!).
> 
> Matt
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[alicexbt]

Hi Michael,

> I agree with Matt. The less said about the "Aw shucks Jeremy didn't know that CTV didn't have community consensus at the time" [0] and "it was the lack of process that was the problem" the better. 

The linked gist cannot be taken seriously and I am not sure why you keep sharing it as some document to prove there was no technical consensus for BIP 119. Nadav has already mentioned this in the comments. If you care about community consensus, maybe feedback about the links in that gist should also be respected. There was chaos, misinformation and lot of drama on twitter. Some people that opposed CTV on twitter still have no clue what CTV actually does and a few were super enthusiastic because of the author for BIP 119.

> I'm not convinced by the name (bitcoin-inquisition, shedpaint, shedpaint, let's park that for the moment) but I do like the idea of signet having soft fork proposals enabled on it [1] whether that be CTV, APO etc and if that requires more of the signet code to be moved out of the Core repo so be it.

Good to see some positivity, finally. Because tx introspection aka covenants would help everyone involved in bitcoin. This idea of experimenting with soft forks on signet together with research and meetings suggested by Antoine should help in better evaluation phase with less drama, politics etc. and more technical discussions to reach a goal.

> I'm surprised more isn't being done on Liquid already with what possible future functionality is (and could be) enabled [2] there but maybe there is more happening than I'm aware of. 

1)Nobody uses Liquid. Signet has more activity than Liquid.
2)Testing something on Liquid will be completely different as its a separate blockchain with some similarities.

I have summarized a few other positives of testing soft forks on signet based on AJ's email:

a)Better evaluation
b)PR implementing soft fork could be reviewed and merged outside core
c)Testing on signet with pre-existing signet infrastructure
d)Can deploy multiple consensus changes so easier to compare alternative approaches (eg CTV vs ANYPREVOUT vs OP_TXHASH vs OP_TX, etc)
e)Pre-baked ability to abandon the soft fork
f)No need to regularly rebase consensus changes against bitcoin core's master branch

/dev/fd0

Sent with Proton Mail secure email.

------- Original Message -------
On Saturday, September 17th, 2022 at 3:53 PM, Michael Folkson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:


> I agree with Matt. The less said about the "Aw shucks Jeremy didn't know that CTV didn't have community consensus at the time" [0] and "it was the lack of process that was the problem" the better. If people don't care about lack of community consensus there is no process in a permissionless, open source community that can stop them or direct them down a more patient, productive path (I tried). I think it is a shame because I think (maybe I'm wrong) at least in the technical community there is an understanding that long term Bitcoin is far from finished in exhausting its potential and we do need people who will work on changes that we'll need in the long term.
> 
> There are a few interesting things in here though. I'm not convinced by the name (bitcoin-inquisition, shedpaint, shedpaint, let's park that for the moment) but I do like the idea of signet having soft fork proposals enabled on it [1] whether that be CTV, APO etc and if that requires more of the signet code to be moved out of the Core repo so be it. I'm surprised more isn't being done on Liquid already with what possible future functionality is (and could be) enabled [2] there but maybe there is more happening than I'm aware of. Protocols or use cases built out and demonstrated on signet (and/or Liquid/sidechains) seem an obvious stepping stone to me for convincing the community that it is potentially worth taking the chain split risk for a particular upgrade. It is a long slog to get community consensus on an upgrade (Taproot certainly was a slog) but I think the vast majority of us think Taproot was worth that slog and Bitcoin would be poorer today without it.
> 
> The Great Consensus Cleanup is an interesting example in that I get Matt doesn't have time to champion it or focus on it with his LDK commitments but I have no idea where it would rank on his long term priority list if he wasn't working on LDK. Similarly I have no idea what people who understand this evolving system much better than I do are thinking with regards to say adding new opcodes, sighash flags versus say waiting on Simplicity and possibly adding new functionality within that potential upgrade. For people like me who are extremely unlikely to propose their own consensus change(s) getting some signal on what to spend time digging into would be useful rather than second guessing what people are thinking. There is a danger that you flirt with perceived public roadmaps when possible authority figures stick their necks out and effectively say "I'm not in charge but in an imaginary world where I was this is my current thinking of the ordering in which we could improve this system
> long term. But this could change depending on x, y and z and possible upgrades are only ready when they're ready and they have community consensus." There is no way people don't play these exercises in their minds. I do, I just have very few answers :) I personally think APO is in prime position to improve Lightning channel state management with eltoo and if it enables some covenant schemes too that seems like an added bonus. On APO versus waiting for APO like functionality in Simplicity I have no idea. Work on APO/eltoo and Simplicity both seem to be progressing in parallel so the APO versus Simplicity discussion perhaps rests on whether people think certain covenants should only really be enabled once we have the security guarantees of Simplicity [3] (if at all).
> 
> Antoine's covenant R&D effort [4] seems really promising and I hope the shenanigans from earlier in the year don't put people off from engaging with that. Hopefully we can see more exploration, development and research in covenants (e.g. this excellent research paper "Bitcoin Covenants: Three Ways to Control the Future" [5]) and we can foster a community which has very high standards, is open to new ideas and new research but can avoid these months long resisting chain split fights. I think the discussion would be much more interesting and much more productive if people didn't have to think "If I express a view now it will be used to attack me personally later" or worse "If I express a view now it will be used to justify an upcoming chain split".
> 
> [0]: https://gist.github.com/michaelfolkson/352a503f4f9fc5de89af528d86a1b718
> [1]: https://bitcoin.stackexchange.com/questions/98642/can-we-experiment-on-signet-with-multiple-proposed-soft-forks-whilst-maintaining
> [2]: https://bitcoin.stackexchange.com/questions/109764/what-opcodes-are-supported-on-liquid-but-not-yet-on-bitcoin
> [3]: https://bitcoinops.org/en/topics/simplicity/
> [4]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-September/020912.html
> [5]: https://arxiv.org/pdf/2006.16714.pdf
> 
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
> 
> 
> ------- Original Message -------
> On Saturday, September 17th, 2022 at 09:39, Matt Corallo via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote:
> 
> 
> 
> > On 9/17/22 2:14 AM, Anthony Towns wrote:
> > 
> > > On Fri, Sep 16, 2022 at 12:46:53PM -0400, Matt Corallo via bitcoin-dev wrote:
> > > 
> > > > On 9/16/22 3:15 AM, Anthony Towns via bitcoin-dev wrote:
> > > > 
> > > > > As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
> > > > > in the year [0], the question of "how to successfully get soft fork
> > > > > ideas from concept to deployment" doesn't really have a good answer today.
> > > > > I strongly disagree with this.
> > > 
> > > Okay? "X is good" is obviously just a statement of opinion, so if you
> > > want to disagree, that's obviously allowed.
> > > 
> > > I also kind of feel like that's the least interesting paragraph in the
> > > entire email to talk further about; if you think the current answer's
> > > already good, then the rest of the mail's just about (hopefully) making
> > > it better, which would be worthwhile anyway?
> > 
> > No, I think its at least a good chunk of the "statement of problem". Yes, more testing is good, and
> > this project is a way to get that. Cool. But implying that lack of test frameworks is in any
> > material way part of the lack of movement on forks in Bitcoin I think is very wrong, so its worth
> > pointing out, whether the particular project is useful or not is separate.
> > 
> > > > Going back many, many years we've had many
> > > > discussions about fork process, and the parts people (historically) agreed
> > > > with tend to be:
> > > > (1) come up with an idea
> > > > (2) socialize the idea in the technical community, see if anyone comes up
> > > > with any major issues or can suggest better ideas which solve the same
> > > > use-cases in cleaner ways
> > > > (3) propose the concrete idea with a more well-defined strawman, socialize
> > > > that, get some kind of rough consensus in the loosely-defined, subjective,
> > > > "technical community" (ie just ask people and adapt to feedback until you
> > > > have found some kind of average of the opinions of people you, the
> > > > fork-champion, think are reasonably well-informed!).
> > > > (4) okay, admittedly beyond this is a bit less defined, but we can deal with it when we get there.
> > > > Turns out, the issue today is a lack of champions following steps 1-3, we
> > > > can debate what the correct answer is to step (4) once we actually have
> > > > people who want to be champions who are willing to (humbly) push an idea
> > > > forward towards rough agreement of the world of technical bitcoiners
> > > > (without which I highly doubt you'd ever see broader-community consensus).
> > > 
> > > Personally, I think this is easily refuted by contradiction.
> > > 
> > > 1) If we did have a good answer for how to progress a soft-fork, then
> > > the great consensus cleanup [0] would have made more progress over the
> > > past 3.5 years
> > 
> > No? Who is the champion for it? I haven't been. No one else is obliged to take up the reins and run
> > with it, that's not how open-source works. And no one has emerged who has strong interest in doing
> > so, and that's totally fine. It means it hasn't made any progress, but that's an indication that no
> > one feels strongly enough about it that its risen to the top of their personal priority list so
> > clearly doesn't need to make progress.
> > 
> > > Maybe not all of the ideas in it were unambiguously good
> > > [1], but personally, I'm convinced at least some of them are, and I
> > > don't think I'm alone in thinking that. Even if the excuse is that its
> > > original champion wasn't humble enough, there's something wrong with
> > > the process if there doesn't exist some other potential champion with
> > > the right balance of humility, confidence, interest and time who could
> > > have taken it over in that timeframe.
> > 
> > No? Its not up to the community to find a champion for someone who wants a fork to happen. Either
> > someone thinks its a good enough idea that they step up, or no one does. If no one does, then so be
> > it. If the original proper (me, in this case) thought it was that important then its their
> > responsibility to be the champion, no one else's.
> > 
> > > 2) Many will argue that CTV has already done steps (1) through (3) above:
> > > certainly there's been an idea, it's been socialised through giving talks,
> > > having discussion forums, having research workshops [2], documenting use
> > > cases use cases; there's been a concrete implementation for years now,
> > > with a test network that supports the proposed feature, and new tools
> > > that demonstrate some of the proposed use cases, and while alternative
> > > approaches have been suggested [3], none of them have even really made
> > > it to step (2), let alone step (3).
> > 
> > I don't really see how you can make this argument seriously. Honestly, if a soft-fork BIP only has
> > one author on the list, then I'm not sure one can argue that step (3) has really been completed, and
> > maybe not even step (2).
> > 
> > > So that leaves a few possibilities
> > > to my mind:
> > 
> > > * CTV should be in step (4), and its lack of definition is a problem,
> > > and trying the "deal with it when we get there" approach is precisely
> > > what didn't work back in April.
> > > 
> > > * The evaluation process is too inconclusive: it should either be
> > > saying "CTV is not good enough, fix these problems", or "CTV hasn't
> > > sufficiently demonstrated its value/cost, work on X next", but it
> > > isn't.
> > > 
> > > * Parts (2) to (3) are too hard, and that's preventing alternatives
> > > from making progress, which in turn is preventing people from
> > > being able to decide whether CTV is the superior approach, or some
> > > alternative is.
> > 
> > I think this is most of it, but its not that they're too hard, its that people are too busy. There
> > seemed to be more positive feedback, for example, to Rusty's proposal, but being the champion for a
> > soft-fork is a full-time job for months on end, and last I checked Rusty has a lightning
> > implementation to maintain, which tends to be a more-than-full-time job already.
> > 
> > To my knowledge, no one but Jeremy has made any serious attempt at being the champion for a
> > soft-fork since Taproot, and before that Segwit (if someone reading this who contributes to Core
> > already wants to, and isn't sure how to, there's lots of people who would happily mentor you! I'm
> > sure you can figure out who to reach out to!).
> > 
> > Matt
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Michael Folkson]

Hi alicexbt

> Good to see some positivity, finally.

I had enthusiasm for this concept of enabling proposed soft fork functionality on signet 2 years ago [0]. Nothing has changed, still enthusiastic :) Not enthusiastic about the months wasted on unnecessary contentious soft fork drama since but can't change the past.

> 1)Nobody uses Liquid. Signet has more activity than Liquid.
  2)Testing something on Liquid will be completely different as its 
  a separate blockchain with some similarities.

Perhaps you should take your own advice with regards to positivity (or at least have more of an open mind) with regards Liquid and sidechains. Signet Bitcoin are totally free [1] and experimentation doesn't ever result in loss of real monetary value so you would expect more experimentation on signet versus Liquid long term. However, building protocols and prototypes with real monetary value is a step up from doing so with worthless signet coins. So I don't really see them as direct competitors. Some things take a lot longer to come to fruition than others but the original vision [2] of sidechains still makes perfect sense to me. Competing sets of consensus rules aren't possible on a single mainnet blockchain. Hence you either go the sidechain(-like) route or you go the altcoin route if you want to take the step up from signet/testnet and start using real monetary value. I much prefer the sidechain model to the altcoin route myself. Especially when in say vaults you do want the equivalent of Bitcoin to be locked up rather than a more volatile altcoin.

Thanks
Michael

[0]: https://bitcoin.stackexchange.com/questions/98642/can-we-experiment-on-signet-with-multiple-proposed-soft-forks-whilst-maintaining
[1]: https://signetfaucet.com/
[2]: https://blockstream.com/sidechains.pdf

--
Michael Folkson
Email: michaelfolkson at protonmail.com
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3


------- Original Message -------
On Sunday, September 18th, 2022 at 13:27, alicexbt <alicexbt@protonmail.com> wrote:


> Hi Michael,
> 
> > I agree with Matt. The less said about the "Aw shucks Jeremy didn't know that CTV didn't have community consensus at the time" 0 and "it was the lack of process that was the problem" the better.
> 
> 
> The linked gist cannot be taken seriously and I am not sure why you keep sharing it as some document to prove there was no technical consensus for BIP 119. Nadav has already mentioned this in the comments. If you care about community consensus, maybe feedback about the links in that gist should also be respected. There was chaos, misinformation and lot of drama on twitter. Some people that opposed CTV on twitter still have no clue what CTV actually does and a few were super enthusiastic because of the author for BIP 119.
> 
> > I'm not convinced by the name (bitcoin-inquisition, shedpaint, shedpaint, let's park that for the moment) but I do like the idea of signet having soft fork proposals enabled on it 1 whether that be CTV, APO etc and if that requires more of the signet code to be moved out of the Core repo so be it.
> 
> 
> Good to see some positivity, finally. Because tx introspection aka covenants would help everyone involved in bitcoin. This idea of experimenting with soft forks on signet together with research and meetings suggested by Antoine should help in better evaluation phase with less drama, politics etc. and more technical discussions to reach a goal.
> 
> > I'm surprised more isn't being done on Liquid already with what possible future functionality is (and could be) enabled 2 there but maybe there is more happening than I'm aware of.
> 
> 
> 1)Nobody uses Liquid. Signet has more activity than Liquid.
> 2)Testing something on Liquid will be completely different as its a separate blockchain with some similarities.
> 
> I have summarized a few other positives of testing soft forks on signet based on AJ's email:
> 
> a)Better evaluation
> b)PR implementing soft fork could be reviewed and merged outside core
> c)Testing on signet with pre-existing signet infrastructure
> d)Can deploy multiple consensus changes so easier to compare alternative approaches (eg CTV vs ANYPREVOUT vs OP_TXHASH vs OP_TX, etc)
> e)Pre-baked ability to abandon the soft fork
> f)No need to regularly rebase consensus changes against bitcoin core's master branch
> 
> /dev/fd0
> 
> Sent with Proton Mail secure email.
> 
> ------- Original Message -------
> On Saturday, September 17th, 2022 at 3:53 PM, Michael Folkson via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote:
> 
> 
> 
> > I agree with Matt. The less said about the "Aw shucks Jeremy didn't know that CTV didn't have community consensus at the time" 0 and "it was the lack of process that was the problem" the better. If people don't care about lack of community consensus there is no process in a permissionless, open source community that can stop them or direct them down a more patient, productive path (I tried). I think it is a shame because I think (maybe I'm wrong) at least in the technical community there is an understanding that long term Bitcoin is far from finished in exhausting its potential and we do need people who will work on changes that we'll need in the long term.
> > 
> > There are a few interesting things in here though. I'm not convinced by the name (bitcoin-inquisition, shedpaint, shedpaint, let's park that for the moment) but I do like the idea of signet having soft fork proposals enabled on it 1 whether that be CTV, APO etc and if that requires more of the signet code to be moved out of the Core repo so be it. I'm surprised more isn't being done on Liquid already with what possible future functionality is (and could be) enabled 2 there but maybe there is more happening than I'm aware of. Protocols or use cases built out and demonstrated on signet (and/or Liquid/sidechains) seem an obvious stepping stone to me for convincing the community that it is potentially worth taking the chain split risk for a particular upgrade. It is a long slog to get community consensus on an upgrade (Taproot certainly was a slog) but I think the vast majority of us think Taproot was worth that slog and Bitcoin would be poorer today without it.
> > 
> > The Great Consensus Cleanup is an interesting example in that I get Matt doesn't have time to champion it or focus on it with his LDK commitments but I have no idea where it would rank on his long term priority list if he wasn't working on LDK. Similarly I have no idea what people who understand this evolving system much better than I do are thinking with regards to say adding new opcodes, sighash flags versus say waiting on Simplicity and possibly adding new functionality within that potential upgrade. For people like me who are extremely unlikely to propose their own consensus change(s) getting some signal on what to spend time digging into would be useful rather than second guessing what people are thinking. There is a danger that you flirt with perceived public roadmaps when possible authority figures stick their necks out and effectively say "I'm not in charge but in an imaginary world where I was this is my current thinking of the ordering in which we could improve this system
> > long term. But this could change depending on x, y and z and possible upgrades are only ready when they're ready and they have community consensus." There is no way people don't play these exercises in their minds. I do, I just have very few answers :) I personally think APO is in prime position to improve Lightning channel state management with eltoo and if it enables some covenant schemes too that seems like an added bonus. On APO versus waiting for APO like functionality in Simplicity I have no idea. Work on APO/eltoo and Simplicity both seem to be progressing in parallel so the APO versus Simplicity discussion perhaps rests on whether people think certain covenants should only really be enabled once we have the security guarantees of Simplicity 3 (if at all).
> > 
> > Antoine's covenant R&D effort 4 seems really promising and I hope the shenanigans from earlier in the year don't put people off from engaging with that. Hopefully we can see more exploration, development and research in covenants (e.g. this excellent research paper "Bitcoin Covenants: Three Ways to Control the Future" 5) and we can foster a community which has very high standards, is open to new ideas and new research but can avoid these months long resisting chain split fights. I think the discussion would be much more interesting and much more productive if people didn't have to think "If I express a view now it will be used to attack me personally later" or worse "If I express a view now it will be used to justify an upcoming chain split".
> > 
> > --
> > Michael Folkson
> > Email: michaelfolkson at protonmail.com
> > Keybase: michaelfolkson
> > PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
> > 
> > ------- Original Message -------
> > On Saturday, September 17th, 2022 at 09:39, Matt Corallo via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote:
> > 
> > > On 9/17/22 2:14 AM, Anthony Towns wrote:
> > > 
> > > > On Fri, Sep 16, 2022 at 12:46:53PM -0400, Matt Corallo via bitcoin-dev wrote:
> > > > 
> > > > > On 9/16/22 3:15 AM, Anthony Towns via bitcoin-dev wrote:
> > > > > 
> > > > > > As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
> > > > > > in the year 0, the question of "how to successfully get soft fork
> > > > > > ideas from concept to deployment" doesn't really have a good answer today.
> > > > > > I strongly disagree with this.
> > > > 
> > > > Okay? "X is good" is obviously just a statement of opinion, so if you
> > > > want to disagree, that's obviously allowed.
> > > > 
> > > > I also kind of feel like that's the least interesting paragraph in the
> > > > entire email to talk further about; if you think the current answer's
> > > > already good, then the rest of the mail's just about (hopefully) making
> > > > it better, which would be worthwhile anyway?
> > > 
> > > No, I think its at least a good chunk of the "statement of problem". Yes, more testing is good, and
> > > this project is a way to get that. Cool. But implying that lack of test frameworks is in any
> > > material way part of the lack of movement on forks in Bitcoin I think is very wrong, so its worth
> > > pointing out, whether the particular project is useful or not is separate.
> > > 
> > > > > Going back many, many years we've had many
> > > > > discussions about fork process, and the parts people (historically) agreed
> > > > > with tend to be:
> > > > > (1) come up with an idea
> > > > > (2) socialize the idea in the technical community, see if anyone comes up
> > > > > with any major issues or can suggest better ideas which solve the same
> > > > > use-cases in cleaner ways
> > > > > (3) propose the concrete idea with a more well-defined strawman, socialize
> > > > > that, get some kind of rough consensus in the loosely-defined, subjective,
> > > > > "technical community" (ie just ask people and adapt to feedback until you
> > > > > have found some kind of average of the opinions of people you, the
> > > > > fork-champion, think are reasonably well-informed!).
> > > > > (4) okay, admittedly beyond this is a bit less defined, but we can deal with it when we get there.
> > > > > Turns out, the issue today is a lack of champions following steps 1-3, we
> > > > > can debate what the correct answer is to step (4) once we actually have
> > > > > people who want to be champions who are willing to (humbly) push an idea
> > > > > forward towards rough agreement of the world of technical bitcoiners
> > > > > (without which I highly doubt you'd ever see broader-community consensus).
> > > > 
> > > > Personally, I think this is easily refuted by contradiction.
> > > > 
> > > > 1) If we did have a good answer for how to progress a soft-fork, then
> > > > the great consensus cleanup 0 would have made more progress over the
> > > > past 3.5 years
> > > 
> > > No? Who is the champion for it? I haven't been. No one else is obliged to take up the reins and run
> > > with it, that's not how open-source works. And no one has emerged who has strong interest in doing
> > > so, and that's totally fine. It means it hasn't made any progress, but that's an indication that no
> > > one feels strongly enough about it that its risen to the top of their personal priority list so
> > > clearly doesn't need to make progress.
> > > 
> > > > Maybe not all of the ideas in it were unambiguously good
> > > > 1, but personally, I'm convinced at least some of them are, and I
> > > > don't think I'm alone in thinking that. Even if the excuse is that its
> > > > original champion wasn't humble enough, there's something wrong with
> > > > the process if there doesn't exist some other potential champion with
> > > > the right balance of humility, confidence, interest and time who could
> > > > have taken it over in that timeframe.
> > > 
> > > No? Its not up to the community to find a champion for someone who wants a fork to happen. Either
> > > someone thinks its a good enough idea that they step up, or no one does. If no one does, then so be
> > > it. If the original proper (me, in this case) thought it was that important then its their
> > > responsibility to be the champion, no one else's.
> > > 
> > > > 2) Many will argue that CTV has already done steps (1) through (3) above:
> > > > certainly there's been an idea, it's been socialised through giving talks,
> > > > having discussion forums, having research workshops 2, documenting use
> > > > cases use cases; there's been a concrete implementation for years now,
> > > > with a test network that supports the proposed feature, and new tools
> > > > that demonstrate some of the proposed use cases, and while alternative
> > > > approaches have been suggested 3, none of them have even really made
> > > > it to step (2), let alone step (3).
> > > 
> > > I don't really see how you can make this argument seriously. Honestly, if a soft-fork BIP only has
> > > one author on the list, then I'm not sure one can argue that step (3) has really been completed, and
> > > maybe not even step (2).
> > > 
> > > > So that leaves a few possibilities
> > > > to my mind:
> > > 
> > > > * CTV should be in step (4), and its lack of definition is a problem,
> > > > and trying the "deal with it when we get there" approach is precisely
> > > > what didn't work back in April.
> > > > 
> > > > * The evaluation process is too inconclusive: it should either be
> > > > saying "CTV is not good enough, fix these problems", or "CTV hasn't
> > > > sufficiently demonstrated its value/cost, work on X next", but it
> > > > isn't.
> > > > 
> > > > * Parts (2) to (3) are too hard, and that's preventing alternatives
> > > > from making progress, which in turn is preventing people from
> > > > being able to decide whether CTV is the superior approach, or some
> > > > alternative is.
> > > 
> > > I think this is most of it, but its not that they're too hard, its that people are too busy. There
> > > seemed to be more positive feedback, for example, to Rusty's proposal, but being the champion for a
> > > soft-fork is a full-time job for months on end, and last I checked Rusty has a lightning
> > > implementation to maintain, which tends to be a more-than-full-time job already.
> > > 
> > > To my knowledge, no one but Jeremy has made any serious attempt at being the champion for a
> > > soft-fork since Taproot, and before that Segwit (if someone reading this who contributes to Core
> > > already wants to, and isn't sure how to, there's lots of people who would happily mentor you! I'm
> > > sure you can figure out who to reach out to!).
> > > 
> > > Matt
> > > _______________________________________________
> > > bitcoin-dev mailing list
> > > bitcoin-dev@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> > 
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Antoine Riard]

[-- Attachment #1: Type: text/plain, Size: 14906 bytes --]

Hi AJ,

Thanks to setup a new laboratory for consensus upgrades experiment! Idea
was exposed during the last LN Summit, glad to see there is a useful fork
now.

While I think one of the problem particular in the current stagnation about
consensus upgrades has been well scoped by your proposal, namely on
formalizing the thorough analysis process to which an upgrade proposal
should be subject too, there are more issues or bounds to wonder on, at
least from my perspective.

Laying out fine-grained stages of the development process (research,
development, evaluation, deployment) sounds compelling to bring clarity to
everyone. However, I doubt it captures well the more realistic, chaotic
process from which new Bitcoin ideas and techniques are emerging. In
practice, consensus upgrades, akin to the sustenance of new scientific
theories or engineering principles in the wider creative areas of life, are
following an uncertain path of hazardous steps: seminal half-baked
intuitions, whiteboard modeling, code or quantitative experiments, loose
set of ideas pollination, peers feedbacks integration, etc before to mature
in some solidified proposals.

Said succinctly, in the genesis of creative ideas, evaluation doesn't
happen at a single clear point but all along the idea lifetime, where this
evaluation is as much done by the original author than its peers and a
wider audience. Sometimes really formally, e.g in academics with PhD thesis
defense. For Bitcoin, rather than to "declare" on the when and where
upgrades evaluation should happen once for all, I think a more open
evaluation process we can carry on is gathering and maintaining the factual
material and reasoning frameworks around solidified proposals, on which
each community stakeholder individually can assign a grounded judgement.
Those judgments are likely sources of new refinement of the upgrades
themselves.

Under that perspective, I believe a functional upgrades experimentation
platform as proposed by bitcoin-inquisition is very valuable, as it should
allow upgrades "champions" (CTV, ANYPREVOUT, TLUV, "fraud proofs" ops
primitives, etc) to loop faster in the R&D cycles, raise earlier awareness
on their work existence and as it's all open to assemble team around their
proposals. (Effectively, covenants upgrades and their associated use-cases
offered so much complexity that it's becoming less and less a one-man
job...).

I would still expose a concern to not downgrade in the pure empiricism in
matter of consensus upgrades. I.e, slowly emerging the norm of a working
prototype running on bitcoin-inquisition` as a determining factor of the
soundness of a proposal. E.g with "upgrading lightning to support eltoo", a
running e2e won't save us to think the thousands variants of pinnings, the
game-theory soundness of a eltoo as mechanism in face of congestions, the
evolvability of APO with more known upgrades proposals or the
implementation complexity of a fully fleshed-out state machine and more
questions.

That said, a e2e implementation, partial or complete, would at least make
the serious analysis process easier. Moreover, the benefit of having e2e
implems runnable by everyone on bitcoin-inquisition would likely lower the
bar to have independent consensus upgrade analysis, likely a source of new
relevant feedback.

I can only share the sentiment expressed that this alternative open
approach of consensus changes avoids the gradual establishment of a trusted
group, even informal. In the past, to the best of my knowledge, most of the
substance of the Taproot softfork daily development happened on
semi-offline communication channels and the strong design rational
decisions at CoreDev editions. While not discrediting the high-quality
feedback than one can gather during those types of in-person engineering
meetings, for neutrality and openness of the Bitcoin upgrades process it
could be great to only consider them as source of feedbacks and move
progressively the crux of the upgrades R&D process online, open to anyone
interested. Moreover, it would bind more adequately the reality of a
growing development ecosystem, where we have to deal with an increasing
diversity of technical, social and geographical community stakeholders. I
acknowledge there is a hard challenge to maintain high-signal, low-noise
online communication channels and spaces about context-rich issues like
upgrades, however that might be the type of challenge we have to solve if
we care about everyone using Bitcoin to truly be peers. At least my 2 sats.

About the risk of latent centralization of global default signets
miners/bitcoin-inquisition maintainers, I don't think I'm worried about it.
With time, I would guess we might have multiple experimental signets with
different series of patches, as some patches might invalidate the
observations of another upgrade. E,g if one implements the "weird" ideas
about changes in the block reward issuance schedule discussed during the
summer, another one might not want "noise" interferences with new
fee-bumping primitives as the miner incentives are modified. About the
bitcoin-inquisition fork maintainers themselves becoming gatekeepers of
consensus upgrades changes, best we can do is maintain high-quality
documentation and knowledge base to lower the forking cost of the platform
for any community stakeholder.

Anyway, I hold the belief that the more initiatives we see to modernize the
"consensus-upgrades" production factory in order to scale with the current
dimensions of the Bitcoin ecosystem, better we're. I hope the upcoming
Contracting Primitives WG will be able to document and discuss some of the
relevant experiments run on bitcoin-inquisition. Time and work will tell
how they fit all together, where they complement each other and synergies
that are nurtured.

Speaking for myself, looking forward to experimenting with CoinPool code
components on bitcoin-inquistion in the future!

Best,
Antoine

Le ven. 16 sept. 2022 à 03:16, Anthony Towns via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> a écrit :

> Subhead: "Nobody expects a Bitcoin Inquistion? C'mon man, *everyone*
> expects a Bitcoin Inquisition."
>
> As we've seen from the attempt at a CHECKTEMPLATEVERIFY activation earlier
> in the year [0], the question of "how to successfully get soft fork
> ideas from concept to deployment" doesn't really have a good answer today.
>
> Obviously, a centralised solution to this problem exists: we could
> establish a trusted group, perhaps one containing devs, industry
> representatives, investors, etc, have them review proposals and their
> implementations, and follow their lead when they decide that a proposal
> has met their standards and should be widely deployed. Some might even
> say "sipa is precisely that group". The problem with having a group of
> that nature isn't one of effectiveness, but rather that they are then
> vulnerable to pressure and corruption, which isn't desirable if we want
> everyone using Bitcoin to truly be peers, and often isn't desirable for
> the prospective members of the group either. So that's not something we
> should want people to volunteer for, nor is it a duty we should thrust
> on people. Or at least, that's my opinion, anyway.
>
> I think any alternative approach to doing consensus changes (while
> avoiding a chain split) has to look something like this:
>
>  * propose an idea (research phase)
>  * implement the idea (development phase)
>  * demonstrate the idea is worthwhile (evaluation phase)
>  * once everyone is convinced, activate (deployment phase)
>
> Without an evaluation phase that is thorough enough to convince (almost)
> everyone, I think deployment becomes controversial and perhaps effectively
> impossible (at least without some trusted leadership group). But with an
> evaluation phase that demonstrates to everyone who's interested that the
> proposal has actual value, minimal cost and no risk, I think activation
> could be fairly easy and straightforward.
>
> I contend that the most significant problem we have is in the "evaluation
> phase". How do you convince enough people that a change is sufficiently
> beneficial to justify the risk of messing with their money? If you're
> only trying to convince a few experts, then perhaps you can do that with
> papers and talks; but limiting the evaluation to only a few experts is
> effectively just falling back to the centralised approach.
>
> So I think that means that part of the "evaluation phase" should involve
> implementing real systems on top of the proposed change, so that you
> can demonstrate real value from the change. It's easy to say that
> "CTV can enable vaults" or "CTV can make opening a lightning channel
> non-interactive" -- but it's harder to go from saying something
> is possible to actually making it happen, so, at least to me, it
> seems reasonable to be skeptical of people claiming benefits without
> demonstrating they're achievable in practice.
>
> I contend the easiest way we could make it easy to demonstrate a soft
> fork working as designed is to deploy it on the default global signet,
> essentially as soon as it has a fully specified proposal and a reasonably
> high-quality implementation.
>
> The problem with that idea is that creates a conundrum: you can't activate
> a soft fork on the default signet without first merging the code into
> bitcoin core, you can't merge the code into bitcoin core until it's been
> fully evaluated, and the way you evaluate it is by activating it on the
> default signet?
>
> I think the weakest link in that loop is the first one: what if we did
> activate soft forks on the default signet prior to the code being merged
> into core? To that end, I'm proposing a fork of core that I'm calling
> "bitcoin-inquisition", with the idea that it branches from stable
> releases of core and adds support for proposed changes to consensus
> (CTV, ANYPREVOUT, TLUV, OP_CAT, etc...) and potentially also relay
> policy (relay changes are often implied by consensus changes, but also
> potentially things like package relay).
>
>   https://github.com/bitcoin-inquisition/bitcoin/wiki
>   https://github.com/bitcoin-inquisition/bitcoin/pulls
>
> The idea being that if you're trying to work on "upgrading lightning
> to support eltoo", you can iterate through changes needed to consensus
> (via bitcoin-inquisition) and client apps (cln, lnd, eclair etc), while
> testing them in public (on signet) and having any/all the pre-existing
> signet infrastructure available (faucets, explorers etc) without having
> to redeploy it yourself. Having multiple consensus changes deployed in
> one place also seems like it might make it easier to compare alternative
> approaches (eg CTV vs ANYPREVOUT vs OP_TXHASH vs OP_TX, etc).
>
> So that's the concept. For practical purposes, I haven't yet merged
> either CTV or APO support into the bitcoin-inquisition 23.0 branch yet,
> and before actually mining blocks I want to make the signet miner able
> to automatically detect/recover if the bitcoin-inquisition node either
> crashes or starts producing incompatible blocks.
>
> Anyway, I wanted to post the idea publicly, both to give folks an
> opportunity to poke holes in the idea, or to suggest any further
> improvements or otherwise do any review before the CTV and APO patches
> get merged.
>
> Some other details that may be of interest.
>
> The biggest challenge with soft forks and the idea of "iterating
> through changes" is that making improvements can create a hard fork,
> which then forces everyone running old software to update, which can be
> pretty inconvenient, especially if you don't actually care about that
> change. Since signet (and regtest) mining is effectively permissioned,
> we can avoid that problem by having all these proposed soft forks
> come with a pre-baked ability to abandon the soft fork (much as David
> Harding described in [1]). Once a soft fork is abandoned, it can either
> be ignored forever (and later versions of the software can not include
> the code to enforce it at all), or it can be replaced by a new version
> of the soft fork.
>
> Another benefit that comes from signet chains being permissioned is
> that miners can be expected to coordinate upgrading out of band, so
> there is no need for a 90% signalling threshold. Instead, activation
> (and abandonment) of a soft fork can be triggered by a single block
> signalling. That further means there is no need for any individual
> block to signal for multiple forks, and instead of having 29 different
> signals, we can instead easily have up to 2**29. I've chosen to make
> the standard signal have 16 bits for specifying a bip number (0-65535)
> and 8 bits for specifying a version of that bip, which seems like it
> should be more than enough at least for a while. More details at [2].
>
> I'm basing bitcoin-inquisition solely off stable releases. This is partly
> because it can be annoying to constantly rebase consensus changes aginst
> bitcoin core's master branch, but also I think it might help consensus
> changes be easily backported once they pass the "evaluation phase"
> and move into the "deployment phase".
>
> I'm not sure what level of code quality PRs should have before being
> merged into bitcoin-inquisition. I think CTV is plenty good enough,
> but I'm not sure about APO, particularly its test coverage. If you want
> to influence what becomes the tradition here, contributing a review,
> or posting patches against the upsteam branch might be a good start?
>
> Does this make the global default signet miners, or perhaps the
> bitcoin-inquisition maintainers the "trusted group" that we want to
> avoid? Hopefully not -- anyone can run their own fork or do their own
> fork of bitcoin core, so if the miners/maintainers start trying to
> arbitrarily block proposals they can be worked around without too much
> hassle. And since they're clearly separate from any of the actions that
> need to be taken for actual deployment once activation is complete,
> they shouldn't have any ability to unduly promote fork proposals that
> people aren't fully satisfied are ready for deployment.
>
> Cheers,
> aj
>
> [0]
> https://bitcoinops.org/en/newsletters/2022/04/27/#discussion-about-activating-ctv
>
> [1]
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-April/020242.html
>
> [2]
> https://github.com/bitcoin-inquisition/bitcoin/wiki/Heretical-Deployments
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 16619 bytes --]

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Anthony Towns]

On Sun, Sep 18, 2022 at 02:47:38PM -0400, Antoine Riard via bitcoin-dev wrote:
> Said succinctly, in the genesis of creative ideas, evaluation doesn't
> happen at a single clear point but all along the idea lifetime, where this
> evaluation is as much done by the original author than its peers and a
> wider audience.

Sure. I definitely didn't mean to imply a waterfall development model,
or that the phases wouldn't overlap etc.

> I would still expose a concern to not downgrade in the pure empiricism in
> matter of consensus upgrades. I.e, slowly emerging the norm of a working
> prototype running on bitcoin-inquisition` as a determining factor of the
> soundness of a proposal. E.g with "upgrading lightning to support eltoo", a
> running e2e won't save us to think the thousands variants of pinnings, the
> game-theory soundness of a eltoo as mechanism in face of congestions, the
> evolvability of APO with more known upgrades proposals or the
> implementation complexity of a fully fleshed-out state machine and more
> questions.

I agree here; but I think not doing prototypes also hinders thinking
about all the thousands of details in a fork. It's easy to handwave
details away when describing things on a whiteboard; and only realise
they're trickier than you thought when you go to implement things.

> E,g if one implements the "weird" ideas
> about changes in the block reward issuance schedule discussed during the
> summer, another one might not want "noise" interferences with new
> fee-bumping primitives as the miner incentives are modified. 

(I don't think "miner incentives" are really something that can be
investigated on signet. You can assume how miners will respond to
incentives and program the mining software to act that way; but there's
no competitive pressure in signet mining so I don't think that really
demonstrates anything very much. Likewise, there's much less demand for
blockspace on signet than on mainnet, so it's probably hard to experiment
with "fee incentives" too)

> I hope the upcoming
> Contracting Primitives WG will be able to document and discuss some of the
> relevant experiments run on bitcoin-inquisition. 

Likewise.

(Lots trimmed due to either agreeing with it or having nothing to add)

Cheers,
aj

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Michael Folkson]

I've given this some extra thought and discussed with others who may later chime in on this thread. I'm now convinced this should be done on a custom public signet rather than on the default signet. SegWit was added to a new testnet (Segnet) for testing rather than the pre-existing testnet and I think future soft fork proposals should follow a similar approach.

Even if there is community consensus on what soft fork proposals should be added to the default signet today (which may or may not be case) I find it highly unlikely this will always be the case. We then get into the situation where the block signers (currently AJ and Kalle) are the gatekeepers on what soft fork proposals are added. The default signet is directly supported with the -signet flag in Bitcoin Core. Even if we are moving the proposed soft fork code to an external repo (to avoid it being merged into Core prematurely) it is still determining what soft forks are accessible from the signet flag in Bitcoin Core. I don't think it is fair on the signet block signers to put them in that position and I don't think it is wise to put other Bitcoin Core contributors/maintainers in the position of having to defend why some proposed soft forks are accessible on the default signet while others aren't.

The default signet was a long term project to address the unreliability and weaknesses of testnet. Many default signet users won't be interested in testing soft fork proposals and it is not reasonable for them to be subject to a stalling or forked blockchain because changes to a soft fork proposal or a buggy soft fork proposal pushed to the default signet makes previous valid/invalid transactions invalid/valid. If they want to test proposed soft forks on a custom signet they are opting in to possible disruption rather than it being forced upon them.

By focusing on custom signets rather than the default signet it also allows for more experimentation. Don't like the choices of which soft fork proposals have been added to bitcoin-inquisition? Set up your own custom signet with a different set of soft fork proposals and get users for your custom signet on a level playing field to bitcoin-inquisition. A soft fork proposal is found to be strictly inferior to another soft fork proposal? Just spin down the custom signet with that inferior soft fork proposal on it without impacting default signet users.

So TL;DR still enthusiastic about this concept. Just with a strong preference that it is done on a custom signet rather than on the default signet.

Thanks
Michael

--
Michael Folkson
Email: michaelfolkson at protonmail.com
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3


------- Original Message -------
On Monday, September 19th, 2022 at 11:05, Anthony Towns via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:


> On Sun, Sep 18, 2022 at 02:47:38PM -0400, Antoine Riard via bitcoin-dev wrote:
> 
> > Said succinctly, in the genesis of creative ideas, evaluation doesn't
> > happen at a single clear point but all along the idea lifetime, where this
> > evaluation is as much done by the original author than its peers and a
> > wider audience.
> 
> 
> Sure. I definitely didn't mean to imply a waterfall development model,
> or that the phases wouldn't overlap etc.
> 
> > I would still expose a concern to not downgrade in the pure empiricism in
> > matter of consensus upgrades. I.e, slowly emerging the norm of a working
> > prototype running on bitcoin-inquisition` as a determining factor of the
> > soundness of a proposal. E.g with "upgrading lightning to support eltoo", a
> > running e2e won't save us to think the thousands variants of pinnings, the
> > game-theory soundness of a eltoo as mechanism in face of congestions, the
> > evolvability of APO with more known upgrades proposals or the
> > implementation complexity of a fully fleshed-out state machine and more
> > questions.
> 
> 
> I agree here; but I think not doing prototypes also hinders thinking
> about all the thousands of details in a fork. It's easy to handwave
> details away when describing things on a whiteboard; and only realise
> they're trickier than you thought when you go to implement things.
> 
> > E,g if one implements the "weird" ideas
> > about changes in the block reward issuance schedule discussed during the
> > summer, another one might not want "noise" interferences with new
> > fee-bumping primitives as the miner incentives are modified.
> 
> 
> (I don't think "miner incentives" are really something that can be
> investigated on signet. You can assume how miners will respond to
> incentives and program the mining software to act that way; but there's
> no competitive pressure in signet mining so I don't think that really
> demonstrates anything very much. Likewise, there's much less demand for
> blockspace on signet than on mainnet, so it's probably hard to experiment
> with "fee incentives" too)
> 
> > I hope the upcoming
> > Contracting Primitives WG will be able to document and discuss some of the
> > relevant experiments run on bitcoin-inquisition.
> 
> 
> Likewise.
> 
> (Lots trimmed due to either agreeing with it or having nothing to add)
> 
> Cheers,
> aj
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[alicexbt]

Hi Michael,

> We then get into the situation where the block signers (currently AJ and Kalle) are the gatekeepers on what soft fork proposals are added.

Things that could solve the gatekeeping issue:

1) Adding more maintainers that are experienced enough to review consensus code. 
2) Every soft fork that is implemented on signet should be discussed on mailing list before merging the pull request.
3) Pull request that implements a soft fork should have at least 2 ACKs from the maintainers, 3 ACKs from developers who have either authored or reviewed enough consensus related pull requests in bitcoin core and 1 ACK from maintainers of other implementations (knots, btcd, libbitcoin, bcoin etc.)
4) Every technical NACK from any developer or user in the pull request should be resolved/answered before merging the pull request.

This was not the case in [implementing BIP][1] 119 however it could be used in future or something similar that everyone agrees upon including AJ and Kalle. Pull request implementing BIP 119 in bitcoin core is already reviewed by lot of developers and I think AJ found a [bug][2] recently.

Gatekeeping at several levels already exists in bitcoin development and difficult to solve. If some developers believe a soft fork should have been implemented on global signet but it was not, there is always the possibility of running custom signet with certain trade-offs.

> The default signet is directly supported with the -signet flag in Bitcoin Core. Even if we are moving the proposed soft fork code to an external repo (to avoid it being merged into Core prematurely) it is still determining what soft forks are accessible from the signet flag in Bitcoin Core. I don't think it is fair on the signet block signers to put them in that position and I don't think it is wise to put other Bitcoin Core contributors/maintainers in the position of having to defend why some proposed soft forks are accessible on the default signet while others aren't.

Mainnet and Testnet have already been [removed][3] from the 'bitcoin-inquisition/bitcoin' repository, and signet in bitcoin core is largely used by developers or power users, thus it should not be a problem. Signet could also be removed from bitcoin core binaries that are released regularly while being available if built from source.

Since signet coins have no value, utilizing this chain to experiment with soft forks will only help the bitcoin development process. If we can't even agree to implement something on a network used for testing then it will be nearly impossible to do any soft forks on mainnet. This experiment has several advantages. We can implement many consensus changes and compare alternatives in a better way. Pre-baked ability to abandon the soft fork isn't implemented yet AFAIK, however that could further improve things.


[1]: https://github.com/bitcoin-inquisition/bitcoin/pull/3
[2]: https://github.com/bitcoin/bitcoin/pull/21702#discussion_r979273387
[3]: https://github.com/bitcoin-inquisition/bitcoin/pull/1

/dev/fd0

Sent with Proton Mail secure email.


------- Original Message -------
On Wednesday, September 28th, 2022 at 5:18 PM, Michael Folkson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:


> I've given this some extra thought and discussed with others who may later chime in on this thread. I'm now convinced this should be done on a custom public signet rather than on the default signet. SegWit was added to a new testnet (Segnet) for testing rather than the pre-existing testnet and I think future soft fork proposals should follow a similar approach.
> 
> Even if there is community consensus on what soft fork proposals should be added to the default signet today (which may or may not be case) I find it highly unlikely this will always be the case. We then get into the situation where the block signers (currently AJ and Kalle) are the gatekeepers on what soft fork proposals are added. The default signet is directly supported with the -signet flag in Bitcoin Core. Even if we are moving the proposed soft fork code to an external repo (to avoid it being merged into Core prematurely) it is still determining what soft forks are accessible from the signet flag in Bitcoin Core. I don't think it is fair on the signet block signers to put them in that position and I don't think it is wise to put other Bitcoin Core contributors/maintainers in the position of having to defend why some proposed soft forks are accessible on the default signet while others aren't.
> 
> The default signet was a long term project to address the unreliability and weaknesses of testnet. Many default signet users won't be interested in testing soft fork proposals and it is not reasonable for them to be subject to a stalling or forked blockchain because changes to a soft fork proposal or a buggy soft fork proposal pushed to the default signet makes previous valid/invalid transactions invalid/valid. If they want to test proposed soft forks on a custom signet they are opting in to possible disruption rather than it being forced upon them.
> 
> By focusing on custom signets rather than the default signet it also allows for more experimentation. Don't like the choices of which soft fork proposals have been added to bitcoin-inquisition? Set up your own custom signet with a different set of soft fork proposals and get users for your custom signet on a level playing field to bitcoin-inquisition. A soft fork proposal is found to be strictly inferior to another soft fork proposal? Just spin down the custom signet with that inferior soft fork proposal on it without impacting default signet users.
> 
> So TL;DR still enthusiastic about this concept. Just with a strong preference that it is done on a custom signet rather than on the default signet.
> 
> Thanks
> Michael
> 
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
> 
> 
> ------- Original Message -------
> On Monday, September 19th, 2022 at 11:05, Anthony Towns via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote:
> 
> 
> 
> > On Sun, Sep 18, 2022 at 02:47:38PM -0400, Antoine Riard via bitcoin-dev wrote:
> > 
> > > Said succinctly, in the genesis of creative ideas, evaluation doesn't
> > > happen at a single clear point but all along the idea lifetime, where this
> > > evaluation is as much done by the original author than its peers and a
> > > wider audience.
> > 
> > Sure. I definitely didn't mean to imply a waterfall development model,
> > or that the phases wouldn't overlap etc.
> > 
> > > I would still expose a concern to not downgrade in the pure empiricism in
> > > matter of consensus upgrades. I.e, slowly emerging the norm of a working
> > > prototype running on bitcoin-inquisition` as a determining factor of the
> > > soundness of a proposal. E.g with "upgrading lightning to support eltoo", a
> > > running e2e won't save us to think the thousands variants of pinnings, the
> > > game-theory soundness of a eltoo as mechanism in face of congestions, the
> > > evolvability of APO with more known upgrades proposals or the
> > > implementation complexity of a fully fleshed-out state machine and more
> > > questions.
> > 
> > I agree here; but I think not doing prototypes also hinders thinking
> > about all the thousands of details in a fork. It's easy to handwave
> > details away when describing things on a whiteboard; and only realise
> > they're trickier than you thought when you go to implement things.
> > 
> > > E,g if one implements the "weird" ideas
> > > about changes in the block reward issuance schedule discussed during the
> > > summer, another one might not want "noise" interferences with new
> > > fee-bumping primitives as the miner incentives are modified.
> > 
> > (I don't think "miner incentives" are really something that can be
> > investigated on signet. You can assume how miners will respond to
> > incentives and program the mining software to act that way; but there's
> > no competitive pressure in signet mining so I don't think that really
> > demonstrates anything very much. Likewise, there's much less demand for
> > blockspace on signet than on mainnet, so it's probably hard to experiment
> > with "fee incentives" too)
> > 
> > > I hope the upcoming
> > > Contracting Primitives WG will be able to document and discuss some of the
> > > relevant experiments run on bitcoin-inquisition.
> > 
> > Likewise.
> > 
> > (Lots trimmed due to either agreeing with it or having nothing to add)
> > 
> > Cheers,
> > aj
> > 
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Anthony Towns]

On Wed, Sep 28, 2022 at 11:48:32AM +0000, Michael Folkson via bitcoin-dev wrote:
> SegWit was added
> to a new testnet (Segnet) for testing rather than the pre-existing testnet
> and I think future soft fork proposals should follow a similar approach.

I think past history falls into a few groups:

 * p2sh was briefly tested on testnet (and an alternative was tested on
   mainnet)
    https://bitcointalk.org/index.php?topic=58579.msg786939#msg786939

 * cltv and csv were mostly tested on elements alpha (liquid precursor
   testnet); though they were activated on testnet 6 and 11 weeks prior
   to mainnet
    http://diyhpl.us/wiki/transcripts/gmaxwell-sidechains-elements/

 * segwit was also tested via elements alpha, though in a different
   form to what was deployed for bitcoin (ie, the elements approach
   would have been a hard fork). because of the p2p changes (you need
   additional data to validate blocks post segwit), segwit had dedicated
   test networks, up to segnet4, from 1st Jan 2016 to 30th Mar 2016.
   segwit was activated on testnet on 13th May 2016, merged into core on
   25th June 2016, and included in the 0.13.1 released on 27th October
   2016. I couldn't find very good public references about segnet, and
   don't think it saw much use outside of people implementing segwit
   consensus features themselves.

 * taproot was merged 15th October 2020 (#18267), and activated on
   signet as of genesis around 21st October 2020 (#20157). It was locked
   in on mainnet on 12th June 2021, activated on testnet on 8th July
   2021, and activated on mainnet on 14th November 2021.

 * CTV had ctv-signet created around 17th December 2020, but it wasn't
   really announced or encouraged until 17th Feb 2022. The core PR
   (#21702) was opened 16th April 2021.
    https://www.erisian.com.au/bitcoin-core-dev/log-2020-12-17.html#l-845
    https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-February/019925.html

 * I think Drivechains has a DriveNet testnet (since 2018 or earlier?),
   though I don't see an explorer, and it looks like the bitcoin code
   it's based on predates taproot.

 * Other than CTV, Drivechains and ideas being explored on Liquid,
   most other ideas for bitcoin consensus changes haven't really
   progressed past a gist, mailing list post, bip or draft patch
   to somewhere that you can actually experiment (great consensus
   cleanup, anyprevout, OP_TX/TXHASH, TLUV, SIGHASH_GROUP, PUSH_ANNEX,
   checkblockatheight, g'root/graftroot, etc...)

I thing segnet was mostly used for development of segwit itself, rather
than testing or application development -- it was reset about once a
month as changes to the segwit design occurred, and after the design
was finalised, was active on testnet, either using -addnode to connect
directly to know segwit-enabled peers, or, eventually, with seed nodes
updated and filtering via the WITNESS feature. The 23rd June 2016
meeting logs have some relevant discussion:
 https://www.erisian.com.au/bitcoin-core-dev/log-2016-06-23.html#l-178

> Even if there is community consensus on what soft fork proposals should be added to the default signet today (which may or may not be case) I find it highly unlikely this will always be the case.

The point of doing it via signet and outside of core is there doesn't
need to be any community consensus on soft forks. Unlike mainnet, signet
sBTC isn't money, and it isn't permissionless; and unlike merging it
into core, there isn't a risk of a mege having unintended side effects
impacting mainnet operation.

> We then get into the situation where the block signers (currently AJ and Kalle) are the gatekeepers on what soft fork proposals are added.

Because signet mining is a closed set (determined by the first block
after genesis when the signetchallenge is locked in), signet soft forks
always have gatekeepers. If signet miners don't opt-in to new soft forks
(by upgrading their node to allow mempool acceptance according to new
soft fork rules, and thus allow inclusion in block templates; or, if
they're running with -acceptnonstdtxn, to reject txs that don't comply
with the rules, so that funds using the new rules aren't actually anyone
can spend) then you can't test the new soft fork rules on signet.

> I don't think it is fair on the signet block signers to put them in that position and I don't think it is wise to put other Bitcoin Core contributors/maintainers in the position of having to defend why some proposed soft forks are accessible on the default signet while others aren't.

So, I think it's more accurate to say signet miners are fundamentally
*already* in that position. They can delegate that power of course,
saying "we'll just implement the default rules for the software we run",
but that just moves the responsibility around.

> The default signet was a long term project to address the unreliability and weaknesses of testnet.

That's certainly one goal. For me, that's one facet of the broader goal,
which is to make it easier to develop and test new and improved bitcoin
applications. So applications like vaults and eltoo that rely on the
creation of new consensus (or relay) features are important to me too.

Having taproot enabled on signet relatively early seemed like a win to me,
and, personally, I think it's something we should work out how to repeat.

> Many default signet users won't be interested in testing soft fork proposals and it is not reasonable for them to be subject to a stalling or forked blockchain 

Sure, that's absolutely a concern. I even raised it in the original
post:

>>> before actually mining blocks I want to make the signet miner able
>>> to automatically detect/recover if the bitcoin-inquisition node either
>>> crashes or starts producing incompatible blocks.

I think I've got a specific approach there that I'm happy with now; it's
PRed as https://github.com/bitcoin-inquisition/bitcoin/pull/7

The idea is that if you want to mine "inquisition" blocks to test out a
soft fork, you run both a core and an inquistion bitcoind, but only use
the inquisition bitcoind for generating blocks -- you *always* submit
blocks to the core bitcoind (and if core rejects and inquisition block,
you generate a new template using core and mine and submit that). That
way blocks are never submitted to the network unless they satisfy core's
consensus rules, and from core's perspective the chain always progresses.

> because changes to a soft fork proposal or a buggy soft fork proposal pushed to the default signet makes previous valid/invalid transactions invalid/valid.

The above addresses the invalid to valid case for consensus rules (ie,
the part that would make a change not be a "soft fork").

The normal cases for a soft fork are:
 * making a consensus valid transaction, invalid
 * making a relay invalid transaction, valid

That leaves the final category for buggy behaviour as making a relay
valid transaction invalid for relay. It's not clear to me that dealing
with that automatically is a good idea -- that's a pretty severe class
of bug that could result in people being unable to spend their funds;
so maybe the best thing is for txs like that to be ignored by the
miners until the build up in the mempool and people start complaining
and insisting the bug get fixed?

That's also a reason to have it happen on the global signet: you don't
want people developing new soft fork rules to introduce such a bug and
not notice it because the people making such transactions don't care about
the soft fork, and then have it suddenly appear once it's already locked
in on mainnet. Better to have the bug impact signet transactions first,
before there's any risk of bugs affecting real money.

> If they want to test proposed soft forks on a custom signet they are opting in to possible disruption rather than it being forced upon them.

If you don't want to risk any disruption, then regtest or a private
signet is a better option. A global p2p network *always* has risk of
disruption at some level or another.

But again, opting-in to potential disruption from future soft forks that
are anywhere near the deployment phase is a *good* idea: far better for
the potential disruption to become real and observable when it only
affects fake money, and to fix it then, rather than having the risk
of losing real money  due to bugs in features you didn't think you
cared about.

Cheers,
aj

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Anthony Towns]

On Fri, Sep 16, 2022 at 05:15:45PM +1000, Anthony Towns via bitcoin-dev wrote:
> So that's the concept. For practical purposes, I haven't yet merged
> either CTV or APO support into the bitcoin-inquisition 23.0 branch yet

I've now merged CTV and updated my signet miner to enforce both CTV and
APO (though you'd need to be either lucky or put some effort into it to
figure out how to get a CTV/APO transaction relayed to my miner).

Updating APO to be compatible with CTV actually seems to have found a
previously unknown bug in the CTV PR against core [0], so that seems
productive at the very least.

[0] https://github.com/bitcoin-inquisition/bitcoin/pull/8
    https://github.com/bitcoin/bitcoin/pull/21702#pullrequestreview-1118047730

I've also mined a couple of test APO transactions [1]; both reusing an
APOAS signature [2], including demonstrating the case where a third party
can replay APO signatures to send funds from duplicate UTXOs to fees,
by spending those UTXOs in a single tx [3] [4].

[1] https://mempool.space/signet/address/tb1pesae595q3cpzp8j3gy07gussw9t9hh580xj027sfz6g8e530u3nqscn0yn

[2] "ec764a8ed632916868ca6dbfdc5bed95f74b83be62d01397aba7ec916982c6721c923fa22d29b5e0c4fddde0148233f0cf401758df23d8cc89c88f04beffc3c3c1" -- sighash of 0xc1 = ANYPREVOUTANYSCRIPT|ALL

    https://mempool.space/signet/tx/ee6f6eda93a3d80f4f65f2e1000334132c9a014b3ed3dec888fdcc1f3441f52c
    https://mempool.space/signet/tx/2cbcc4857e6ee8510d9479c01fbf133a9a2cde3f5c61ccf9439c69b7b83334ba

[3] https://github.com/bitcoin/bips/blob/master/bip-0118.mediawiki#Signature_replay

[4] https://mempool.space/signet/tx/53a9747546e378956072795351e8436cf704da72d235c8ac7560787b554a4d3f

Cheers,
aj

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Michael Folkson]

Thanks for this AJ, especially the history on prior soft forks, the vast majority of which I was unclear on.

> The point of doing it via signet and outside of core is there doesn't
need to be any community consensus on soft forks. Unlike mainnet, signet
sBTC isn't money, and it isn't permissionless; and unlike merging it
into core, there isn't a risk of a mege having unintended side effects
impacting mainnet operation.

Agreed. I'm obviously much happier with proposed consensus changes being activated prematurely on a signet (default or custom) than on mainnet.

> Because signet mining is a closed set (determined by the first block
after genesis when the signetchallenge is locked in), signet soft forks
always have gatekeepers.

I'm also perfectly happy with the status quo of the default signet having block signers and gatekeepers for soft forks activated on the default signet. I'm more concerned with those gatekeepers being under pressure to merge unfinished, buggy soft fork proposals on the default signet which need to be reversed or changed disrupting all default signet users. The bar for mainnet activations is obviously much higher than for the default signet but the default signet does still need a bar.

> If you don't want to risk any disruption, then regtest or a private
signet is a better option. A global p2p network *always* has risk of
disruption at some level or another.

Right but disruption isn't boolean, it is a spectrum. It isn't disruption or zero disruption. The more soft fork proposals that are enabled on the default signet (and the more changes to those soft fork proposals pushed to the default signet) the higher the risk of a stalling blockchain (your signet node rejects a block the rest of the signet network accepts). The small number of block signers (currently 2) should prevent you being forked off entirely onto a different default signet chain with new mined blocks being added to your blockchain tip but your blockchain could stall.

What should happen in this scenario? Say I'm a default signet full node runner and I don't want to run any code outside of say the Bitcoin Core repo. I don't care about the proposed soft forks being tested on the default signet, I just care about testing my application with the existing consensus rules on mainnet. However, my default signet blockchain has stalled because of some consensus rule adjustment (an effective hard fork) made by the signet miners and the block signers. I have to run a patch from bitcoin-inquisition to get my node adding blocks again? I'm essentially being forced to run code from bitcoin-inquisition or wait many months for a default signet checkpoint in a Core release.

I looked into linux-next[0] which you mentioned as an interesting parallel in the Linux ecosystem on last week's Bitcoin Optech Twitter Spaces [1]. In that link linux-next is described as:

"The linux-next tree is the holding area for patches aimed at the next kernel merge window."

I guess I'd also want expectations to be tempered a little for consensus changes on bitcoin-inquisition versus say this description of linux-next. I don't know where the bar will be set for default signet soft fork activations by the block signers and the miners but wherever it is set it will be lower than mainnet. And to be considered for activation on mainnet these proposals do require community consensus if we want to minimize the risk of mainnet chain splits. There are no block signers or regularly updated checkpoints on mainnet. It is certainly possible that soft fork proposals that get activated on the default signet never get activated on mainnet and that being activated on the default signet offers no guarantees or even intentions/aims for the next Bitcoin Core (or any alternative implementation) release. I'd like to avoid the "my soft fork proposal has been activated on the default signet so you should expect it to be activated on mainnet within x months or y years" type thing.

Thanks
Michael

[0]: https://www.kernel.org/doc/man-pages/linux-next.html
[1]: https://twitter.com/bitcoinoptech/status/1574697495325974528?s=20&t=XWkpA459C9qxOOrBuP2fYA

--
Michael Folkson
Email: michaelfolkson at protonmail.com
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3


------- Original Message -------
On Sunday, October 2nd, 2022 at 07:12, Anthony Towns via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:


> On Fri, Sep 16, 2022 at 05:15:45PM +1000, Anthony Towns via bitcoin-dev wrote:
> 
> > So that's the concept. For practical purposes, I haven't yet merged
> > either CTV or APO support into the bitcoin-inquisition 23.0 branch yet
> 
> 
> I've now merged CTV and updated my signet miner to enforce both CTV and
> APO (though you'd need to be either lucky or put some effort into it to
> figure out how to get a CTV/APO transaction relayed to my miner).
> 
> Updating APO to be compatible with CTV actually seems to have found a
> previously unknown bug in the CTV PR against core [0], so that seems
> productive at the very least.
> 
> [0] https://github.com/bitcoin-inquisition/bitcoin/pull/8
> https://github.com/bitcoin/bitcoin/pull/21702#pullrequestreview-1118047730
> 
> I've also mined a couple of test APO transactions [1]; both reusing an
> APOAS signature [2], including demonstrating the case where a third party
> can replay APO signatures to send funds from duplicate UTXOs to fees,
> by spending those UTXOs in a single tx [3] [4].
> 
> [1] https://mempool.space/signet/address/tb1pesae595q3cpzp8j3gy07gussw9t9hh580xj027sfz6g8e530u3nqscn0yn
> 
> [2] "ec764a8ed632916868ca6dbfdc5bed95f74b83be62d01397aba7ec916982c6721c923fa22d29b5e0c4fddde0148233f0cf401758df23d8cc89c88f04beffc3c3c1" -- sighash of 0xc1 = ANYPREVOUTANYSCRIPT|ALL
> 
> https://mempool.space/signet/tx/ee6f6eda93a3d80f4f65f2e1000334132c9a014b3ed3dec888fdcc1f3441f52c
> https://mempool.space/signet/tx/2cbcc4857e6ee8510d9479c01fbf133a9a2cde3f5c61ccf9439c69b7b83334ba
> 
> [3] https://github.com/bitcoin/bips/blob/master/bip-0118.mediawiki#Signature_replay
> 
> [4] https://mempool.space/signet/tx/53a9747546e378956072795351e8436cf704da72d235c8ac7560787b554a4d3f
> 
> Cheers,
> aj
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] bitcoin-inquistion: evaluating soft forks on signet

[Anthony Towns]

On Sun, Oct 02, 2022 at 03:25:19PM +0000, Michael Folkson via bitcoin-dev wrote:
> I'm also perfectly happy with the status quo of the default signet
> having block signers and gatekeepers for soft forks activated on the
> default signet. I'm more concerned with those gatekeepers being under
> pressure to merge unfinished, buggy soft fork proposals on the default
> signet which need to be reversed or changed disrupting all default
> signet users.

First, I think it's far better for signet miners to be under that pressure
than either mainnet miners or maintainers/devs of bitcoin core. Or for
that matter, users of bitcoin who are just trying to use bitcoin and
not lose their money to bank confiscation or central bank hyperinflation.

That's where we stand today, whether you look solely at historical
precedent (cltv, csv, segwit were only testable on blockstream's elements
alpha prior to being merged into core, and combined with confidential
assets, that's not really a 1:1 test environment; taproot wasn't really
testable anywhere prior to being merged into core), or you consider the
focus of people actively trying to get forks deployed currently (ctv
has been pushing for a merge [0], and considered trying to get users
and miners to adopt it [1]; likewise the great consensus cleanup first
proposed a PR for core [2] before posting a bip draft [3] and progress
stopped when the PR didn't move forwards; likewise drivechains/bip300's
current deployment approach is "do a uasf on mainnet"); or see sentiment
such as [4].

[0] https://www.erisian.com.au/bitcoin-core-dev/log-2022-01-13.html#l-490
    https://rubin.io/bitcoin/2021/12/24/advent-27/

[1] https://rubin.io/bitcoin/2022/04/17/next-steps-bip119/

[2] https://github.com/bitcoin/bitcoin/pull/15482

[3] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-March/016714.html

[4] https://twitter.com/CobraBitcoin/status/1570380739010793479

It's *great* that core maintainers, reviewers, devs, URSF advocates, etc
are able to resist pressure to merge bad things; what's not great is
directing the time and attention of researchers and devs and businesses
who are trying to come up with good things for bitcoin at something that
doesn't encourage useful forward progress.

But second, APO and CTV aren't being kept out of core because they're
"unfinished and buggy" per se (which isn't to say they aren't buggy or
shouldn't be kept out for that reason); at least in my view, they're
being kept out because of a combination of (a) it's not clear that
they're desirable to deploy on mainnet (whether at all, or in comparison
to other ways of obtaining similar functionality); and (b) prioritising
reducing risk on mainnet, vs improving the ability to test new ideas
outside of mainnet.

Bugs are much easier to deal with in comparison: you put a bunch of
testing/dev effort in to figure out what bugs there might be, then you
analyse them, then you fix them. If it were just a matter of finding
and fixing bugs, that's still hard, sure, but it's something we know
how to do.

It's the broader questions that are trickier: eg, do we want CTV first, or
CTV+APO at the same time, or just APO first? do we want some subtle tweaks
to CTV or APO rules to make them better? do we want OP_TXHASH or OP_TX
or some other variant instead? do we want to skip the intermediate steps
and go straight to simplicity/lisp? do we want to never have anything
that may risk covenant-like behaviour ever? Without even an idea how to
get answers to those, it's not clear that it even makes sense to spend
the time working on finding/fixing obscure implementation bugs in the
proposals.

(Ultimately, in my opinion, it's the same thing with drivechains and the
great consensus cleanup: are these ideas sensible to deploy on mainnet? If
the answer to that were a clear yes for either of them, then it would
make sense to work on merging them in core and activating on mainnet;
but at least to me, it's not clear whether the answer should be yes,
yes after some specific set of changes, or no. Nor is it clear what work
would help produce a clear answer)

I think breaking the loop there is helpful: get these ideas out on
signet, where finding and fixing bugs does matter and is worth doing, but
where you *don't* have to deal with deep existential questions because
you're not messing with a multi billion dollar system and committing to
supporting the feature for the entire future of humanity.

Then, if there are alternative approaches that people think might be
better, get them out on signet too so that you can do apples-to-apples
comparisons: see how much code they are to actually implement, how
convenient they are to build on, whether there are any unexpected
differences between theory and practice, etc. Then you can build up real
answers to "is this a sensible thing to deploy on mainnet?"

For that, to get things onto signet you really only need to establish:

 * it's interesting enough to be worth spending time on
 * it's gone through a decent level of review and there are no known
   bugs
 * it doesn't conflict too heavily with the other interesting changes
   we'd like to look at

and as a result you get to see the change in a production-like
environment, and can use that to help get better answers to the deeper,
harder questions.

There's definitely some threshold where a proposed soft fork would be
too much effort to add to inquisition -- perhaps that comes with adding
something like Simplicity ("227 changed files with 72,617 additions"
[5]), or perhaps it would be doing something like confidential assets
which is both intrusive and perhaps undesirable for mainnet deployment,
or perhaps it's just numbers: I had to adjust the APO patches to rebase
them on top of CTV; doing that N-1 times (in perhaps N**2 locations?) for
N soft forks will no doubt get tedious as N increases -- so maybe only
merging the "top 10" proposals in any six month period would make sense? I
don't really see the problem with crossing that bridge when we come to
it though.

[5] https://github.com/ElementsProject/elements/compare/simplicity

I guess I don't really mind if it's just me and Kalle deciding what the
"top 10" proposals are, or deciding at what point additional PRs get
too hard to merge. But in my ideal world, we'd have multiple devs and
researchers reviewing PRs in the inquisition repo, and as the ones doing
the work, it would make sense for them also to be the ones deciding what
projects are the most interesting and worth spending that effort on, and
thus which proposals are included and which ones aren't. At least that
way wannabe gatekeepers have to at least contribute useful review effort.

> Right but disruption isn't boolean, it is a spectrum. It isn't
> disruption or zero disruption. The more soft fork proposals that are
> enabled on the default signet (and the more changes to those soft fork
> proposals pushed to the default signet) the higher the risk of a stalling
> blockchain

Like I said, I believe PR#7 makes that particular risk negligible (ie,
for people following signet with bitcoin core, the risk of a stalling
chain is no greater than it would be if all the signet miners were also
only running bitcoin core).

But you're right, it is a spectrum: eg, there's also the risk that
a bug in one soft fork interferes with testing another soft fork
(perhaps core nodes see signet continuing to add blocks, but inquisition
nodes do not, because the inquisition node's getblocktemplate resulted
in a block that core accepts but inquisition rejects). There's three
potential ways of mitigating that risk:

 * finding bugs like that during review, before merging the code, let
   alone running it
 * quickly noticing such bugs, and reorging blocks that trigger them out
 * using the -renounce feature of bitcoin-inquisition to temporarily
   disable enforcing a buggy soft fork, until a fix can be merged and
   deployed

But that risk only affects people following signet using an inquisition
node, and occurs whether or not it's a shared chain with bitcoin core
nodes. I'd hope that we can have good enough review that consensus bugs
are pretty rare in practice; but in the event that we do have them,
probably better that inquisition nodes do fail in obvious ways, so that
the bugs get noticed quickly and fixed.

> "The linux-next tree is the holding area for patches aimed at the next kernel merge window."
> I guess I'd also want expectations to be tempered a little for consensus changes on bitcoin-inquisition versus say this description of linux-next.

I think you're misinterpreting that description. "aimed at" doesn't
mean "will be accepted during", and more importantly, linux-next
is just an inspiration, not a template to follow literally. Anyway,
https://lwn.net/Articles/287155/ might be a better jumping off point if
you're interested in that rabbit hole.

> I'd like to avoid the "my soft fork proposal has been activated on
> the default signet so you should expect it to be activated on mainnet
> within x months or y years" type thing.

Like I said: this is a way to improve the "evaluation phase". Think of
it like the proposal being a kid sitting an exam; that they sit the exam
doesn't mean they're going to get an A+, even if you already have to do
a lot of work to sit the exam in the first place.

I think the ideal result from a soft fork proposal evaluation would be:

 - this is the explicit proposal [bipN], here are the corresponding
   changes to the code [PR#N]

 - the performance impact on validators/miners of this change is [p]
   so in the context of the applications mentioned above, that's [x.p,
   y.p, z.p]. you can observe worst case performance under normal
   conditions (where relay rules apply) by looking at signet blocks
   [a, b, c]; worst case performance if a miner is attacking (using
   non-standard transactions) may look like [d].

 - people have come up with other alternative ideas [x, y]. this
   proposal is superior to [x] because of [objective reason],
   and superior to [y] because [when we tried it, y turned out to
   be too annoying to implement/use].

 - here are real, functioning examples of useful, new/improved
   applications that you can build with this feature. if it were activated
   on mainnet, they could be deployed on day 1, and see real use: [x,
   y, z]

We've been pretty good at the first two already; it's the second two
that I think are holding back current proposals, and that this would help
improve. At least for me, an "A+" answer to all of the above would cause
me to advocate for a proposal to be deployed on mainnet. My concept of
an A+ answer here is "this is such a good idea that it's now obvious to
essentially everyone, and there's no meaningful debate left to have".

A "B" answer, where, say, applications using the feature exist, but
don't seem very interesting or valuable is also possible; I'd think
that's a "needs improvement" result, where maybe you go back and try
to come up with a better proposal that enables more useful results,
rather than trying to get it deployed on mainnet. A "B" answer still
leaves open the question of "is there really a point? changes are risky,
and signet's not going to test every possible scenario..."

Having the outcome of an evaluation be an "F" for fail is also useful
-- maybe it turns out that despite a bunch of people thinking CTV
or drivechains are cool, that they do make it too easy to destroy
everything. In that case, having an objective demonstration of the
failure mode is a great outcome of an evaluation process: it allows us
to say "sorry, it's a waste of time working on this; you'll need to come
up with an entirely new approach that avoids this flaw" and have R&D
effort spent on useful things instead. Far better that than not giving
an answer and letting people assume "oh, we just need to hire someone
full time to advocate and shepherd the proposal" and spend more
R&D effort on a dead end.

(In the event that a proposed soft fork that gets added to inquisition
enables interesting/non-obvious miner-only attacks -- drivechains
maybe? -- I think I'd be open to the idea of manually mining some
non-standard signet blocks in order to crystallize what that sort of
attack might look like)

I'd say the "length of time" thing should look more like:

 - here's our awesome idea, isn't it exciting?
 - wow, people really are excited, let's implement it and deploy it
   on signet!
 - great, it's been on signet for a while: here's the applications
   people have built using our idea: you should have a look!
 - it seems like we've resolved all the issues, and people are pretty
   excited about using the new apps with real money, let's deploy it
   on mainnet

that is "it's been on signet a long time" is more about "here are the
apps that people have developed in that time" and "here's the adversarial
analysis people have done over that period to see if the idea is safe
or not". Whether something gets deployed on mainnet is more a question
of "are these apps actually valuable", "have the risks been thoroughly
explored and minimised", and "have alternatives been explored". If the
answer to some/all of those is still "no", then having had a long time
for that work to happen is probably more a negative than a positive...

Cheers,
aj