[bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning

[bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning

[Michael Folkson]

[-- Attachment #1: Type: text/plain, Size: 2978 bytes --]

I tweeted this [0] back in November 2022.

"With the btcd bugs and the analysis paralysis on a RBF policy option in Core increasingly thinking @BitcoinKnots and consensus compatible forks of Core are the future. Gonna chalk that one up to another thing @LukeDashjr was right about all along."

A new bare bones Knots style Bitcoin implementation (in C++/C) integrated with Core Lightning was a long term idea I had (and presumably many others have had) but the dysfunction on the Bitcoin Core project this week (if anything it has been getting worse over time, not better) has made me start to take the idea more seriously. It is clear to me that the current way the Bitcoin Core project is being managed is not how I would like an open source project to be managed. Very little discussion is public anymore and decisions seem to be increasingly made behind closed doors or in private IRC channels (to the extent that decisions are made at all). Core Lightning seems to have the opposite problem. It is managed effectively in the open (admittedly with fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin Core does. Regardless, selfishly I at some point would like a bare bones Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin Core codebase has collected a lot of cruft over time and the ultra conservatism that is needed when treating (potential) consensus code seems to permeate into parts of the codebase that no one is using, definitely isn't consensus code and should probably just be removed.

The libbitcoinkernel project was (is?) an attempt to extract the consensus engine out of Core but it seems like it won't achieve that as consensus is just too slippery a concept and Knots style consensus compatible codebase forks of Bitcoin Core seem to still the model. To what extent you can safely chop off this cruft and effectively maintain this less crufty fork of Bitcoin Core also isn't clear to me yet.

Then there is the question of whether it makes sense to mix C and C++ code that people have different views on. C++ is obviously a superset of C but assuming this merging of Bitcoin Core and Core Lightning is/was the optimal final destination it surely would have been better if Core Lightning was written in the same language (i.e. with classes) as Bitcoin Core.

I'm just floating the idea to (hopefully) hear from people who are much more familiar with the entirety of the Bitcoin Core and Core Lightning codebases. It would be an ambitious long term project but it would be nice to focus on some ambitious project(s) (even if just conceptually) for a while given (thankfully) there seems to be a lull in soft fork activation chaos.

Thanks
Michael

[0]: https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw

--
Michael Folkson
Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

[-- Attachment #2: Type: text/html, Size: 6335 bytes --]

Re: [bitcoin-dev] [Lightning-dev] A new Bitcoin implementation integrated with Core Lightning

[Fabian]

[-- Attachment #1: Type: text/plain, Size: 3321 bytes --]

Hi Michael,

have you seen Mako? It might at least be a good start for what you would like to achieve: https://github.com/chjj/mako

Best,
Fabian
------- Original Message -------
On Saturday, January 14th, 2023 at 9:26 PM, Michael Folkson via Lightning-dev <lightning-dev@lists.linuxfoundation.org> wrote:

> I tweeted this [0] back in November 2022.
>
> "With the btcd bugs and the analysis paralysis on a RBF policy option in Core increasingly thinking @BitcoinKnots and consensus compatible forks of Core are the future. Gonna chalk that one up to another thing @LukeDashjr was right about all along."
>
> A new bare bones Knots style Bitcoin implementation (in C++/C) integrated with Core Lightning was a long term idea I had (and presumably many others have had) but the dysfunction on the Bitcoin Core project this week (if anything it has been getting worse over time, not better) has made me start to take the idea more seriously. It is clear to me that the current way the Bitcoin Core project is being managed is not how I would like an open source project to be managed. Very little discussion is public anymore and decisions seem to be increasingly made behind closed doors or in private IRC channels (to the extent that decisions are made at all). Core Lightning seems to have the opposite problem. It is managed effectively in the open (admittedly with fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin Core does. Regardless, selfishly I at some point would like a bare bones Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin Core codebase has collected a lot of cruft over time and the ultra conservatism that is needed when treating (potential) consensus code seems to permeate into parts of the codebase that no one is using, definitely isn't consensus code and should probably just be removed.
>
> The libbitcoinkernel project was (is?) an attempt to extract the consensus engine out of Core but it seems like it won't achieve that as consensus is just too slippery a concept and Knots style consensus compatible codebase forks of Bitcoin Core seem to still the model. To what extent you can safely chop off this cruft and effectively maintain this less crufty fork of Bitcoin Core also isn't clear to me yet.
>
> Then there is the question of whether it makes sense to mix C and C++ code that people have different views on. C++ is obviously a superset of C but assuming this merging of Bitcoin Core and Core Lightning is/was the optimal final destination it surely would have been better if Core Lightning was written in the same language (i.e. with classes) as Bitcoin Core.
>
> I'm just floating the idea to (hopefully) hear from people who are much more familiar with the entirety of the Bitcoin Core and Core Lightning codebases. It would be an ambitious long term project but it would be nice to focus on some ambitious project(s) (even if just conceptually) for a while given (thankfully) there seems to be a lull in soft fork activation chaos.
>
> Thanks
> Michael
>
> [0]: https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw
>
> --
> Michael Folkson
> Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

[-- Attachment #2: Type: text/html, Size: 7317 bytes --]

Re: [bitcoin-dev] [Lightning-dev] A new Bitcoin implementation integrated with Core Lightning

[Michael Folkson]

[-- Attachment #1: Type: text/plain, Size: 4358 bytes --]

I saw it was announced, yes. The author is brilliant, he has now managed two alternative implementations of Core in two different languages :)

The problem though and why I and many others think the Knots style fork of Core is the better option is because you avoid reimplementing consensus code in a different language. If you're ultra conservative about consensus code you either want to run Core in parallel with your alternative implementation to check they don't go out of consensus or you want to run the same consensus code as Core in a Knots like fork. Hence a Knots like fork of Core in C++ integrated with Core Lightning in C seems like the better option to me for serious running in production like use cases.

--
Michael Folkson
Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

------- Original Message -------
On Saturday, January 14th, 2023 at 20:34, Fabian <fjahr@protonmail.com> wrote:

> Hi Michael,
>
> have you seen Mako? It might at least be a good start for what you would like to achieve: https://github.com/chjj/mako
>
> Best,
> Fabian
> ------- Original Message -------
> On Saturday, January 14th, 2023 at 9:26 PM, Michael Folkson via Lightning-dev <lightning-dev@lists.linuxfoundation.org> wrote:
>
>> I tweeted this [0] back in November 2022.
>>
>> "With the btcd bugs and the analysis paralysis on a RBF policy option in Core increasingly thinking @BitcoinKnots and consensus compatible forks of Core are the future. Gonna chalk that one up to another thing @LukeDashjr was right about all along."
>>
>> A new bare bones Knots style Bitcoin implementation (in C++/C) integrated with Core Lightning was a long term idea I had (and presumably many others have had) but the dysfunction on the Bitcoin Core project this week (if anything it has been getting worse over time, not better) has made me start to take the idea more seriously. It is clear to me that the current way the Bitcoin Core project is being managed is not how I would like an open source project to be managed. Very little discussion is public anymore and decisions seem to be increasingly made behind closed doors or in private IRC channels (to the extent that decisions are made at all). Core Lightning seems to have the opposite problem. It is managed effectively in the open (admittedly with fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin Core does. Regardless, selfishly I at some point would like a bare bones Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin Core codebase has collected a lot of cruft over time and the ultra conservatism that is needed when treating (potential) consensus code seems to permeate into parts of the codebase that no one is using, definitely isn't consensus code and should probably just be removed.
>>
>> The libbitcoinkernel project was (is?) an attempt to extract the consensus engine out of Core but it seems like it won't achieve that as consensus is just too slippery a concept and Knots style consensus compatible codebase forks of Bitcoin Core seem to still the model. To what extent you can safely chop off this cruft and effectively maintain this less crufty fork of Bitcoin Core also isn't clear to me yet.
>>
>> Then there is the question of whether it makes sense to mix C and C++ code that people have different views on. C++ is obviously a superset of C but assuming this merging of Bitcoin Core and Core Lightning is/was the optimal final destination it surely would have been better if Core Lightning was written in the same language (i.e. with classes) as Bitcoin Core.
>>
>> I'm just floating the idea to (hopefully) hear from people who are much more familiar with the entirety of the Bitcoin Core and Core Lightning codebases. It would be an ambitious long term project but it would be nice to focus on some ambitious project(s) (even if just conceptually) for a while given (thankfully) there seems to be a lull in soft fork activation chaos.
>>
>> Thanks
>> Michael
>>
>> [0]: https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw
>>
>> --
>> Michael Folkson
>> Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
>> Keybase: michaelfolkson
>> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

[-- Attachment #2: Type: text/html, Size: 10751 bytes --]

Re: [bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning

[alicexbt]

Hi Michael,

If I were to fork bitcoin core and maintain an implementation, I wouldn't integrate any lightning implementation with it. Instead remove some things from bitcoin core and keep it simple. There is also scope for improving privacy. Example: https://github.com/bitcoinknots/bitcoin/issues/50

You might find the commits in this branch interesting if you are looking to remove things from bitcoin core and maintain an implementation with no gui, wallet, less RPCs etc.

https://github.com/jeremyRubin/bitcoin/commits/delete-it-all


/dev/fd0
floppy disc guy


Sent with Proton Mail secure email.

------- Original Message -------
On Sunday, January 15th, 2023 at 1:56 AM, Michael Folkson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:


> I tweeted this [0] back in November 2022.
> 
> "With the btcd bugs and the analysis paralysis on a RBF policy option in Core increasingly thinking @BitcoinKnots and consensus compatible forks of Core are the future. Gonna chalk that one up to another thing @LukeDashjr was right about all along."
> 
> A new bare bones Knots style Bitcoin implementation (in C++/C) integrated with Core Lightning was a long term idea I had (and presumably many others have had) but the dysfunction on the Bitcoin Core project this week (if anything it has been getting worse over time, not better) has made me start to take the idea more seriously. It is clear to me that the current way the Bitcoin Core project is being managed is not how I would like an open source project to be managed. Very little discussion is public anymore and decisions seem to be increasingly made behind closed doors or in private IRC channels (to the extent that decisions are made at all). Core Lightning seems to have the opposite problem. It is managed effectively in the open (admittedly with fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin Core does. Regardless, selfishly I at some point would like a bare bones Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin Core codebase has collected a lot of cruft over time and the ultra conservatism that is needed when treating (potential) consensus code seems to permeate into parts of the codebase that no one is using, definitely isn't consensus code and should probably just be removed.
> 
> The libbitcoinkernel project was (is?) an attempt to extract the consensus engine out of Core but it seems like it won't achieve that as consensus is just too slippery a concept and Knots style consensus compatible codebase forks of Bitcoin Core seem to still the model. To what extent you can safely chop off this cruft and effectively maintain this less crufty fork of Bitcoin Core also isn't clear to me yet.
> 
> Then there is the question of whether it makes sense to mix C and C++ code that people have different views on. C++ is obviously a superset of C but assuming this merging of Bitcoin Core and Core Lightning is/was the optimal final destination it surely would have been better if Core Lightning was written in the same language (i.e. with classes) as Bitcoin Core.
> 
> I'm just floating the idea to (hopefully) hear from people who are much more familiar with the entirety of the Bitcoin Core and Core Lightning codebases. It would be an ambitious long term project but it would be nice to focus on some ambitious project(s) (even if just conceptually) for a while given (thankfully) there seems to be a lull in soft fork activation chaos.
> 
> Thanks
> Michael
> 
> [0]: https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw
> 
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

Re: [bitcoin-dev] [Lightning-dev] A new Bitcoin implementation integrated with Core Lightning

[Michael Folkson]

Hi alicexbt

Thanks for the suggestion. I'll take a look at the branch.

I'm personally pretty bullish on Lightning and Core Lightning is criminally underused. Plus it is more exciting (and hopefully will attract more contributors) to try something ambitious than just trim Core. I'll see if it is something the Core Lightning contributors might be interested in helping out on. I remember that Rusty said on a podcast that if he had another life he'd have liked to have worked on Core. This way he could potentially do both :)

Thanks
Michael

--
Michael Folkson
Email: michaelfolkson at protonmail.com
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3


------- Original Message -------
On Sunday, January 15th, 2023 at 12:58, alicexbt <alicexbt@protonmail.com> wrote:


> Hi Michael,
> 
> If I were to fork bitcoin core and maintain an implementation, I wouldn't integrate any lightning implementation with it. Instead remove some things from bitcoin core and keep it simple. There is also scope for improving privacy. Example: https://github.com/bitcoinknots/bitcoin/issues/50
> 
> You might find the commits in this branch interesting if you are looking to remove things from bitcoin core and maintain an implementation with no gui, wallet, less RPCs etc.
> 
> https://github.com/jeremyRubin/bitcoin/commits/delete-it-all
> 
> 
> /dev/fd0
> floppy disc guy
> 
> Sent with Proton Mail secure email.
> 
> ------- Original Message -------
> On Sunday, January 15th, 2023 at 1:56 AM, Michael Folkson via Lightning-dev lightning-dev@lists.linuxfoundation.org wrote:
> 
> 
> 
> > I tweeted this 0 back in November 2022.
> > 
> > "With the btcd bugs and the analysis paralysis on a RBF policy option in Core increasingly thinking @BitcoinKnots and consensus compatible forks of Core are the future. Gonna chalk that one up to another thing @LukeDashjr was right about all along."
> > 
> > A new bare bones Knots style Bitcoin implementation (in C++/C) integrated with Core Lightning was a long term idea I had (and presumably many others have had) but the dysfunction on the Bitcoin Core project this week (if anything it has been getting worse over time, not better) has made me start to take the idea more seriously. It is clear to me that the current way the Bitcoin Core project is being managed is not how I would like an open source project to be managed. Very little discussion is public anymore and decisions seem to be increasingly made behind closed doors or in private IRC channels (to the extent that decisions are made at all). Core Lightning seems to have the opposite problem. It is managed effectively in the open (admittedly with fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin Core does. Regardless, selfishly I at some point would like a bare bones Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin Core codebase has collected a lot of cruft over time and the ultra conservatism that is needed when treating (potential) consensus code seems to permeate into parts of the codebase that no one is using, definitely isn't consensus code and should probably just be removed.
> > 
> > The libbitcoinkernel project was (is?) an attempt to extract the consensus engine out of Core but it seems like it won't achieve that as consensus is just too slippery a concept and Knots style consensus compatible codebase forks of Bitcoin Core seem to still the model. To what extent you can safely chop off this cruft and effectively maintain this less crufty fork of Bitcoin Core also isn't clear to me yet.
> > 
> > Then there is the question of whether it makes sense to mix C and C++ code that people have different views on. C++ is obviously a superset of C but assuming this merging of Bitcoin Core and Core Lightning is/was the optimal final destination it surely would have been better if Core Lightning was written in the same language (i.e. with classes) as Bitcoin Core.
> > 
> > I'm just floating the idea to (hopefully) hear from people who are much more familiar with the entirety of the Bitcoin Core and Core Lightning codebases. It would be an ambitious long term project but it would be nice to focus on some ambitious project(s) (even if just conceptually) for a while given (thankfully) there seems to be a lull in soft fork activation chaos.
> > 
> > Thanks
> > Michael
> > 
> > --
> > Michael Folkson
> > Email: michaelfolkson at protonmail.com
> > Keybase: michaelfolkson
> > PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

Re: [bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning

[Michael Folkson]

[-- Attachment #1: Type: text/plain, Size: 4471 bytes --]

Any thoughts on this from the Core Lightning contributors? The way I see it with upcoming proposed changes to default policy (primarily though not exclusively for Lightning) and a soft fork activation attempt of APO/APOAS (primarily though not exclusively for Lightning) that a tighter coupling between the full node and the Lightning node could eventually make sense. In a world where transaction fees were much higher you'd think almost every full node would also want to be a Lightning node and so the separation of concerns would make less sense. Having two separate P2P networks and two separate P2P protocols also wouldn't make much sense in this scenario. You could obviously still opt out of Lightning P2P messages if you weren't interested in Lightning.

The alternative would be just to focus on Knots style consensus compatible forks of Core with limited additional functionality. But I think we've reached the point of no return on Core dominance and not having widely used "distros". As the ecosystem scales systems and processes should be constantly evolving and improving and to me if anything Core's seem to be going backwards.

Thanks
Michael

--
Michael Folkson
Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

------- Original Message -------
On Saturday, January 14th, 2023 at 20:26, Michael Folkson <michaelfolkson@protonmail.com> wrote:

> I tweeted this [0] back in November 2022.
>
> "With the btcd bugs and the analysis paralysis on a RBF policy option in Core increasingly thinking @BitcoinKnots and consensus compatible forks of Core are the future. Gonna chalk that one up to another thing @LukeDashjr was right about all along."
>
> A new bare bones Knots style Bitcoin implementation (in C++/C) integrated with Core Lightning was a long term idea I had (and presumably many others have had) but the dysfunction on the Bitcoin Core project this week (if anything it has been getting worse over time, not better) has made me start to take the idea more seriously. It is clear to me that the current way the Bitcoin Core project is being managed is not how I would like an open source project to be managed. Very little discussion is public anymore and decisions seem to be increasingly made behind closed doors or in private IRC channels (to the extent that decisions are made at all). Core Lightning seems to have the opposite problem. It is managed effectively in the open (admittedly with fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin Core does. Regardless, selfishly I at some point would like a bare bones Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin Core codebase has collected a lot of cruft over time and the ultra conservatism that is needed when treating (potential) consensus code seems to permeate into parts of the codebase that no one is using, definitely isn't consensus code and should probably just be removed.
>
> The libbitcoinkernel project was (is?) an attempt to extract the consensus engine out of Core but it seems like it won't achieve that as consensus is just too slippery a concept and Knots style consensus compatible codebase forks of Bitcoin Core seem to still the model. To what extent you can safely chop off this cruft and effectively maintain this less crufty fork of Bitcoin Core also isn't clear to me yet.
>
> Then there is the question of whether it makes sense to mix C and C++ code that people have different views on. C++ is obviously a superset of C but assuming this merging of Bitcoin Core and Core Lightning is/was the optimal final destination it surely would have been better if Core Lightning was written in the same language (i.e. with classes) as Bitcoin Core.
>
> I'm just floating the idea to (hopefully) hear from people who are much more familiar with the entirety of the Bitcoin Core and Core Lightning codebases. It would be an ambitious long term project but it would be nice to focus on some ambitious project(s) (even if just conceptually) for a while given (thankfully) there seems to be a lull in soft fork activation chaos.
>
> Thanks
> Michael
>
> [0]: https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw
>
> --
> Michael Folkson
> Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

[-- Attachment #2: Type: text/html, Size: 10554 bytes --]

Re: [bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning

[Kostas Karasavvas]

[-- Attachment #1: Type: text/plain, Size: 5924 bytes --]

Hi Michael,

On Wed, Apr 19, 2023 at 2:40 AM Michael Folkson via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Any thoughts on this from the Core Lightning contributors? The way I see
> it with upcoming proposed changes to default policy (primarily though not
> exclusively for Lightning) and a soft fork activation attempt of APO/APOAS
> (primarily though not
>

Could you please point me to a resource that describes the default policy
changes (that are happening for lightning)? I have seen discussions here
and there but it would help if they are aggregated somewhere for reviewing.


> exclusively for Lightning) that a tighter coupling between the full node
> and the Lightning node could eventually make sense. In a world where
> transaction fees were much higher you'd think almost every full node would
> also want to be a Lightning node and so the separation of concerns would
> make less sense.
>

Separation of concerns always makes sense to manage complexity. One would
need to have really strong incentives to counter the complexity argument.

I might be missing some context here but what would the actual benefit of
integrating them be? Not having to install lightning node separately and
maybe a more intuitive UX?


Having two separate P2P networks and two separate P2P protocols also
> wouldn't make much sense in this scenario. You could obviously still opt
> out of Lightning P2P messages if you weren't interested in Lightning.
>
> The alternative would be just to focus on Knots style consensus compatible
> forks of Core with limited additional functionality. But I think we've
> reached the point of no return on Core dominance and not having widely used
> "distros". As the ecosystem scales systems and processes should be
> constantly evolving and improving and to me if anything Core's seem to be
> going backwards.
>
>
Personally, I have great difficulty seeing lightning as something other
than an L2 build on top of Bitcoin. There will be other L2s.

Regards,
Kostas

PS. Besides, the amount of effort would be significant. Wouldn't that
effort be better spent on, say, separating the consensus logic (i.e. a
second libbitcoinkernel attempt)?



> Thanks
> Michael
>
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>
> ------- Original Message -------
> On Saturday, January 14th, 2023 at 20:26, Michael Folkson <
> michaelfolkson@protonmail.com> wrote:
>
> I tweeted this [0] back in November 2022.
>
> "With the btcd bugs and the analysis paralysis on a RBF policy option in
> Core increasingly thinking @BitcoinKnots and consensus compatible forks of
> Core are the future. Gonna chalk that one up to another thing @LukeDashjr
> was right about all along."
>
> A new bare bones Knots style Bitcoin implementation (in C++/C) integrated
> with Core Lightning was a long term idea I had (and presumably many others
> have had) but the dysfunction on the Bitcoin Core project this week (if
> anything it has been getting worse over time, not better) has made me start
> to take the idea more seriously. It is clear to me that the current way the
> Bitcoin Core project is being managed is not how I would like an open
> source project to be managed. Very little discussion is public anymore and
> decisions seem to be increasingly made behind closed doors or in private
> IRC channels (to the extent that decisions are made at all). Core Lightning
> seems to have the opposite problem. It is managed effectively in the open
> (admittedly with fewer contributors) but doesn't have the eyeballs or the
> usage that Bitcoin Core does. Regardless, selfishly I at some point would
> like a bare bones Bitcoin and Lightning implementation integrated in one
> codebase. The Bitcoin Core codebase has collected a lot of cruft over time
> and the ultra conservatism that is needed when treating (potential)
> consensus code seems to permeate into parts of the codebase that no one is
> using, definitely isn't consensus code and should probably just be removed.
>
> The libbitcoinkernel project was (is?) an attempt to extract the consensus
> engine out of Core but it seems like it won't achieve that as consensus is
> just too slippery a concept and Knots style consensus compatible codebase
> forks of Bitcoin Core seem to still the model. To what extent you can
> safely chop off this cruft and effectively maintain this less crufty fork
> of Bitcoin Core also isn't clear to me yet.
>
> Then there is the question of whether it makes sense to mix C and C++ code
> that people have different views on. C++ is obviously a superset of C but
> assuming this merging of Bitcoin Core and Core Lightning is/was the optimal
> final destination it surely would have been better if Core Lightning was
> written in the same language (i.e. with classes) as Bitcoin Core.
>
> I'm just floating the idea to (hopefully) hear from people who are much
> more familiar with the entirety of the Bitcoin Core and Core Lightning
> codebases. It would be an ambitious long term project but it would be nice
> to focus on some ambitious project(s) (even if just conceptually) for a
> while given (thankfully) there seems to be a lull in soft fork activation
> chaos.
>
> Thanks
> Michael
>
> [0]:
> https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw
>
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>


-- 
https://twitter.com/kkarasavvas

[-- Attachment #2: Type: text/html, Size: 13116 bytes --]

Re: [bitcoin-dev] A new Bitcoin implementation integrated with Core Lightning

[Michael Folkson]

[-- Attachment #1: Type: text/plain, Size: 9806 bytes --]

Hi Kostas

> Could you please point me to a resource that describes the default policy changes (that are happening for lightning)? I have seen discussions here and there but it would help if they are aggregated somewhere for reviewing.

It is hard to follow because most of the discussions aren't on public channels, a number of the devs working on these proposed default policy changes aren't taking the BIP process seriously and no one is willing to discuss the criteria for merging default policy changes (and consensus changes for that matter) into bitcoin-inquisition (default signet). In addition the work (to the extent that it is public) is scattered all over the place. So yeah it seems like a mess to me from the perspective of someone is seeking to follow, review and monitor it.

This Bitcoin StackExchange post [0] is my first attempt to do what you're looking for and these Bitcoin Core PR review clubs [1], [2] were really good from Gloria. But yeah the BIP process (as I've said a thousand times and been ignored) is the place to hammer out specifications for complex things like default policy proposals and when people don't care about formalizing specifications it becomes very hard for people like you and me to follow.

> Separation of concerns always makes sense to manage complexity. One would need to have really strong incentives to counter the complexity argument.
>> I might be missing some context here but what would the actual benefit of integrating them be? Not having to install lightning node separately and maybe a more intuitive UX?

As I say in the original email having two separate P2P networks and two separate P2P protocols (perhaps) doesn't make much sense if all (or most of the nodes) are both full nodes and Lightning nodes. A testing framework that integrates both base layer and Lightning tests could potentially be easier to track edge cases which fall in the grey area between base layer and Lightning but are critically important for Lightning. A Core wallet that doesn't support Lightning doesn't make much sense in a world where transaction fees are really high and you have to use Lightning unless you are transferring huge amounts. I agree with you that these arguments have to be strong to counter the separation of concerns angle and maybe it is too early to consider it. But if moving in the direction of more widely used consensus compatible forks of Core then having Lightning integrated could make it an attractive option.

> PS. Besides, the amount of effort would be significant. Wouldn't that effort be better spent on, say, separating the consensus logic (i.e. a second libbitcoinkernel attempt)?

libbitcoinkernel can achieve smaller (and still important) goals but I'm skeptical that the more ambitious goal of having lots of different implementations in different languages with libbitcoinkernel at its core and not having to worry about consensus bugs will be reached in the medium term. As we saw with the recent btcd/lnd bugs [3] consensus bugs can crop up in places you might not expect. In that case it was a wire parsing library that you wouldn't expect to be part of a libbitcoinkernel. So if you're still going to encounter consensus bugs outside of libbitcoinkernel there isn't much point (in my view) in using it for alternative implementations.

Thanks
Michael

[0]: https://bitcoin.stackexchange.com/questions/117315/what-and-where-are-the-current-status-of-the-bip-125-replacement-the-v3-policy
[1]: https://bitcoincore.reviews/25038
[2]: https://bitcoincore.reviews/25038-2
[3]: https://bitcoin.stackexchange.com/questions/115527/what-is-the-october-2022-bug-in-lnd-what-caused-it-and-what-would-prevent-a-sim

--
Michael Folkson
Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3

------- Original Message -------
On Wednesday, April 19th, 2023 at 10:05, Kostas Karasavvas <kkarasavvas@gmail.com> wrote:

> Hi Michael,
>
> On Wed, Apr 19, 2023 at 2:40 AM Michael Folkson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Any thoughts on this from the Core Lightning contributors? The way I see it with upcoming proposed changes to default policy (primarily though not exclusively for Lightning) and a soft fork activation attempt of APO/APOAS (primarily though not
>
> Could you please point me to a resource that describes the default policy changes (that are happening for lightning)? I have seen discussions here and there but it would help if they are aggregated somewhere for reviewing.
>
>> exclusively for Lightning) that a tighter coupling between the full node and the Lightning node could eventually make sense. In a world where transaction fees were much higher you'd think almost every full node would also want to be a Lightning node and so the separation of concerns would make less sense.
>
> Separation of concerns always makes sense to manage complexity. One would need to have really strong incentives to counter the complexity argument.
>
> I might be missing some context here but what would the actual benefit of integrating them be? Not having to install lightning node separately and maybe a more intuitive UX?
>
>> Having two separate P2P networks and two separate P2P protocols also wouldn't make much sense in this scenario. You could obviously still opt out of Lightning P2P messages if you weren't interested in Lightning.
>
>> The alternative would be just to focus on Knots style consensus compatible forks of Core with limited additional functionality. But I think we've reached the point of no return on Core dominance and not having widely used "distros". As the ecosystem scales systems and processes should be constantly evolving and improving and to me if anything Core's seem to be going backwards.
>
> Personally, I have great difficulty seeing lightning as something other than an L2 build on top of Bitcoin. There will be other L2s.
>
> Regards,
> Kostas
>
> PS. Besides, the amount of effort would be significant. Wouldn't that effort be better spent on, say, separating the consensus logic (i.e. a second libbitcoinkernel attempt)?
>
>> Thanks
>> Michael
>>
>> --
>> Michael Folkson
>> Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
>> Keybase: michaelfolkson
>> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>>
>> ------- Original Message -------
>> On Saturday, January 14th, 2023 at 20:26, Michael Folkson <michaelfolkson@protonmail.com> wrote:
>>
>>> I tweeted this [0] back in November 2022.
>>>
>>> "With the btcd bugs and the analysis paralysis on a RBF policy option in Core increasingly thinking @BitcoinKnots and consensus compatible forks of Core are the future. Gonna chalk that one up to another thing @LukeDashjr was right about all along."
>>>
>>> A new bare bones Knots style Bitcoin implementation (in C++/C) integrated with Core Lightning was a long term idea I had (and presumably many others have had) but the dysfunction on the Bitcoin Core project this week (if anything it has been getting worse over time, not better) has made me start to take the idea more seriously. It is clear to me that the current way the Bitcoin Core project is being managed is not how I would like an open source project to be managed. Very little discussion is public anymore and decisions seem to be increasingly made behind closed doors or in private IRC channels (to the extent that decisions are made at all). Core Lightning seems to have the opposite problem. It is managed effectively in the open (admittedly with fewer contributors) but doesn't have the eyeballs or the usage that Bitcoin Core does. Regardless, selfishly I at some point would like a bare bones Bitcoin and Lightning implementation integrated in one codebase. The Bitcoin Core codebase has collected a lot of cruft over time and the ultra conservatism that is needed when treating (potential) consensus code seems to permeate into parts of the codebase that no one is using, definitely isn't consensus code and should probably just be removed.
>>>
>>> The libbitcoinkernel project was (is?) an attempt to extract the consensus engine out of Core but it seems like it won't achieve that as consensus is just too slippery a concept and Knots style consensus compatible codebase forks of Bitcoin Core seem to still the model. To what extent you can safely chop off this cruft and effectively maintain this less crufty fork of Bitcoin Core also isn't clear to me yet.
>>>
>>> Then there is the question of whether it makes sense to mix C and C++ code that people have different views on. C++ is obviously a superset of C but assuming this merging of Bitcoin Core and Core Lightning is/was the optimal final destination it surely would have been better if Core Lightning was written in the same language (i.e. with classes) as Bitcoin Core.
>>>
>>> I'm just floating the idea to (hopefully) hear from people who are much more familiar with the entirety of the Bitcoin Core and Core Lightning codebases. It would be an ambitious long term project but it would be nice to focus on some ambitious project(s) (even if just conceptually) for a while given (thankfully) there seems to be a lull in soft fork activation chaos.
>>>
>>> Thanks
>>> Michael
>>>
>>> [0]: https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw
>>>
>>> --
>>> Michael Folkson
>>> Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
>>> Keybase: michaelfolkson
>>> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
> --
>
> https://twitter.com/kkarasavvas

[-- Attachment #2: Type: text/html, Size: 21978 bytes --]

Re: [bitcoin-dev] [Lightning-dev] A new Bitcoin implementation integrated with Core Lightning

[niftynei]

[-- Attachment #1: Type: text/plain, Size: 6292 bytes --]

Hi Michael,

CLN as implemented is currently nicely decoupled from the block source; as
a project we assume that the node runner will choose a block backend that
fits their self-sovereignty goals.

This provides us with a nice separation of concerns. The block source is
responsible for ensuring that only consensus valid data is delivered to the
node, which in turn allows us to focus on processing and reacting to that
data, as necessary.

Bitcoin core as a project implements a broad swath of functionality
(wallet, consensus, peering, rpc server, coin selection, key management,
etc); breaking out the validation and peering functions into more
composable parts would def open up more opportunities for building block
sources for a wide variety of projects, not just CLN.

There’s probably a real opportunity to “comingle” the peering of LN gossip
+ block data networks, this has been suggested a few times but never
seriously pursued from the LN side. Having the peering functions of
bitcoin-core broken out into a more composable/reusable piece may be a good
first step here, and as a project would largely be on the bitcoin core
side. Maybe this work is already in progress? I havent been keeping up with
developments there.

Thanks for the email, it’s definitely a good question.

Lisa


On Mon, Apr 24, 2023 at 02:09 Michael Folkson via Lightning-dev <
lightning-dev@lists.linuxfoundation.org> wrote:

> Any thoughts on this from the Core Lightning contributors? The way I see
> it with upcoming proposed changes to default policy (primarily though not
> exclusively for Lightning) and a soft fork activation attempt of APO/APOAS
> (primarily though not exclusively for Lightning) that a tighter coupling
> between the full node and the Lightning node could eventually make sense.
> In a world where transaction fees were much higher you'd think almost every
> full node would also want to be a Lightning node and so the separation of
> concerns would make less sense. Having two separate P2P networks and two
> separate P2P protocols also wouldn't make much sense in this scenario. You
> could obviously still opt out of Lightning P2P messages if you weren't
> interested in Lightning.
>
> The alternative would be just to focus on Knots style consensus compatible
> forks of Core with limited additional functionality. But I think we've
> reached the point of no return on Core dominance and not having widely used
> "distros". As the ecosystem scales systems and processes should be
> constantly evolving and improving and to me if anything Core's seem to be
> going backwards.
>
> Thanks
> Michael
>
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>
> ------- Original Message -------
>
> On Saturday, January 14th, 2023 at 20:26, Michael Folkson <
> michaelfolkson@protonmail.com> wrote:
>
> I tweeted this [0] back in November 2022.
>
> "With the btcd bugs and the analysis paralysis on a RBF policy option in
> Core increasingly thinking @BitcoinKnots and consensus compatible forks of
> Core are the future. Gonna chalk that one up to another thing @LukeDashjr
> was right about all along."
>
> A new bare bones Knots style Bitcoin implementation (in C++/C) integrated
> with Core Lightning was a long term idea I had (and presumably many others
> have had) but the dysfunction on the Bitcoin Core project this week (if
> anything it has been getting worse over time, not better) has made me start
> to take the idea more seriously. It is clear to me that the current way the
> Bitcoin Core project is being managed is not how I would like an open
> source project to be managed. Very little discussion is public anymore and
> decisions seem to be increasingly made behind closed doors or in private
> IRC channels (to the extent that decisions are made at all). Core Lightning
> seems to have the opposite problem. It is managed effectively in the open
> (admittedly with fewer contributors) but doesn't have the eyeballs or the
> usage that Bitcoin Core does. Regardless, selfishly I at some point would
> like a bare bones Bitcoin and Lightning implementation integrated in one
> codebase. The Bitcoin Core codebase has collected a lot of cruft over time
> and the ultra conservatism that is needed when treating (potential)
> consensus code seems to permeate into parts of the codebase that no one is
> using, definitely isn't consensus code and should probably just be removed.
>
> The libbitcoinkernel project was (is?) an attempt to extract the consensus
> engine out of Core but it seems like it won't achieve that as consensus is
> just too slippery a concept and Knots style consensus compatible codebase
> forks of Bitcoin Core seem to still the model. To what extent you can
> safely chop off this cruft and effectively maintain this less crufty fork
> of Bitcoin Core also isn't clear to me yet.
>
> Then there is the question of whether it makes sense to mix C and C++ code
> that people have different views on. C++ is obviously a superset of C but
> assuming this merging of Bitcoin Core and Core Lightning is/was the optimal
> final destination it surely would have been better if Core Lightning was
> written in the same language (i.e. with classes) as Bitcoin Core.
>
> I'm just floating the idea to (hopefully) hear from people who are much
> more familiar with the entirety of the Bitcoin Core and Core Lightning
> codebases. It would be an ambitious long term project but it would be nice
> to focus on some ambitious project(s) (even if just conceptually) for a
> while given (thankfully) there seems to be a lull in soft fork activation
> chaos.
>
> Thanks
> Michael
>
> [0]:
> https://twitter.com/michaelfolkson/status/1589220155006910464?s=20&t=GbPm7w5BqS7rS3kiVFTNcw
>
> --
> Michael Folkson
> Email: michaelfolkson at protonmail.com
> Keybase: michaelfolkson
> PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
>
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>

[-- Attachment #2: Type: text/html, Size: 12514 bytes --]

Re: [bitcoin-dev] [Lightning-dev] A new Bitcoin implementation integrated with Core Lightning

[Vincenzo Palazzo]

Hi Michael and Lisa,

> Hi Michael,
>
> CLN as implemented is currently nicely decoupled from the block source; as
> a project we assume that the node runner will choose a block backend that
> fits their self-sovereignty goals.
>
> This provides us with a nice separation of concerns. The block source is
> responsible for ensuring that only consensus valid data is delivered to the
> node, which in turn allows us to focus on processing and reacting to that
> data, as necessary.

Let me just mention that [1] I have been working on a plugin 
that allows experimentation with different types of Bitcoin Core 
node alternatives (including core too), and it also supports BIP 157 
with nakamoto [2].

In the upcoming months, I plan to allocate some time to work 
directly on Nakamoto.

> There’s probably a real opportunity to “comingle” the peering of LN gossip
> + block data networks, this has been suggested a few times but never
> seriously pursued from the LN side. Having the peering functions of
> bitcoin-core broken out into a more composable/reusable piece may be a good
> first step here, and as a project would largely be on the bitcoin core
> side. Maybe this work is already in progress? I havent been keeping up with
> developments there.

A missing piece at the moment is a unified approach to fee calculation. 
This logic is critical for Lightning nodes, so if we don't have a uniform 
way of estimating fees, it could lead to several issues.

P.S: The fee estimation problem may have already been solved by Neutrino, 
but I'm not aware of it.

[1] https://github.com/coffee-tools/folgore
[2] https://github.com/cloudhead/nakamoto

Cheers!

Vincent.

Re: [bitcoin-dev] [Lightning-dev] A new Bitcoin implementation integrated with Core Lightning

[Matt Corallo]

[-- Attachment #1: Type: text/html, Size: 7886 bytes --]