[bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Segue]

On 3/17/18, someone posted on the Lightning-dev list, "Can I try 
Lightning without running a fully-fledged bitcoin block chain? (Yubin 
Ruan)."  The inquirer was asking because he didn't have much space to 
store the entire blockchain on his laptop.

I replied:

"Developers,

On THIS note and slightly off-topic but relevant, why can't chunks of 
blockchain peel off the backend periodically and be archived, say on 
minimum of 150 computers across 7 continents?

It seems crazy to continue adding on to an increasingly long chain to 
infinity if the old chapters (i.e. more than, say, 2 years old) could be 
stored in an evenly distributed manner across the planet. The same 150 
computers would not need to store every chapter either, just the index 
would need to be widely distributed in order to reconnect with a chapter 
if needed. Then maybe it is no longer a limitation in the future for 
people like Yubin. "

It was suggested by a couple of lightning developers that I post this 
idea on the bitcoin-dev list.  So, here I post :).

Segue

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[アルム カールヨハン]

He can use pruning to only store the last X MB of the blockchain. It
will store the UTXO set though, which is a couple of GBs. In total, a
pruned node with pruning set to 1000 MB ends up using 4.5 GB
currently, but it varies slightly due to the # of UTXOs in existence.

On Thu, May 10, 2018 at 9:56 AM, Segue via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> On 3/17/18, someone posted on the Lightning-dev list, "Can I try Lightning
> without running a fully-fledged bitcoin block chain? (Yubin Ruan)."  The
> inquirer was asking because he didn't have much space to store the entire
> blockchain on his laptop.
>
> I replied:
>
> "Developers,
>
> On THIS note and slightly off-topic but relevant, why can't chunks of
> blockchain peel off the backend periodically and be archived, say on minimum
> of 150 computers across 7 continents?
>
> It seems crazy to continue adding on to an increasingly long chain to
> infinity if the old chapters (i.e. more than, say, 2 years old) could be
> stored in an evenly distributed manner across the planet. The same 150
> computers would not need to store every chapter either, just the index would
> need to be widely distributed in order to reconnect with a chapter if
> needed. Then maybe it is no longer a limitation in the future for people
> like Yubin. "
>
> It was suggested by a couple of lightning developers that I post this idea
> on the bitcoin-dev list.  So, here I post :).
>
> Segue
>
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Jim Posen]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

That is a good observation that most of the historical data does not need
to be kept around. I believe what you are suggested is already implemented,
however. Bitcoin Core can operate in a pruned mode, where the bulk of the
historical block data is discarded and only the current UTXO set (and a few
recent blocks) are kept. As you note, some nodes on the network need to run
in archive mode to help new nodes get in sync. BIP 159 helps identify these
archive nodes at the gossip layer.

In the case of lightning, some implementations made use of the additional
txindex, which is not compatible with pruned mode.

On Wed, May 9, 2018 at 5:56 PM, Segue via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On 3/17/18, someone posted on the Lightning-dev list, "Can I try Lightning
> without running a fully-fledged bitcoin block chain? (Yubin Ruan)."  The
> inquirer was asking because he didn't have much space to store the entire
> blockchain on his laptop.
>
> I replied:
>
> "Developers,
>
> On THIS note and slightly off-topic but relevant, why can't chunks of
> blockchain peel off the backend periodically and be archived, say on
> minimum of 150 computers across 7 continents?
>
> It seems crazy to continue adding on to an increasingly long chain to
> infinity if the old chapters (i.e. more than, say, 2 years old) could be
> stored in an evenly distributed manner across the planet. The same 150
> computers would not need to store every chapter either, just the index
> would need to be widely distributed in order to reconnect with a chapter if
> needed. Then maybe it is no longer a limitation in the future for people
> like Yubin. "
>
> It was suggested by a couple of lightning developers that I post this idea
> on the bitcoin-dev list.  So, here I post :).
>
> Segue
>
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 2608 bytes --]

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[ZmnSCPxj]

Good morning karl and Segue,

Specifically for c-lightning, we are not yet rated for pruned bitcoind use, although if you installed and started running bitcoind before installing the lightningd, caught up to the chain, and then installed lightningd and set things up so that bitcoind will get killed if lightningd stops running (so that bitcoind will "never" leave lightningd too far behind).

Officially though pruned bitcoind is not supported for c-lightning, so loss of funds due to doing the above idea is entirely your fault.

On the topic of such a "chapter-based" archiving, it needs to get implemented and reviewed.  As-is I see no reason why it cannot be done, but I think the details are far more important.

1.  How do we select the archive servers?
2.  How can we ensure that no chapter has only a small number of actual owners who could easily coordinate to deny access to historical blockchain data to those they deem undesirable?

Regards,
ZmnSCPxj

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On May 10, 2018 2:48 PM, アルム　カールヨハン via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> He can use pruning to only store the last X MB of the blockchain. It
> 
> will store the UTXO set though, which is a couple of GBs. In total, a
> 
> pruned node with pruning set to 1000 MB ends up using 4.5 GB
> 
> currently, but it varies slightly due to the # of UTXOs in existence.
> 
> On Thu, May 10, 2018 at 9:56 AM, Segue via bitcoin-dev
> 
> bitcoin-dev@lists.linuxfoundation.org wrote:
> 
> > On 3/17/18, someone posted on the Lightning-dev list, "Can I try Lightning
> > 
> > without running a fully-fledged bitcoin block chain? (Yubin Ruan)." The
> > 
> > inquirer was asking because he didn't have much space to store the entire
> > 
> > blockchain on his laptop.
> > 
> > I replied:
> > 
> > "Developers,
> > 
> > On THIS note and slightly off-topic but relevant, why can't chunks of
> > 
> > blockchain peel off the backend periodically and be archived, say on minimum
> > 
> > of 150 computers across 7 continents?
> > 
> > It seems crazy to continue adding on to an increasingly long chain to
> > 
> > infinity if the old chapters (i.e. more than, say, 2 years old) could be
> > 
> > stored in an evenly distributed manner across the planet. The same 150
> > 
> > computers would not need to store every chapter either, just the index would
> > 
> > need to be widely distributed in order to reconnect with a chapter if
> > 
> > needed. Then maybe it is no longer a limitation in the future for people
> > 
> > like Yubin. "
> > 
> > It was suggested by a couple of lightning developers that I post this idea
> > 
> > on the bitcoin-dev list. So, here I post :).
> > 
> > Segue
> > 
> > bitcoin-dev mailing list
> > 
> > bitcoin-dev@lists.linuxfoundation.org
> > 
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> bitcoin-dev mailing list
> 
> bitcoin-dev@lists.linuxfoundation.org
> 
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Patrick Shirkey]

> On 3/17/18, someone posted on the Lightning-dev list, "Can I try
> Lightning without running a fully-fledged bitcoin block chain? (Yubin
> Ruan)."  The inquirer was asking because he didn't have much space to
> store the entire blockchain on his laptop.
>
> I replied:
>
> "Developers,
>
> On THIS note and slightly off-topic but relevant, why can't chunks of
> blockchain peel off the backend periodically and be archived, say on
> minimum of 150 computers across 7 continents?
>
> It seems crazy to continue adding on to an increasingly long chain to
> infinity if the old chapters (i.e. more than, say, 2 years old) could be
> stored in an evenly distributed manner across the planet. The same 150
> computers would not need to store every chapter either, just the index
> would need to be widely distributed in order to reconnect with a chapter
> if needed. Then maybe it is no longer a limitation in the future for
> people like Yubin. "
>
> It was suggested by a couple of lightning developers that I post this
> idea on the bitcoin-dev list.  So, here I post :).
>

You can already use the "prune" flag to get a snapshot of the blockchain
but it is incompatible with "txindex" and "rescan" so maybe that is and
issue for lightning nodes?




-- 
Patrick Shirkey
Boost Hardware

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Kulpreet Singh]

[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]

Apologies for a noob question.

But if I understand correctly, lightning nodes need to check if a
counterparty is broadcasting an old channel state and in response broadcast
a penalty/justice transaction. Does that mean lightning nodes only need to
watch for transactions that come after the funding transaction? Is that the
only reason lightning needs to run bitcoind with txindex?

If that is the case, and a lightning node only needs to query transactions
broadcast after the funding transaction, then a pruned bitcoind instance
with txindex might be a bit handy.

Also from [1] it seems that indexing pruned nodes is not supported because
it doesn't make sense, not that it was infeasible. Now with the lightning
requirements, does an indexed pruned node start to make sense?

Once again, please forgive my naive understanding of some of the issues
involved and thanks for your patience.

Regards
Kulpreet

[1]
https://bitcoin.stackexchange.com/questions/52889/bitcoin-core-txindex-vs-default-mode-vs-pruned-mode-in-depth#52894


On Thu, 10 May 2018, 14:47 Patrick Shirkey via bitcoin-dev, <
bitcoin-dev@lists.linuxfoundation.org> wrote:

>
> > On 3/17/18, someone posted on the Lightning-dev list, "Can I try
> > Lightning without running a fully-fledged bitcoin block chain? (Yubin
> > Ruan)."  The inquirer was asking because he didn't have much space to
> > store the entire blockchain on his laptop.
> >
> > I replied:
> >
> > "Developers,
> >
> > On THIS note and slightly off-topic but relevant, why can't chunks of
> > blockchain peel off the backend periodically and be archived, say on
> > minimum of 150 computers across 7 continents?
> >
> > It seems crazy to continue adding on to an increasingly long chain to
> > infinity if the old chapters (i.e. more than, say, 2 years old) could be
> > stored in an evenly distributed manner across the planet. The same 150
> > computers would not need to store every chapter either, just the index
> > would need to be widely distributed in order to reconnect with a chapter
> > if needed. Then maybe it is no longer a limitation in the future for
> > people like Yubin. "
> >
> > It was suggested by a couple of lightning developers that I post this
> > idea on the bitcoin-dev list.  So, here I post :).
> >
>
> You can already use the "prune" flag to get a snapshot of the blockchain
> but it is incompatible with "txindex" and "rescan" so maybe that is and
> issue for lightning nodes?
>
>
>
>
> --
> Patrick Shirkey
> Boost Hardware
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 3704 bytes --]

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Christian Decker]

Kulpreet Singh via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
writes:
> But if I understand correctly, lightning nodes need to check if a
> counterparty is broadcasting an old channel state and in response
> broadcast a penalty/justice transaction. Does that mean lightning
> nodes only need to watch for transactions that come after the funding
> transaction? Is that the only reason lightning needs to run bitcoind
> with txindex?

Yes, Lightning nodes need to monitor the network for transactions that
they need to react to. This is basically tailing the blockchain and
looking for anything suspicious. The `bitcoind` sitting next to the
lightning node however does not need to keep an index of the
transactions, at least for c-lightning, because we just ask for the full
block that then gets scanned for transactions of interest and then we
discard the rest of the block. We never ask for a specific transaction
from `bitcoind` and therefore we don't need to run with `-txindex`.

> If that is the case, and a lightning node only needs to query
> transactions broadcast after the funding transaction, then a pruned
> bitcoind instance with txindex might be a bit handy.

Pruned nodes should work, as long as the current blockchain head that
the lightning node has seen does not fall into the pruned range, since
in that case it won't be able to fetch and process the blocks anymore.

> Also from [1] it seems that indexing pruned nodes is not supported
> because it doesn't make sense, not that it was infeasible. Now with
> the lightning requirements, does an indexed pruned node start to make
> sense?

I don't think we should ever require `-txindex` to run a lightning node
(I know some implementations did in the past), since that'd be a very
onerous requirement to run a lightning node. Tailing the blockchain is
more than sufficient to get the necessary data, and hopefully we can get
our reliance on `bitcoind` down to a minimum in the future.

> Once again, please forgive my naive understanding of some of the issues
> involved and thanks for your patience.

Absolutely no problem, it is a common misconception that `-txindex` is
required to run a lightning node in all cases :-)

Cheers,
Christian

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Brian Lockhart]

[-- Attachment #1: Type: text/plain, Size: 3394 bytes --]

Somewhat related question -

In the interest of avoiding running multiple bitcoind full nodes - is there
a method to allow a Lightning node to point to / access a separate
already-existing node, vs. requiring it to have its own dedicated local
instance of bitcoind running?

I.e. if I already have a full bitcoin node running, could I use RPC calls
or something to tell my Lightning node to use that node, instead of
spinning up *another* full node? I’m currently minimizing the network
thrashing by whitelisting my LN bitcoind node to only point to my existing
full node for updates, but if I could just point my whole LN node at it,
that’s save on disk storage etc. etc. etc.


Apologies if this is already in there (or has been added) and I missed it
because I haven’t kept up with release notes…




On June 13, 2018 at 6:35:49 AM, Christian Decker via bitcoin-dev (
bitcoin-dev@lists.linuxfoundation.org) wrote:

Kulpreet Singh via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
writes:
> But if I understand correctly, lightning nodes need to check if a
> counterparty is broadcasting an old channel state and in response
> broadcast a penalty/justice transaction. Does that mean lightning
> nodes only need to watch for transactions that come after the funding
> transaction? Is that the only reason lightning needs to run bitcoind
> with txindex?

Yes, Lightning nodes need to monitor the network for transactions that
they need to react to. This is basically tailing the blockchain and
looking for anything suspicious. The `bitcoind` sitting next to the
lightning node however does not need to keep an index of the
transactions, at least for c-lightning, because we just ask for the full
block that then gets scanned for transactions of interest and then we
discard the rest of the block. We never ask for a specific transaction
from `bitcoind` and therefore we don't need to run with `-txindex`.

> If that is the case, and a lightning node only needs to query
> transactions broadcast after the funding transaction, then a pruned
> bitcoind instance with txindex might be a bit handy.

Pruned nodes should work, as long as the current blockchain head that
the lightning node has seen does not fall into the pruned range, since
in that case it won't be able to fetch and process the blocks anymore.

> Also from [1] it seems that indexing pruned nodes is not supported
> because it doesn't make sense, not that it was infeasible. Now with
> the lightning requirements, does an indexed pruned node start to make
> sense?

I don't think we should ever require `-txindex` to run a lightning node
(I know some implementations did in the past), since that'd be a very
onerous requirement to run a lightning node. Tailing the blockchain is
more than sufficient to get the necessary data, and hopefully we can get
our reliance on `bitcoind` down to a minimum in the future.

> Once again, please forgive my naive understanding of some of the issues
> involved and thanks for your patience.

Absolutely no problem, it is a common misconception that `-txindex` is
required to run a lightning node in all cases :-)

Cheers,
Christian
_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 5608 bytes --]

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Christian Decker]

Brian Lockhart <brianlockhart@gmail.com> writes:
> In the interest of avoiding running multiple bitcoind full nodes - is there
> a method to allow a Lightning node to point to / access a separate
> already-existing node, vs. requiring it to have its own dedicated local
> instance of bitcoind running?
>
> I.e. if I already have a full bitcoin node running, could I use RPC calls
> or something to tell my Lightning node to use that node, instead of
> spinning up *another* full node? I’m currently minimizing the network
> thrashing by whitelisting my LN bitcoind node to only point to my existing
> full node for updates, but if I could just point my whole LN node at it,
> that’s save on disk storage etc. etc. etc.

Certainly, that's supported by all 3 implementations:

 - With c-lightning you can either configure `bitcoin-cli` to connect to
   a remote node with the `rpcconnect`, `rpcuser`, and `rpcpassword`
   options in the `bitcoin.conf` file (at which point all calls to
   `bitcoin-cli` will use that node) or you can use the following
   command line options when starting `lightningd`: `--bitcoin-rpcuser`,
   `--bitcoin-rpcpassword` and `--bitcoin-rpcconnect`
 - lnd allows you to specify the node to connect to using the command
   line options `--bitcoind.rpchost`, `--bitcoind.rpcuser`, and
   `--bitcoind.rpcpass`.
 - Eclair requires you to edit the configuration file [1] before
   compiling afaik

Cheers,
Christian


[1] https://github.com/ACINQ/eclair/blob/master/eclair-core/src/main/resources/reference.conf

Re: [bitcoin-dev] Why not archive the backend of Bitcoin blockchain?

[Brian Lockhart]

Resolved -> RTFM

Worked like a champ, first try. Thanks! Wish I had thought to look into that sooner! My c-lightning node’s resource footprint is even smaller now. Unfairly small.

And now that I’ve unexpectedly regained ~180 GB worth of free SSD space from not having that extra full node, I’m feeling wealthy. Accordingly, I’m off to squander my newfound riches over at satoshis’s place. )



> On Jun 13, 2018, at 9:17 AM, Christian Decker <decker.christian@gmail.com> wrote:
> 
> Brian Lockhart <brianlockhart@gmail.com> writes:
>> In the interest of avoiding running multiple bitcoind full nodes - is there
>> a method to allow a Lightning node to point to / access a separate
>> already-existing node, vs. requiring it to have its own dedicated local
>> instance of bitcoind running?
>> 
>> I.e. if I already have a full bitcoin node running, could I use RPC calls
>> or something to tell my Lightning node to use that node, instead of
>> spinning up *another* full node? I’m currently minimizing the network
>> thrashing by whitelisting my LN bitcoind node to only point to my existing
>> full node for updates, but if I could just point my whole LN node at it,
>> that’s save on disk storage etc. etc. etc.
> 
> Certainly, that's supported by all 3 implementations:
> 
> - With c-lightning you can either configure `bitcoin-cli` to connect to
>   a remote node with the `rpcconnect`, `rpcuser`, and `rpcpassword`
>   options in the `bitcoin.conf` file (at which point all calls to
>   `bitcoin-cli` will use that node) or you can use the following
>   command line options when starting `lightningd`: `--bitcoin-rpcuser`,
>   `--bitcoin-rpcpassword` and `--bitcoin-rpcconnect`
> - lnd allows you to specify the node to connect to using the command
>   line options `--bitcoind.rpchost`, `--bitcoind.rpcuser`, and
>   `--bitcoind.rpcpass`.
> - Eclair requires you to edit the configuration file [1] before
>   compiling afaik
> 
> Cheers,
> Christian
> 
> 
> [1] https://github.com/ACINQ/eclair/blob/master/eclair-core/src/main/resources/reference.conf