[bitcoin-dev] Idea: Efficient bitcoin block propagation

[bitcoin-dev] Idea: Efficient bitcoin block propagation

[Arnoud Kouwenhoven - Pukaki Corp]

[-- Attachment #1: Type: text/plain, Size: 2151 bytes --]

Hello all.

We’d like to share an idea we have to dramatically increase the bitcoin
block propagation speed after a new block has been mined for the first time.

Efficient bitcoin block propagation
A proposed solution to provide near-instantaneous block propagation on the
bitcoin network, even with slow network connections or large block sizes.
Increasing mining efficiency for everyone while decreasing transaction
confirmation times and strengthening the distributed nature of bitcoin.

Short summary: we propose to introduce bitcoin-backed guarantees
(“Guarantee Messages”) between miners. This would allow miners to mine on
blocks that are not yet fully transmitted. This reduces the effect of slow
internet connections, leveling the playing field between the 1st world
fiberoptic datacenter miners and the rest of the world. We also believe it
strengthens the bitcoin network by using existing processing power that is
currently wasted into further securing the blockchain, and it reduces the
likelihood of transactions becoming confirmed, then unconfirmed and then
-hopefully- confirmed again (due to different miners finding competing
blocks with different transactions at approx the same time).

It is possible to implement our idea as a fork of bitcoind, or as layer
between the standard bitcoind and the mining equipment. In the future it
could be incorporated in the bitcoin core if and when that becomes a
priority, but that step would not make sense until it becomes a priority.

There are a lot of nuances in this idea, and the first reaction is quite
probably that this is a crazy idea. We have attempted to address the most
important nuances in our proposal, which is currently at v.0.2.

We cannot guarantee that there are no ‘hidden devils in the details’ and we
invite you to be critical in a friendly and constructive manner. We will do
our best to answer all questions that arise.

The ‘official’ proposal is at:
PDF: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.pdf
HTML: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.html

-- Arnoud Kouwenhoven

[-- Attachment #2: Type: text/html, Size: 2367 bytes --]

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Matt Corallo]

[-- Attachment #1: Type: text/plain, Size: 3423 bytes --]

See-also: Bitcoinrelaynetwork.org. It's already in use my the majority of large miners, is publicly available to anyone, and the protocol is rather simple and the client could be tweaked easily to keep exactly it's block ready to quickly relay to the nearest server (ie only have to relay the header, the coinbase transaction, and only small other data... Experience shows this is really easy to fit into one packet on the wire). It's not nearly as complicated as your suggestion, but may still marginally favor well-connected miners, but hopefully not much (when you're taking about single packets, it should all be latency, and the servers are well distributed). If you feel so inclined, there are some todos to make it really meet is efficiency limits filled on github.com/TheBlueMatt/RelayNode, feel free to rewrite the protocol if you really want :).

Matt

On August 5, 2015 9:07:44 PM GMT+02:00, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>Hello all.
>
>We’d like to share an idea we have to dramatically increase the bitcoin
>block propagation speed after a new block has been mined for the first
>time.
>
>Efficient bitcoin block propagation
>A proposed solution to provide near-instantaneous block propagation on
>the
>bitcoin network, even with slow network connections or large block
>sizes.
>Increasing mining efficiency for everyone while decreasing transaction
>confirmation times and strengthening the distributed nature of bitcoin.
>
>Short summary: we propose to introduce bitcoin-backed guarantees
>(“Guarantee Messages”) between miners. This would allow miners to mine
>on
>blocks that are not yet fully transmitted. This reduces the effect of
>slow
>internet connections, leveling the playing field between the 1st world
>fiberoptic datacenter miners and the rest of the world. We also believe
>it
>strengthens the bitcoin network by using existing processing power that
>is
>currently wasted into further securing the blockchain, and it reduces
>the
>likelihood of transactions becoming confirmed, then unconfirmed and
>then
>-hopefully- confirmed again (due to different miners finding competing
>blocks with different transactions at approx the same time).
>
>It is possible to implement our idea as a fork of bitcoind, or as layer
>between the standard bitcoind and the mining equipment. In the future
>it
>could be incorporated in the bitcoin core if and when that becomes a
>priority, but that step would not make sense until it becomes a
>priority.
>
>There are a lot of nuances in this idea, and the first reaction is
>quite
>probably that this is a crazy idea. We have attempted to address the
>most
>important nuances in our proposal, which is currently at v.0.2.
>
>We cannot guarantee that there are no ‘hidden devils in the details’
>and we
>invite you to be critical in a friendly and constructive manner. We
>will do
>our best to answer all questions that arise.
>
>The ‘official’ proposal is at:
>PDF: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.pdf
>HTML: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.html
>
>-- Arnoud Kouwenhoven
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>bitcoin-dev mailing list
>bitcoin-dev@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 4096 bytes --]

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Arnoud Kouwenhoven - Pukaki Corp]

[-- Attachment #1: Type: text/plain, Size: 4752 bytes --]

Thanks for the reply. My understanding is that the bitcoin relay network is
a backbone of connected high speed servers to increase the rate at which
transactions and new blocks propagate - and remove a number of delays in
processing. But it would still require the miners to download the entire
block before building on top of it with any degree of confidence. With a
tweak to only send the required information for other miners to build on
top of that block, this is a step towards what we propose, yet would
require trust that the header information sent is accurate. The bitcoin
relay network website states that blocks are not fully verified and should
be checked by the miners before building on top of them.

What we propose is more complex (granted!), yet that complexity serves a
purpose. We reduce (and hopefully eliminate) the adverse incentive to
entice miners to build on inaccurate data. This is achieved by making the
financial losses of fake messages outweigh the financial gains of such
attack vectors. It could also help in the block size debate if this
proposed solution would eliminate the disadvantages of large blocks.

On Wed, Aug 5, 2015 at 1:27 PM, Matt Corallo <lf-lists@mattcorallo.com>
wrote:

> See-also: Bitcoinrelaynetwork.org. It's already in use my the majority of
> large miners, is publicly available to anyone, and the protocol is rather
> simple and the client could be tweaked easily to keep exactly it's block
> ready to quickly relay to the nearest server (ie only have to relay the
> header, the coinbase transaction, and only small other data... Experience
> shows this is really easy to fit into one packet on the wire). It's not
> nearly as complicated as your suggestion, but may still marginally favor
> well-connected miners, but hopefully not much (when you're taking about
> single packets, it should all be latency, and the servers are well
> distributed). If you feel so inclined, there are some todos to make it
> really meet is efficiency limits filled on
> github.com/TheBlueMatt/RelayNode, feel free to rewrite the protocol if
> you really want :).
>
> Matt
>
> On August 5, 2015 9:07:44 PM GMT+02:00, Arnoud Kouwenhoven - Pukaki Corp
> via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Hello all.
>>
>> We’d like to share an idea we have to dramatically increase the bitcoin
>> block propagation speed after a new block has been mined for the first time.
>>
>> Efficient bitcoin block propagation
>> A proposed solution to provide near-instantaneous block propagation on
>> the bitcoin network, even with slow network connections or large block
>> sizes. Increasing mining efficiency for everyone while decreasing
>> transaction confirmation times and strengthening the distributed nature of
>> bitcoin.
>>
>> Short summary: we propose to introduce bitcoin-backed guarantees
>> (“Guarantee Messages”) between miners. This would allow miners to mine on
>> blocks that are not yet fully transmitted. This reduces the effect of slow
>> internet connections, leveling the playing field between the 1st world
>> fiberoptic datacenter miners and the rest of the world. We also believe it
>> strengthens the bitcoin network by using existing processing power that is
>> currently wasted into further securing the blockchain, and it reduces the
>> likelihood of transactions becoming confirmed, then unconfirmed and then
>> -hopefully- confirmed again (due to different miners finding competing
>> blocks with different transactions at approx the same time).
>>
>> It is possible to implement our idea as a fork of bitcoind, or as layer
>> between the standard bitcoind and the mining equipment. In the future it
>> could be incorporated in the bitcoin core if and when that becomes a
>> priority, but that step would not make sense until it becomes a priority.
>>
>> There are a lot of nuances in this idea, and the first reaction is quite
>> probably that this is a crazy idea. We have attempted to address the most
>> important nuances in our proposal, which is currently at v.0.2.
>>
>> We cannot guarantee that there are no ‘hidden devils in the details’ and
>> we invite you to be critical in a friendly and constructive manner. We will
>> do our best to answer all questions that arise.
>>
>> The ‘official’ proposal is at:
>> PDF: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.pdf
>> HTML: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.html
>>
>> -- Arnoud Kouwenhoven
>>
>> ------------------------------
>>
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>

[-- Attachment #2: Type: text/html, Size: 5833 bytes --]

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Gregory Maxwell]

On Wed, Aug 5, 2015 at 7:53 PM, Arnoud Kouwenhoven - Pukaki Corp via
bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> Thanks for the reply. My understanding is that the bitcoin relay network is
> a backbone of connected high speed servers to increase the rate at which
> transactions and new blocks propagate - and remove a number of delays in
> processing. But it would still require the miners to download the entire
> block before building on top of it with any degree of confidence.

Your understanding is outdated.

The relay network includes an optimized transmission protocol which
enables sending the "entire" block typically in just a smal number of
bytes (much smaller than the summaries you suggest, which still leave
the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol
sent it using at most 4906 bytes.

No trust is required in this scheme because the entire block is
communicated using only a couple packets.

The current scheme is highly simplified and its efficiency could be
increased greatly with small improvements, or if miners created blocks
in an aware manner.... but with a maximum size blocks turning into 5kb
with the current setup, there hardly appears to be a reason to do so
right now.

Ultimately there is no need for information communicated with a block
at discovery time proportional to the size of the block; with the
right affordances it can be accomplished with a small constant amount
of data.

If not for this already being deployed I personally believe the
network would have already fallen into complete centeralization as a
response to larger blocks: this was constructed and deployed in order
to pull the network back from having a single pool with more than half
the hashrate.

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Arnoud Kouwenhoven - Pukaki Corp]

[-- Attachment #1: Type: text/plain, Size: 3130 bytes --]

Thanks for this (direct) feedback. It would make sense that if blocks can
be submitted using ~5kb packets, that no further optimizations would be
needed at this point. I will look into the relay network transmission
protocol to understand how it works!

I hear that you are saying that this network solves speed of transmission
and thereby (technical) block size issues. Presumably it would solve speed
of block validation too by prevalidating transactions. Assuming this is all
true, and I have no reason to doubt that at this point, I do not understand
why there is any discussion at all about the (technical) impact of large
blocks, or why there are large numbers of miners building on invalid blocks
(SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or why
there is any discussion about the speed of block validation (cpu processing
time to verify blocks and transactions in blocks being a limitation).

Our proposal aims at solving all three issues.

Now I would be glad if the suggestions we made are already implemented,
especially if that is in a more elegant approach. Great! Yet we still see
all three discussions, which is a surprise if they have been solved.

On Wed, Aug 5, 2015 at 2:16 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:

> On Wed, Aug 5, 2015 at 7:53 PM, Arnoud Kouwenhoven - Pukaki Corp via
> bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> > Thanks for the reply. My understanding is that the bitcoin relay network
> is
> > a backbone of connected high speed servers to increase the rate at which
> > transactions and new blocks propagate - and remove a number of delays in
> > processing. But it would still require the miners to download the entire
> > block before building on top of it with any degree of confidence.
>
> Your understanding is outdated.
>
> The relay network includes an optimized transmission protocol which
> enables sending the "entire" block typically in just a smal number of
> bytes (much smaller than the summaries you suggest, which still leave
> the participants needing to send the block).
>
> E.g. block 000ce90846 was 999950 bytes and the relay network protocol
> sent it using at most 4906 bytes.
>
> No trust is required in this scheme because the entire block is
> communicated using only a couple packets.
>
> The current scheme is highly simplified and its efficiency could be
> increased greatly with small improvements, or if miners created blocks
> in an aware manner.... but with a maximum size blocks turning into 5kb
> with the current setup, there hardly appears to be a reason to do so
> right now.
>
> Ultimately there is no need for information communicated with a block
> at discovery time proportional to the size of the block; with the
> right affordances it can be accomplished with a small constant amount
> of data.
>
> If not for this already being deployed I personally believe the
> network would have already fallen into complete centeralization as a
> response to larger blocks: this was constructed and deployed in order
> to pull the network back from having a single pool with more than half
> the hashrate.
>

[-- Attachment #2: Type: text/html, Size: 3792 bytes --]

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Gregory Maxwell]

On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp
<arnoud@pukaki.bz> wrote:
> Thanks for this (direct) feedback. It would make sense that if blocks can be
> submitted using ~5kb packets, that no further optimizations would be needed
> at this point. I will look into the relay network transmission protocol to
> understand how it works!
>
> I hear that you are saying that this network solves speed of transmission
> and thereby (technical) block size issues. Presumably it would solve speed
> of block validation too by prevalidating transactions.


Correct. Bitcoin Core has cached validation for many years now... if
not for that and other optimizations, things would be really broken
right now. :)

> Assuming this is all
> true, and I have no reason to doubt that at this point, I do not understand
> why there is any discussion at all about the (technical) impact of large
> blocks, why there are large numbers of miners building on invalid blocks
> (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or why
> there is any discussion about the speed of block validation (cpu processing
> time to verify blocks and transactions in blocks being a limitation).

I'm also mystified by a lot of the large block discussion, much of it
is completely divorced from the technology as deployed; much less what
we-- in industry-- know to be possible. I don't blame you or anyone in
particular on this; it's a new area and we don't yet know what we need
to know to know what we need to know; or to the extent that we do it
hasn't had time to get effectively communicated.

The technical/security implications of larger blocks are related to
other things than propagation time, if you assume people are using the
available efficient relay protocol (or better).

SPV mining is a bit of a misnomer (If I coined the term, I'm sorry).
What these parties are actually doing is blinding mining on top of
other pools' stratum work. You can think of it as sub-pooling with
hopping onto whatever pool has the highest block (I'll call it VFSSP
in this post-- validation free stratum subpooling).  It's very easy to
implement, and there are other considerations.

It was initially deployed at a time when a single pool in Europe has
amassed more than half of the hashrate. This pool had propagation
problems and a very high orphan rate, it may have (perhaps
unintentionally) been performing a selfish mining attack; mining off
their stratum work was an easy fix which massively cut down the orphan
rates for anyone who did it.  This was before the relay network
protocol existed (the fact that all the hashpower was consolidating on
a single pool was a major motivation for creating it).

VFSSP also cuts through a number of practical issues miners have had:
Miners that run their own bitcoin nodes in far away colocation
(>100ms) due to local bandwidth or connectivity issues (censored
internet); relay network hubs not being anywhere near by due to
strange internet routing (e.g. japan to china going via the US for ...
reasons...); the CreateNewBlock() function being very slow and
unoptimized, etc.   There are many other things like this-- and VFSSP
avoids them causing delays even when you don't understand them or know
about them. So even when they're easily fixed the VFSSP is a more
general workaround.

Mining operations are also usually operated in a largely fire and
forget manner. There is a long history in (esp pooled) mining where
someone sets up an operation and then hardly maintains it after the
fact... so some of the use of VFSSP appears to just be inertia-- we
have better solutions now, but they they work to deploy and changing
things involves risk (which is heightened by a lack of good
monitoring-- participants learn they are too latent by observing
orphaned blocks at a cost of 25 BTC each).

One of the frustrating things about incentives in this space is that
bad outcomes are possible even when they're not necessary. E.g. if a
miner can lower their orphan rate by deploying a new protocol (or
simply fixing some faulty hardware in their infrastructure, like
Bitcoin nodes running on cheap VPSes with remote storage)  OR they can
lower their orphan rate by pointing their hashpower at a free
centeralized pool, they're likely to do the latter because it takes
less effort.

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Sergio Demian Lerner]

[-- Attachment #1: Type: text/plain, Size: 4955 bytes --]

Is there any up to date documentation about TheBlueMatt relay network
including what kind of block compression it is currently doing? (apart from
the source code)

Regards, Sergio.

On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp
> <arnoud@pukaki.bz> wrote:
> > Thanks for this (direct) feedback. It would make sense that if blocks
> can be
> > submitted using ~5kb packets, that no further optimizations would be
> needed
> > at this point. I will look into the relay network transmission protocol
> to
> > understand how it works!
> >
> > I hear that you are saying that this network solves speed of transmission
> > and thereby (technical) block size issues. Presumably it would solve
> speed
> > of block validation too by prevalidating transactions.
>
>
> Correct. Bitcoin Core has cached validation for many years now... if
> not for that and other optimizations, things would be really broken
> right now. :)
>
> > Assuming this is all
> > true, and I have no reason to doubt that at this point, I do not
> understand
> > why there is any discussion at all about the (technical) impact of large
> > blocks, why there are large numbers of miners building on invalid blocks
> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or why
> > there is any discussion about the speed of block validation (cpu
> processing
> > time to verify blocks and transactions in blocks being a limitation).
>
> I'm also mystified by a lot of the large block discussion, much of it
> is completely divorced from the technology as deployed; much less what
> we-- in industry-- know to be possible. I don't blame you or anyone in
> particular on this; it's a new area and we don't yet know what we need
> to know to know what we need to know; or to the extent that we do it
> hasn't had time to get effectively communicated.
>
> The technical/security implications of larger blocks are related to
> other things than propagation time, if you assume people are using the
> available efficient relay protocol (or better).
>
> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry).
> What these parties are actually doing is blinding mining on top of
> other pools' stratum work. You can think of it as sub-pooling with
> hopping onto whatever pool has the highest block (I'll call it VFSSP
> in this post-- validation free stratum subpooling).  It's very easy to
> implement, and there are other considerations.
>
> It was initially deployed at a time when a single pool in Europe has
> amassed more than half of the hashrate. This pool had propagation
> problems and a very high orphan rate, it may have (perhaps
> unintentionally) been performing a selfish mining attack; mining off
> their stratum work was an easy fix which massively cut down the orphan
> rates for anyone who did it.  This was before the relay network
> protocol existed (the fact that all the hashpower was consolidating on
> a single pool was a major motivation for creating it).
>
> VFSSP also cuts through a number of practical issues miners have had:
> Miners that run their own bitcoin nodes in far away colocation
> (>100ms) due to local bandwidth or connectivity issues (censored
> internet); relay network hubs not being anywhere near by due to
> strange internet routing (e.g. japan to china going via the US for ...
> reasons...); the CreateNewBlock() function being very slow and
> unoptimized, etc.   There are many other things like this-- and VFSSP
> avoids them causing delays even when you don't understand them or know
> about them. So even when they're easily fixed the VFSSP is a more
> general workaround.
>
> Mining operations are also usually operated in a largely fire and
> forget manner. There is a long history in (esp pooled) mining where
> someone sets up an operation and then hardly maintains it after the
> fact... so some of the use of VFSSP appears to just be inertia-- we
> have better solutions now, but they they work to deploy and changing
> things involves risk (which is heightened by a lack of good
> monitoring-- participants learn they are too latent by observing
> orphaned blocks at a cost of 25 BTC each).
>
> One of the frustrating things about incentives in this space is that
> bad outcomes are possible even when they're not necessary. E.g. if a
> miner can lower their orphan rate by deploying a new protocol (or
> simply fixing some faulty hardware in their infrastructure, like
> Bitcoin nodes running on cheap VPSes with remote storage)  OR they can
> lower their orphan rate by pointing their hashpower at a free
> centeralized pool, they're likely to do the latter because it takes
> less effort.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 6065 bytes --]

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Olaoluwa Osuntokun]

[-- Attachment #1: Type: text/plain, Size: 5586 bytes --]

Other than the source code, the best documentation I've come across is a few
lines on IRC explaining the high-level design of the protocol:
https://botbot.me/freenode/bitcoin-wizards/2015-07-10/?msg=44146764&page=2

On Thu, Aug 6, 2015 at 10:18 AM Sergio Demian Lerner via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Is there any up to date documentation about TheBlueMatt relay network
> including what kind of block compression it is currently doing? (apart from
> the source code)
>
> Regards, Sergio.
>
> On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp
>> <arnoud@pukaki.bz> wrote:
>> > Thanks for this (direct) feedback. It would make sense that if blocks
>> can be
>> > submitted using ~5kb packets, that no further optimizations would be
>> needed
>> > at this point. I will look into the relay network transmission protocol
>> to
>> > understand how it works!
>> >
>> > I hear that you are saying that this network solves speed of
>> transmission
>> > and thereby (technical) block size issues. Presumably it would solve
>> speed
>> > of block validation too by prevalidating transactions.
>>
>>
>> Correct. Bitcoin Core has cached validation for many years now... if
>> not for that and other optimizations, things would be really broken
>> right now. :)
>>
>> > Assuming this is all
>> > true, and I have no reason to doubt that at this point, I do not
>> understand
>> > why there is any discussion at all about the (technical) impact of large
>> > blocks, why there are large numbers of miners building on invalid blocks
>> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or
>> why
>> > there is any discussion about the speed of block validation (cpu
>> processing
>> > time to verify blocks and transactions in blocks being a limitation).
>>
>> I'm also mystified by a lot of the large block discussion, much of it
>> is completely divorced from the technology as deployed; much less what
>> we-- in industry-- know to be possible. I don't blame you or anyone in
>> particular on this; it's a new area and we don't yet know what we need
>> to know to know what we need to know; or to the extent that we do it
>> hasn't had time to get effectively communicated.
>>
>> The technical/security implications of larger blocks are related to
>> other things than propagation time, if you assume people are using the
>> available efficient relay protocol (or better).
>>
>> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry).
>> What these parties are actually doing is blinding mining on top of
>> other pools' stratum work. You can think of it as sub-pooling with
>> hopping onto whatever pool has the highest block (I'll call it VFSSP
>> in this post-- validation free stratum subpooling).  It's very easy to
>> implement, and there are other considerations.
>>
>> It was initially deployed at a time when a single pool in Europe has
>> amassed more than half of the hashrate. This pool had propagation
>> problems and a very high orphan rate, it may have (perhaps
>> unintentionally) been performing a selfish mining attack; mining off
>> their stratum work was an easy fix which massively cut down the orphan
>> rates for anyone who did it.  This was before the relay network
>> protocol existed (the fact that all the hashpower was consolidating on
>> a single pool was a major motivation for creating it).
>>
>> VFSSP also cuts through a number of practical issues miners have had:
>> Miners that run their own bitcoin nodes in far away colocation
>> (>100ms) due to local bandwidth or connectivity issues (censored
>> internet); relay network hubs not being anywhere near by due to
>> strange internet routing (e.g. japan to china going via the US for ...
>> reasons...); the CreateNewBlock() function being very slow and
>> unoptimized, etc.   There are many other things like this-- and VFSSP
>> avoids them causing delays even when you don't understand them or know
>> about them. So even when they're easily fixed the VFSSP is a more
>> general workaround.
>>
>> Mining operations are also usually operated in a largely fire and
>> forget manner. There is a long history in (esp pooled) mining where
>> someone sets up an operation and then hardly maintains it after the
>> fact... so some of the use of VFSSP appears to just be inertia-- we
>> have better solutions now, but they they work to deploy and changing
>> things involves risk (which is heightened by a lack of good
>> monitoring-- participants learn they are too latent by observing
>> orphaned blocks at a cost of 25 BTC each).
>>
>> One of the frustrating things about incentives in this space is that
>> bad outcomes are possible even when they're not necessary. E.g. if a
>> miner can lower their orphan rate by deploying a new protocol (or
>> simply fixing some faulty hardware in their infrastructure, like
>> Bitcoin nodes running on cheap VPSes with remote storage)  OR they can
>> lower their orphan rate by pointing their hashpower at a free
>> centeralized pool, they're likely to do the latter because it takes
>> less effort.
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 7205 bytes --]

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Tom Harding]

On 8/6/2015 10:16 AM, Sergio Demian Lerner via bitcoin-dev wrote:
> Is there any up to date documentation about TheBlueMatt relay network
> including what kind of block compression it is currently doing? (apart
> from the source code)
>

Another question.

Did the "relay network" relay
0000000000000000009cc829aa25b40b2cd4eb83dd498c12ad0d26d90c439d99, the
BTC Nuggets block that was invalid post-softfork?  If so,

 - Is there reason to believe that by so doing, it contributed to the
growth of the 2015-07-04 fork?
 - Will the relay network at least validate block version numbers in the
future?

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Gregory Maxwell]

On Thu, Aug 6, 2015 at 6:17 PM, Tom Harding via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
>  - Will the relay network at least validate block version numbers in the
> future?

It already validates block version numbers.

It only relays valid transactions.

Although, the block relaying itself is explicitly "unvalidated" and
the software client can only usefully be used with a mempool
maintaining full node (otherwise it doesn't provide much value,
because the node must wait to validate the things). ... but that
doesn't actually mean no validation at all is performed, many
stateless checks are performed.

On Thu, Aug 6, 2015 at 5:16 PM, Sergio Demian Lerner via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Is there any up to date documentation about TheBlueMatt relay network
> including what kind of block compression it is currently doing? (apart from
> the source code)

I don't know if Matt has an extensive writeup. But the basic
optimization it performs is trivial.  I wouldn't call it compression,
though it does have some analog to RTP "header compression".

All it does is relay transactions verified by a local node and keeps a
FIFO of the relayed transactions in both directions, which is
synchronous on each side.

When a block is recieved on either side, it replaces transactions with
their indexes in the FIFO and relays it along. Transactions not in the
fifo are escaped and sent whole. On the other side the block is
reconstructed using the stored data and handed to the node (where the
preforwarded transactions would have also been pre-validated).

There is some more than basic elaboration for resource management
(e.g. multiple queues for different transaction sizes)-- and more
recently using block templates to learn transaction priority be a bit
more immune to spam attacks, but its fairly simple.

Much better could be done about intelligently managing the queues or
efficiently transmitting the membership sets, etc.  It's just
basically the simplest thing that isn't completely stupid.

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Matt Corallo]

On August 6, 2015 8:42:38 PM GMT+02:00, Gregory Maxwell via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>On Thu, Aug 6, 2015 at 6:17 PM, Tom Harding via bitcoin-dev
><bitcoin-dev@lists.linuxfoundation.org> wrote:
>>  - Will the relay network at least validate block version numbers in
>the
>> future?
>
>It already validates block version numbers.
>
>It only relays valid transactions.
>
>Although, the block relaying itself is explicitly "unvalidated" and
>the software client can only usefully be used with a mempool
>maintaining full node (otherwise it doesn't provide much value,
>because the node must wait to validate the things). ... but that
>doesn't actually mean no validation at all is performed, many
>stateless checks are performed.
>
>On Thu, Aug 6, 2015 at 5:16 PM, Sergio Demian Lerner via bitcoin-dev
><bitcoin-dev@lists.linuxfoundation.org> wrote:
>> Is there any up to date documentation about TheBlueMatt relay network
>> including what kind of block compression it is currently doing?
>(apart from
>> the source code)
>
>I don't know if Matt has an extensive writeup. But the basic
>optimization it performs is trivial.  I wouldn't call it compression,
>though it does have some analog to RTP "header compression".
>
>All it does is relay transactions verified by a local node and keeps a
>FIFO of the relayed transactions in both directions, which is
>synchronous on each side.
>
>When a block is recieved on either side, it replaces transactions with
>their indexes in the FIFO and relays it along. Transactions not in the
>fifo are escaped and sent whole. On the other side the block is
>reconstructed using the stored data and handed to the node (where the
>preforwarded transactions would have also been pre-validated).
>
>There is some more than basic elaboration for resource management
>(e.g. multiple queues for different transaction sizes)-- and more

No, just one queue, but it has a count-of-oversize-txn-limit, in addition to a size.

>recently using block templates to learn transaction priority be a bit
>more immune to spam attacks, but its fairly simple.

Except it doesn't really work :( (see https://github.com/TheBlueMatt/RelayNode/issues/12#issuecomment-128234446)

>Much better could be done about intelligently managing the queues or
>efficiently transmitting the membership sets, etc.  It's just
>basically the simplest thing that isn't completely stupid.

Patches welcome :) (read the issues list first... Rewriting the protocol from scratch is by far not the biggest win here).

Matt

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Matt Corallo]

On August 6, 2015 8:17:35 PM GMT+02:00, Tom Harding via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>On 8/6/2015 10:16 AM, Sergio Demian Lerner via bitcoin-dev wrote:
>> Is there any up to date documentation about TheBlueMatt relay network
>> including what kind of block compression it is currently doing?
>(apart
>> from the source code)
>>
>
>Another question.
>
>Did the "relay network" relay
>0000000000000000009cc829aa25b40b2cd4eb83dd498c12ad0d26d90c439d99, the
>BTC Nuggets block that was invalid post-softfork?  If so,

The version check was only added hours after the initial fork, so it should have (assuming BTC Nuggets or anyone who accepted it is running a client)

> - Is there reason to believe that by so doing, it contributed to the
>growth of the 2015-07-04 fork?

The reason other miners mined on that fork is because they were watching each other's stratum servers, so the relay network should not have had a significant effect. Still, even in a different fork, miners already aggressively relay around the network/between each other, so I'm not so worried.

>- Will the relay network at least validate block version numbers in the
>future?
>
>_______________________________________________
>bitcoin-dev mailing list
>bitcoin-dev@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[Matt Corallo]

[-- Attachment #1: Type: text/plain, Size: 5827 bytes --]

No, don't think so, the protocol is, essentially, relay transactions, when you get a block, send header, iterate over transactions, for each, either use two bytes for nth-recent-transaction-relayed, use 0xffff-3-byte-length-transaction-data. There are quite a few implementation details, and lots of things could be improved, but that is pretty much how it works.

Matt

On August 6, 2015 7:16:56 PM GMT+02:00, Sergio Demian Lerner via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>Is there any up to date documentation about TheBlueMatt relay network
>including what kind of block compression it is currently doing? (apart
>from
>the source code)
>
>Regards, Sergio.
>
>On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev <
>bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp
>> <arnoud@pukaki.bz> wrote:
>> > Thanks for this (direct) feedback. It would make sense that if
>blocks
>> can be
>> > submitted using ~5kb packets, that no further optimizations would
>be
>> needed
>> > at this point. I will look into the relay network transmission
>protocol
>> to
>> > understand how it works!
>> >
>> > I hear that you are saying that this network solves speed of
>transmission
>> > and thereby (technical) block size issues. Presumably it would
>solve
>> speed
>> > of block validation too by prevalidating transactions.
>>
>>
>> Correct. Bitcoin Core has cached validation for many years now... if
>> not for that and other optimizations, things would be really broken
>> right now. :)
>>
>> > Assuming this is all
>> > true, and I have no reason to doubt that at this point, I do not
>> understand
>> > why there is any discussion at all about the (technical) impact of
>large
>> > blocks, why there are large numbers of miners building on invalid
>blocks
>> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining),
>or why
>> > there is any discussion about the speed of block validation (cpu
>> processing
>> > time to verify blocks and transactions in blocks being a
>limitation).
>>
>> I'm also mystified by a lot of the large block discussion, much of it
>> is completely divorced from the technology as deployed; much less
>what
>> we-- in industry-- know to be possible. I don't blame you or anyone
>in
>> particular on this; it's a new area and we don't yet know what we
>need
>> to know to know what we need to know; or to the extent that we do it
>> hasn't had time to get effectively communicated.
>>
>> The technical/security implications of larger blocks are related to
>> other things than propagation time, if you assume people are using
>the
>> available efficient relay protocol (or better).
>>
>> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry).
>> What these parties are actually doing is blinding mining on top of
>> other pools' stratum work. You can think of it as sub-pooling with
>> hopping onto whatever pool has the highest block (I'll call it VFSSP
>> in this post-- validation free stratum subpooling).  It's very easy
>to
>> implement, and there are other considerations.
>>
>> It was initially deployed at a time when a single pool in Europe has
>> amassed more than half of the hashrate. This pool had propagation
>> problems and a very high orphan rate, it may have (perhaps
>> unintentionally) been performing a selfish mining attack; mining off
>> their stratum work was an easy fix which massively cut down the
>orphan
>> rates for anyone who did it.  This was before the relay network
>> protocol existed (the fact that all the hashpower was consolidating
>on
>> a single pool was a major motivation for creating it).
>>
>> VFSSP also cuts through a number of practical issues miners have had:
>> Miners that run their own bitcoin nodes in far away colocation
>> (>100ms) due to local bandwidth or connectivity issues (censored
>> internet); relay network hubs not being anywhere near by due to
>> strange internet routing (e.g. japan to china going via the US for
>...
>> reasons...); the CreateNewBlock() function being very slow and
>> unoptimized, etc.   There are many other things like this-- and VFSSP
>> avoids them causing delays even when you don't understand them or
>know
>> about them. So even when they're easily fixed the VFSSP is a more
>> general workaround.
>>
>> Mining operations are also usually operated in a largely fire and
>> forget manner. There is a long history in (esp pooled) mining where
>> someone sets up an operation and then hardly maintains it after the
>> fact... so some of the use of VFSSP appears to just be inertia-- we
>> have better solutions now, but they they work to deploy and changing
>> things involves risk (which is heightened by a lack of good
>> monitoring-- participants learn they are too latent by observing
>> orphaned blocks at a cost of 25 BTC each).
>>
>> One of the frustrating things about incentives in this space is that
>> bad outcomes are possible even when they're not necessary. E.g. if a
>> miner can lower their orphan rate by deploying a new protocol (or
>> simply fixing some faulty hardware in their infrastructure, like
>> Bitcoin nodes running on cheap VPSes with remote storage)  OR they
>can
>> lower their orphan rate by pointing their hashpower at a free
>> centeralized pool, they're likely to do the latter because it takes
>> less effort.
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>bitcoin-dev mailing list
>bitcoin-dev@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 7226 bytes --]

Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation

[jl2012]

Your proposal fails here:
"If the block defined in the Guarantee Message has not been shown"

What is blockchain? You can see blockchain as a mechanism to prove 
something has been shown by certain order. Therefore, it is not possible 
to prove something has not been shown with blockchain.

Your proposal works only with a centralized trusted party.

Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev 於 2015-08-05 15:07 寫到:
> Hello all.
> 
> We’d like to share an idea we have to dramatically increase the
> bitcoin block propagation speed after a new block has been mined for
> the first time.