public inbox for bitcoindev@googlegroups.com
 help / color / mirror / Atom feed
From: Peter Tschipper <peter.tschipper@gmail.com>
To: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"
Date: Tue, 10 Nov 2015 08:17:40 -0800	[thread overview]
Message-ID: <564218A4.8070102@gmail.com> (raw)
In-Reply-To: <5642172C.701@gmail.com>

[-- Attachment #1: Type: text/plain, Size: 5481 bytes --]

On 10/11/2015 8:11 AM, Peter Tschipper wrote:
> On 10/11/2015 1:44 AM, Tier Nolan via bitcoin-dev wrote:
>> The network protocol is not quite consensus critical, but it is
>> important.
>>
>> Two implementations of the decompressor might not be bug for bug
>> compatible.  This (potentially) means that a block could be designed
>> that won't decode properly for some version of the client but would
>> work for another.  This would fork the network.
>>
>> A "raw" network library is unlikely to have the same problem.
>>
>> Rather than just compress the stream, you could compress only block
>> messages only.  A new "cblock" message could be created that is a
>> compressed block.  This shouldn't reduce efficiency by much.
>>
> I chose the more generic datastream compression so we could in the
> future apply to possibly to transactions but currently all that is
> planned, is to compress blocks, and that was really my only original
> intent until I saw that there might be some bandwidth savings for
> transactions as well. 
>
> The compression  however could be applied to any datastream but is not
> *forced* .  Basically it would just be a method call in CDatastream so
> we could do ss.compress and ss.decompress and apply that to blocks and
> possibly transactions if worthwhile and only IF compression is turned
> on.  But there is no intend to apply this to every type of message
> since most would be too small to benefit from compression.
>
> Here are some results of using the code in the PR to
> compress/decompress blocks using zlib compression level = 6.  This
> data was taken from the first 275K blocks in the mainnet blockchain. 
> Clearly once we get past 10KB we get pretty decent compression but
> even below that there is some benefit.  I'm still collecting data and
> will get the same for the whole blockchain.
>
> range = block size range
> ubytes = average size of uncompressed blocks
> cbytes = average size of compressed blocks
> ctime = average time to compress
> dtime = average time to decompress
> cmp_ratio% = compression ratio
> datapoints = number of datapoints taken
>
> range       ubytes    cbytes    ctime    dtime    cmp_ratio%    datapoints
> 0-250b      215         189    0.001    0.000    12.41            79498
> 250-500b    440         405    0.001    0.000    7.82            11903
> 500-1KB     762         702    0.001    0.000    7.83            10448
> 1KB-10KB    4166    3561    0.001    0.000    14.51            50572
> 10KB-100KB  40820    31597    0.005    0.001    22.59            75555
> 100KB-200KB 146238    106320    0.015    0.001    27.30            25024
> 200KB-300KB 242913    175482    0.025    0.002    27.76            20450
> 300KB-400KB 343430    251760    0.034    0.003    26.69            2069
> 400KB-500KB 457448    343495    0.045    0.004    24.91            1889
> 500KB-600KB 540736    424255    0.056    0.007    21.54            90
> 600KB-700KB 647851    506888    0.063    0.007    21.76            59
> 700KB-800KB 749513    586551    0.073    0.007    21.74            48
> 800KB-900KB 859439    652166    0.086    0.008    24.12            39
> 900KB-1MB   952333    725191    0.089    0.009    23.85            78
>
>> If a client fails to decode a cblock, then it can ask for the block
>> to be re-sent as a standard "block" message. 
> interesting idea.
>>
>> This means that it is a pure performance improvement.  If problems
>> occur, then the client can just switch back to uncompressed mode for
>> that block.
>>
>> You should look into the block relay system.  This gives a larger
>> improvement than simply compressing the stream.  The main benefit is
>> latency but it means that actual blocks don't have to be sent, so
>> gives a potential 50% compression ratio.  Normally, a node receives
>> all the transactions and then those transactions are included later
>> in the block.
>>
> There are better ways of sending new blocks, that's certainly true but
> for sending historical blocks and seding transactions I don't think
> so.  This PR is really designed to save bandwidth and not intended to
> be a huge performance improvement in terms of time spent sending.
>>
>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>>     On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev
>>     <bitcoin-dev@lists.linuxfoundation.org
>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>      
>>
>>         I think 25% bandwidth savings is certainly considerable,
>>         especially for people running full nodes in countries like
>>         Australia where internet bandwidth is lower and there are
>>         data caps.
>>
>>
>>     ​This reinforces the idea that such trade-off decisions should be
>>     be local and negotiated between peers, not a required feature of
>>     the network P2P.​
>>      
>>
>>     -- 
>>     Johnathan Corgan
>>     Corgan Labs - SDR Training and Development Services
>>     http://corganlabs.com
>>
>>     _______________________________________________
>>     bitcoin-dev mailing list
>>     bitcoin-dev@lists.linuxfoundation.org
>>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>     https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>


[-- Attachment #2: Type: text/html, Size: 11847 bytes --]

  parent reply	other threads:[~2015-11-10 16:17 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-11-09 19:18 [bitcoin-dev] request BIP number for: "Support for Datastream Compression" Peter Tschipper
2015-11-09 20:41 ` Johnathan Corgan
2015-11-09 21:04 ` Bob McElrath
2015-11-10  1:58   ` gladoscc
2015-11-10  5:40     ` Johnathan Corgan
2015-11-10  9:44       ` Tier Nolan
     [not found]         ` <5642172C.701@gmail.com>
2015-11-10 16:17           ` Peter Tschipper [this message]
2015-11-10 16:21             ` Jonathan Toomim
2015-11-10 16:30           ` Tier Nolan
2015-11-10 16:46             ` Jeff Garzik
2015-11-10 17:09               ` Peter Tschipper
2015-11-11 18:35               ` Peter Tschipper
2015-11-11 18:49                 ` Marco Pontello
2015-11-11 19:05                   ` Jonathan Toomim
2015-11-13 21:58                     ` [bitcoin-dev] Block Compression (Datastream Compression) test results using the PR#6973 compression prototype Peter Tschipper
2015-11-18 14:00                       ` [bitcoin-dev] More findings: " Peter Tschipper
2015-11-11 19:11                   ` [bitcoin-dev] request BIP number for: "Support for Datastream Compression" Peter Tschipper
2015-11-28 14:48               ` [bitcoin-dev] further test results for : "Datastream Compression of Blocks and Tx's" Peter Tschipper
2015-11-29  0:30                 ` Jonathan Toomim
2015-11-29  5:15                   ` Peter Tschipper
     [not found]             ` <56421F1E.4050302@gmail.com>
2015-11-10 16:46               ` [bitcoin-dev] request BIP number for: "Support for Datastream Compression" Peter Tschipper

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=564218A4.8070102@gmail.com \
    --to=peter.tschipper@gmail.com \
    --cc=bitcoin-dev@lists.linuxfoundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox