public inbox for bitcoindev@googlegroups.com
 help / color / mirror / Atom feed
From: Peter Tschipper <peter.tschipper@gmail.com>
To: Matt Corallo <lf-lists@mattcorallo.com>,
	Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions
Date: Wed, 2 Dec 2015 15:05:10 -0800	[thread overview]
Message-ID: <565F7926.103@gmail.com> (raw)
In-Reply-To: <90EF4E6C-9A71-4A35-A938-EAFC1A24DD24@mattcorallo.com>

[-- Attachment #1: Type: text/plain, Size: 3164 bytes --]


On 30/11/2015 9:28 PM, Matt Corallo wrote:
> I'm really not a fan of this at all. To start with, adding a compression library that is directly accessible to the network on financial software is a really, really scary idea. 
Why scary?  LZO has no current security issues, and it will be
configureable by each node operator so it can be turned off completely
if needed or desired. 
> If there were a massive improvement, I'd find it acceptable, but the improvement you've shown really isn't all that much.
Why is 15% at the low end, to 27% at the high end not good?  It sounds
like a very good boost.   
>  The numbers you recently posted show it improving the very beginning of IBD somewhat over high-latency connections, but if we're throughput-limited after the very beginning of IBD, we should fix that, not compress the blocks. 
I only did the compression up to the 200,000 block to better isolate the
transmission of data from the post processing of blocks and determine
whether the compressing of data was adding to much to the total
transmission time.

I think it's clear from the data that as the data (blocks, transactions)
increase in size that (1) they compress better and (2) they have a
bigger and positive impact on improving performance when compressed.

> Additionally, I'd be very surprised if this had any significant effect on the speed at which new blocks traverse the network (do you have any simulations or other thoughts on this?).
From the table below, at 120000 blocks the time to sync the chain was
roughly the same for compressed vs. uncompressed however after that
point as block sizes start increasing, all compression libraries
peformed much faster than uncompressed. The data provided in this
testing clearly shows that as block size increases, the performance
improvement by compressing data also increases.

TABLE 5:
Results shown in seconds with 60ms of induced latency
Num blks sync'd  Uncmp  Zlib-1  Zlib-6  LZO1x-1  LZO1x-999
---------------  -----  ------  ------  -------  ---------
120000           3226   3416    3397    3266     3302
130000           4010   3983    3773    3625     3703
140000           4914   4503    4292    4127     4287
150000           5806   4928    4719    4529     4821
160000           6674   5249    5164    4840     5314
170000           7563   5603    5669    5289     6002
180000           8477   6054    6268    5858     6638
190000           9843   7085    7278    6868     7679
200000           11338  8215    8433    8044     8795


As far as, what happens after the block is received, then obviously
compression isn't going to help in post processing and validating the
block, but in the pure transmission of the object it most certainly and
logically does and in a fairly direct proportion to the file size (a
file that is 20% smaller will be transmited "at least" 20% faster, you
can use any data transfer time calculator
<http://www.calctool.org/CALC/prof/computing/transfer_time> for that). 
The only issue, that I can see that required testing was to show how
much compression there would be, and how much time the compression of
the data would add to the sending of the data.

 

[-- Attachment #2: Type: text/html, Size: 4499 bytes --]

  parent reply	other threads:[~2015-12-02 23:05 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-11-30 23:12 [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions Peter Tschipper
2015-12-01  5:28 ` Matt Corallo
2015-12-01 20:06   ` Pavel Janík
     [not found]     ` <565E30C6.1010002@bitcartel.com>
2015-12-02  6:47       ` Pavel Janík
2015-12-02  7:33         ` Simon Liu
2015-12-02 18:45           ` Patrick Strateman
2015-12-02 18:57   ` Emin Gün Sirer
2015-12-02 20:16     ` Peter Tschipper
2015-12-02 22:23       ` Matt Corallo
2015-12-02 23:02         ` Peter Tschipper
2015-12-04 13:30           ` Matt Corallo
2015-12-03 19:14     ` Gavin Andresen
2015-12-03 23:07       ` Rusty Russell
2015-12-02 23:05   ` Peter Tschipper [this message]
2015-12-03  5:52     ` Dave Scotese

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=565F7926.103@gmail.com \
    --to=peter.tschipper@gmail.com \
    --cc=bitcoin-dev@lists.linuxfoundation.org \
    --cc=lf-lists@mattcorallo.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox