From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1WsdQc-0002eM-2H for bitcoin-development@lists.sourceforge.net; Thu, 05 Jun 2014 19:34:22 +0000 Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.223.178 as permitted sender) client-ip=209.85.223.178; envelope-from=pieter.wuille@gmail.com; helo=mail-ie0-f178.google.com; Received: from mail-ie0-f178.google.com ([209.85.223.178]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1WsdQb-0005PK-5I for bitcoin-development@lists.sourceforge.net; Thu, 05 Jun 2014 19:34:22 +0000 Received: by mail-ie0-f178.google.com with SMTP id rl12so1343102iec.9 for ; Thu, 05 Jun 2014 12:34:15 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.50.65.3 with SMTP id t3mr23640621igs.20.1401996855225; Thu, 05 Jun 2014 12:34:15 -0700 (PDT) Received: by 10.50.184.195 with HTTP; Thu, 5 Jun 2014 12:34:15 -0700 (PDT) In-Reply-To: <2ADB9019-825E-410A-ADED-A7237CBC323C@ricmoo.com> References: <34798C1C-FDA7-4A4C-B136-DBD4E59C254D@ricmoo.com> <2ADB9019-825E-410A-ADED-A7237CBC323C@ricmoo.com> Date: Thu, 5 Jun 2014 21:34:15 +0200 Message-ID: From: Pieter Wuille To: Richard Moore Content-Type: text/plain; charset=ISO-8859-1 X-Spam-Score: -1.6 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (pieter.wuille[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1WsdQb-0005PK-5I Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] Future Feature Proposal - getgist X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jun 2014 19:34:22 -0000 On Thu, Jun 5, 2014 at 7:43 PM, Richard Moore wrote: > I was considering names like getcheckpoints() to use the terminology that > already seemed to be in place, but they were too long :) > > I have been using getheaders() in my thick client to quickly grab all the > headers before downloading the full blocks since I can grab more at a time. > Even with getblocks(), there is the case for a getgist() call. Right now > you call getblocks(), which can take some time to get the corresponding > inv(), at which time you can then start the call to getdata() as well as the > next call to getblocks(). > > With a gist, for example of segment_count 50, you could call getgist(), then > with the response, you could request 50 getblocks() each with a > block_locator of 1 hash from the gist (and optimally the stop_hash of the > next hash in the gist) to 50 different peers, providing 25,000 (50 x 500) > block hashes. I don't understand. If you're using getheaders(), there is no need to use getblocks() anymore. You just do a getdata() immediately for the block hashes you have the headers but not the transactions for. In general, I think we should aim for as much verifiability as possible. Much of the reference client's design is built around doing as much validation on received data as soon as possible, to avoid being misled by a particular peer. Getheaders() provides this: you receive a set of headers starting from a point you already know, in order, and can validate them syntactically and for proof-of-work immediately. That allows building a very-hard-to-beat tree structure locally already, at which point you can start requesting blocks along the good branches of that tree immediately - potentially in parallel from multiple peers. In fact, that's the planned approach for the headers-first synchronization. The getgist() proposed here allows the peer to basically give you bullshit headers at the end, and you won't notice until you've downloaded every single block (or at least every single header) up to that point. That risks wasting time, bandwidth and diskspace, depending on implementation. Based on earlier experimenting with my former experimental headersfirst branch, it's quite possible to have 2 mostly independent synchronization mechanisms going on; 1) asking and downloading headers from every peer, and validating them, and 2) asking and downloading blocks from multiple peers in parallel, for blocks corresponding to validated headers. Downloading the headers succeeds within minutes, and within seconds you have enough to start fetching blocks. After that point, you can keep a "download window" full with outstanding block requests, and as blocks go much slower than headers, the headers process never becomes a blocker for blocks to download. Unless we're talking about a system with billions of headers to download, I don't think this is a worthwhile optimization. -- Pieter