From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id EC11D8A5 for ; Mon, 9 May 2016 12:12:16 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-lf0-f50.google.com (mail-lf0-f50.google.com [209.85.215.50]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C6AE61FF for ; Mon, 9 May 2016 12:12:15 +0000 (UTC) Received: by mail-lf0-f50.google.com with SMTP id m64so197226063lfd.1 for ; Mon, 09 May 2016 05:12:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to; bh=ViuEaPmfOA7Rdd1SKhnzlsa7FSaxNhuOL9bBRcBZO74=; b=wwcLpavV4HLpVOe2toOpJOYdmlWMSJ1eN3U2j52l4rheihIyCV3tfsk3HONAlSy55o e540/gxWAu2oc6WHot/+g9pl47eVanw1Nk6KKFqiCWsPS4lHoEydjw79F5+yfG4pWDkV w/q2Jrm6gep6nLE+7jrXFQe7qFd//kZgsMS78nOOIzm0xSh4jBi08k+MZqJoE2QYKVtu 1sCut3gmLRHdU89kC2dDDEjR3b4gPIphdtX8nSdpBCMUc5BLbUVZU7hTTAeP1578aYw8 cLLnnFLSOhlxhiehO1g6jW6Q7HD8zqpcN/D70qO6QOa6acKZNrxXUVy7IPbNAgfmo5Bk JJfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to; bh=ViuEaPmfOA7Rdd1SKhnzlsa7FSaxNhuOL9bBRcBZO74=; b=dYdAm2yUHdRkLn46l9uehUU3BLPNNiwkF1kAoa1gfPkHsM50RRq4wn3TJamFL6pFKh B0kKE0HMcm+Uzc3fKPWY6haL7pL7kfP1Ox0QXTPMOdl7je2oDYXOKXMUHbdSr8KLnaxD DbjnXUbbOhiaaEPqSgkqthBYjoHovnTq4o9v+obky46Unh59YiH1DCBIGW0dZ7hbcJLA 98FfUWZZD1XHx4RF9eX84041dtVbYagUuLcis7WwaqZA0Uv+YTu5nr53IS09AXNZSchE DbZXZ/koZrlA501FF5T8GNTFufyMkvfV1BAYWydsrovB3pUhkwVcSa5Waypl/wsQ3ie9 IZkw== X-Gm-Message-State: AOPr4FXPWzgWUjVqZoGks6sgg3cw6IToTTeXLm6YWzElCNuKZxQDrXuf/yYCVl+um44n2Sz3kRj0pxRWklLvTA== MIME-Version: 1.0 X-Received: by 10.25.77.204 with SMTP id a195mr15660400lfb.14.1462795934294; Mon, 09 May 2016 05:12:14 -0700 (PDT) Sender: gmaxwell@gmail.com Received: by 10.112.65.108 with HTTP; Mon, 9 May 2016 05:12:14 -0700 (PDT) In-Reply-To: References: <5727D102.1020807@mattcorallo.com> <86058327.pdmfHP132A@kiwi> <2273040.Bd6rtJjYLF@garp> Date: Mon, 9 May 2016 12:12:14 +0000 X-Google-Sender-Auth: LFrjWxBDG12ls7g9auV05DMmX4Y Message-ID: From: Gregory Maxwell To: Bitcoin Dev Content-Type: text/plain; charset=UTF-8 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [bitcoin-dev] Fwd: Compact Block Relay BIP X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 May 2016 12:12:17 -0000 On Mon, May 9, 2016 at 11:32 AM, Tom wrote: > On Monday 09 May 2016 10:43:02 Gregory Maxwell wrote: >> On Mon, May 9, 2016 at 9:35 AM, Tom Zander via bitcoin-dev >> wrote: >> > You misunderstand the networking effects. >> > The fact that your node is required to choose which one to set the >> > announce >> > bit on implies that it needs to predict which node will have the best data >> > in the future. >> >> Not required. It may. > > It is required, in the reference of wanting to actually use compact block > relay. I cannot parse this sentence. A node implementing this does not have to ask peers to send blocks without further solicitation. If they don't, their minimum transfer time increases to the current 1.5 RTT (but sending massively less data). > Apologies, I thought that the term was wider known. "Laboratory situations" > is used where I am from as the opposite of real-world messy and unpredictable > situations. > > So, your measurements may be true, but are not useful to decide how well it > behaves under less optimal situations. aka "the real world". My measurements were made in the real world, on a collection of nodes around the network which were not setup for this purpose and are running standard configurations, over many weeks of logs. This doesn't guarantee that they're representative of everything-- but they don't need to be. >> This also _increases_ robustness. Right now a single peer failing at >> the wrong time will delay blocks with a long time out. > > If your peers that were supposed to send you a compact block fail, then you'll > end up in exactly that same situation again. Only with various timeouts in > between before you get your block making it a magnitude slower. That is incorrect. If a header shows up and a compact block has not shown up, a compact block will be requested. If compactblock shows up reconstruction will be attempted. If any of the requested compact blocks show up (the three in advance, if high bandwidth mode is used, or a requested one, if there was one) then reconstruction proceeds without delay. The addition of the unsolicited input causes no additional timeouts or delays (ignoring bandwidth usage). It does use some more bandwidth than not having it, but still massively less than the status quo. >> > Another problem with your solution is that nodes send a much larger amount >> > of unsolicited data to peers in the form of the thin-block compared to >> > the normal inv or header-first data. >> >> "High bandwidth" mode > > Another place where I may have explained better. > This is not about the difference about the two modes of your design. > This is about the design as a whole. As compared to current. It is massively more efficient than the current protocol, even under fairly poor conditions. In the absolute worst possible case (miner sends a block of completely unexpected transactions, and three peers send compact blocks, it adds about 6% overhead) > Service bits are exactly the right solution to indicate additional p2p > feature-support. With this kind of unsubstantiated axiomatic assertion, I don't think further discussion with you is likely to be productive-- at least I gave a reason. > That's all fine and well, it doesn't at any point take away from my point that > any specification should NOT invent something new that has for decades had a > great specification already. UTF-8 would be a poor fit here for the reasons I explained and others less significant ones (including the additional error cases that must be handled resulting from the inefficient encoding; -- poor handing of invalid UTF-8 have even resulted in security issues in some applications). I am a bit baffled that you'd suggest using UTF-8 as a general compact integer encoding in a binary protocol in the first place. >> > Just the first (highest) 8 bytes of a sha256 hash. >> > >> > The amount of collisions will not be less if you start xoring the rest. >> > The whole reason for doing this extra work is also irrelevant as a spam >> > protection. >> >> Then you expose it to a trivial collision attack: To find two 64 bit >> hashes that collide I need perform only roughly 2^32 computation. Then >> I can send them to the network. > > No, you still need to have done a POW. > > Next to that, your scheme is 2^32 computations *and* some XORs. The XORs are > percentage wise a rounding error on the total time. So your argument also > destroys your own addition. > >> This issue is eliminated by salting the hash. > > The issue is better eliminated by not allowing nodes to send uninvited large > messages. What are you talking about? You seem profoundly confused here. There is no proof of work involved anywhere. I obtain some txouts. I write a transaction spending them in malleable form (e.g. sighash single and an op_return output).. then grind the extra output to produce different hashes. After doing this 2^32 times I am likely to find two which share the same initial 8 bytes of txid. I send one to half the nodes, the other to half the nodes. When a block shows up carrying one or the other of my transactions reconstruction will fail on half the nodes in the network in a protocol with a simple truncated hash. Of course, doing this is easy, so I can keep it going persistently. If I am a miner, I can be sure to filter these transactions from my own blocks-- causing all my competition to suffer higher orphaning. The salted short-ids do not have this easily exploited, and gratuitous vulnerability. This was obvious enough that it this feature was in the very earliest descriptions of these techniques in 2013/2014. The salted short-ids cannot be collided in pre-computation, and cannot be collided with respect to multiple nodes at once.