From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 53DBDB15 for ; Fri, 26 Feb 2016 05:56:58 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-io0-f180.google.com (mail-io0-f180.google.com [209.85.223.180]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 12DFB11F for ; Fri, 26 Feb 2016 05:56:57 +0000 (UTC) Received: by mail-io0-f180.google.com with SMTP id z135so113231744iof.0 for ; Thu, 25 Feb 2016 21:56:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc; bh=c3JO4WRDiqq9k2kYop9lmo0rUjWj8tp4URO2QjNk0mc=; b=IOPCve0z3V9+K8M1YT+haaOkxH9LfF/5AwcsmmyDNd/ykvinFk3MgFLQ2CpofV7nTX RVGjZOoGvioJhPiJTLtXsVkMb9AjBvRxceW+rZrwgC8OIBN77YkMDR6LD0SkQePwuYe2 6FlaD0fIdMkx2lyldPA3VgpO7YmjAV1l5OIdZtu6WIePrkKB/BpHonTqmYfDSS95FFC1 d6IXQhfNRaIilPa7xg9Zd3Mc16R7fqIzRGgwbIRCQcKZsylDRiEYeLJCQ5PDJr5OuuB0 w/kMHSsHdAui3NWDQugd+PqeX7ifBrMeWcvfWdU1EBGBeKKv6zKO3oKaO1uGFtDJhQDO w3ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc; bh=c3JO4WRDiqq9k2kYop9lmo0rUjWj8tp4URO2QjNk0mc=; b=dVVMjpKbU32zH5f8O3IploixxerhU5yratMKvZ6ER7OPtVnd1SXc24i10Z+W2hM3m5 ai3U/nchejZgB/7Pa1WaXqPYDSbdGWMBo/n0Db7+fHr2ChbsF1n/cyxAks8Qt5RfUDwP WLRdeRh7B+vPNSHTPC7EiPBo3SfbTaT4gp3f44wPwHasFnjKqeNTCF1aVdp8PnfnramI QWAezfqp1/TbS7JIXCeJck9wZ5b1vv8oOxvxp2Nt/3xGBKgD1Bb89YTPcO2AovxPXOx4 0QQasfJUKs4q5ZHhoX8NLtiiYM2m6+n47EgubaB3ehPXhv9PZww37Xm9ogGCIChhet/K QeEw== X-Gm-Message-State: AG10YOTbXz+/ui4TBmvD1sS+h45s4Nu6er7FBS4R8FVSYresFUJyROF1EaffDN3r7/AkxJ1Za8lxnvf2YHMPVg== MIME-Version: 1.0 X-Received: by 10.107.132.142 with SMTP id o14mr6500401ioi.75.1456466216559; Thu, 25 Feb 2016 21:56:56 -0800 (PST) Sender: gmaxwell@gmail.com Received: by 10.107.132.75 with HTTP; Thu, 25 Feb 2016 21:56:56 -0800 (PST) In-Reply-To: References: Date: Fri, 26 Feb 2016 05:56:56 +0000 X-Google-Sender-Auth: CjOlxStZYW3hGY077rOMO8OsLAI Message-ID: From: Gregory Maxwell To: Jonathan Toomim Content-Type: text/plain; charset=UTF-8 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev Subject: Re: [bitcoin-dev] INV overhead and batched INVs to reduce full node traffic X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 05:56:58 -0000 On Fri, Feb 26, 2016 at 5:35 AM, Jonathan Toomim via bitcoin-dev wrote: > An improvement that I've been thinking about implementing (after > Blocktorrent) is an option for batched INVs. Including the hashes for two > txes per IP packet instead of one would increase the INV size to 229 bytes > for 64 bytes of payload -- that is, you add 36 bytes to the packet for every > 32 bytes of actual payload. This is a marginal efficiency of 88.8% for each > hash after the first. This is *much* better. > > Waiting a short period of time to accumulate several hashes together and > send them as a batched INV could easily reduce the traffic of running > bitcoin nodes by a factor of 2, Copying my response to you from BitcoinTalk (https://bitcointalk.org/index.php?topic=1377345.msg14013294#msg14013294): Uh. Bitcoin has done this since the very early days. The batching was temporarily somewhat hobbled between 0.10 and 0.12 (especially when you had any abusive frequently pinging peers attached), but is now fully functional again and it now manages to batch many transactions per INV pretty effectively. Turn on net message debugging and you'll see the many INVs that are much larger than the minimum. The average batching size (ignoring the trickle cut-through) is about 5 seconds long-- and usually gets about 10 transactions per INV. My measurements were with these fixes in effect; I expect the blocksonly savings would have been higher otherwise. 2016-02-26 05:47:08 sending: inv (1261 bytes) peer=33900 2016-02-26 05:47:08 sending: inv (109 bytes) peer=32460 2016-02-26 05:47:08 sending: inv (37 bytes) peer=34501 2016-02-26 05:47:08 sending: inv (217 bytes) peer=33897 2016-02-26 05:47:08 sending: inv (145 bytes) peer=41863 2016-02-26 05:47:08 sending: inv (37 bytes) peer=35725 2016-02-26 05:47:08 sending: inv (73 bytes) peer=20567 2016-02-26 05:47:08 sending: inv (289 bytes) peer=44703 2016-02-26 05:47:08 sending: inv (73 bytes) peer=13408 2016-02-26 05:47:09 sending: inv (649 bytes) peer=41279 2016-02-26 05:47:09 sending: inv (145 bytes) peer=42612 2016-02-26 05:47:09 sending: inv (325 bytes) peer=34525 2016-02-26 05:47:09 sending: inv (181 bytes) peer=41174 2016-02-26 05:47:09 sending: inv (469 bytes) peer=41460 2016-02-26 05:47:10 sending: inv (973 bytes) peer=133 2016-02-26 05:47:10 sending: inv (361 bytes) peer=20541 Twiddling here doesn't change the asymptotic efficiency though; which is what my post is about. [I'm also somewhat surprised that you were unaware of this; one of the patches "classic" was talking about patching out was the one restoring the batching... due to a transaction deanonymization service (or troll) claiming it interfered with their operations.]