From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193] helo=mx.sourceforge.net) by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1XD1Cy-0000yD-Ns for bitcoin-development@lists.sourceforge.net; Fri, 01 Aug 2014 01:00:32 +0000 Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.214.170 as permitted sender) client-ip=209.85.214.170; envelope-from=keziahw@gmail.com; helo=mail-ob0-f170.google.com; Received: from mail-ob0-f170.google.com ([209.85.214.170]) by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1XD1Cx-0001u6-SA for bitcoin-development@lists.sourceforge.net; Fri, 01 Aug 2014 01:00:32 +0000 Received: by mail-ob0-f170.google.com with SMTP id wp4so2154566obc.1 for ; Thu, 31 Jul 2014 18:00:26 -0700 (PDT) X-Received: by 10.182.66.130 with SMTP id f2mr2754812obt.84.1406854826079; Thu, 31 Jul 2014 18:00:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.202.61.195 with HTTP; Thu, 31 Jul 2014 18:00:06 -0700 (PDT) In-Reply-To: References: From: Kaz Wesley Date: Thu, 31 Jul 2014 18:00:06 -0700 Message-ID: To: Gregory Maxwell Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.6 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (keziahw[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1XD1Cx-0001u6-SA Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2014 01:00:32 -0000 On Thu, Jul 31, 2014 at 4:18 PM, Gregory Maxwell wrote= : > On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley wrote: >>> the FEC still lets you fill in the missing transactions without knowing= in advance all that will be missing. >> >> I don't see why we need to solve that problem, since the protocol >> already involves exchanging the information necessary to determine >> (with some false positives) what a peer is missing, and needs to >> continue doing so regardless of how blocks are transmitted. > > False positives, and if you have more than one peer=E2=80=94 false negati= ves. > (or a rule for what you must keep which is conservative in order to > avoid creating huge storage requirements=E2=80=94 but then also has false > negatives). I think a rule for what to keep (along with the false-positive rate associated with the rule's conservativeness) is preferable to false negatives, since round-trip cost is potentially very high. The prototype I'm working on will be able to provide data on what the false-positive-missing-tx rate would look like with something like remember-last-N. There are various ways that rule could be upgraded to nearly eliminate the false-positive-missing rate, including learning what txes a peer has dropped via periodic mempool syncing, or specifying the rule explicitly with priority scripts, both of which have other benefits of their own. All of these changes synergize, but as long as the simplistic remembering rule yields worthwhile improvement over the current approach they can all be done independently as incremental improvements. I also really like the idea of the referring to transactions by ids that can themselves provide part of the tx data; I think that could also be added separately, on top of these other changes. (Gregory, your various wiki pages are basically my to-do list of things I'd like to work on.) The idea of mempool synchronization brings up the issue of transaction expiration, since lack of mempool syncing is currently the mechanism for tx expiry. I'm starting a discussion about how to address that in a separate thread; see "deterministic transaction expiration". >> As far as I can tell, channel memory sparseblocks achieve most of the >> possible bandwidth savings, and when FEC-based mempool synchronization >> is implemented its benefits can be applied to the sparseblocks by >> resetting the channel memory to the mutual mempool state each time >> mempool differences are exchanged. Am I missing a benefit to doing FEC >> at block forwarding time that can't be realized by FEC-based mempool >> synchronization, implemented separately from channel-memory based >> index-coding? > > Yes, minimizing latency in the face of multiple peers. > > Otherwise no. And certantly no reason to to implement something simple fi= rst.