From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 02788721 for ; Wed, 11 May 2016 20:17:04 +0000 (UTC) X-Greylist: delayed 00:10:11 by SQLgrey-1.7.6 Received: from mcelrath.org (moya.mcelrath.org [50.31.3.130]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 74AD7162 for ; Wed, 11 May 2016 20:17:03 +0000 (UTC) Received: from mcelrath.org (localhost [127.0.0.1]) by mcelrath.org (8.14.3/8.14.3/Debian-9.4) with ESMTP id u4BK6nWO020517 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 11 May 2016 20:06:49 GMT Received: (from mcelrath@localhost) by mcelrath.org (8.14.3/8.14.3/Submit) id u4BK6mUi020516; Wed, 11 May 2016 20:06:48 GMT X-Authentication-Warning: mcelrath.org: mcelrath set sender to bob_bitcoin@mcelrath.org using -f Date: Wed, 11 May 2016 20:06:48 +0000 From: Bob McElrath To: bfd@cock.lu Message-ID: <20160511200648.GQ20063@mcelrath.org> References: <71d822e413ac457a530e1c367811cc24@cock.lu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="61jdw2sOBCFtR2d/" Content-Disposition: inline In-Reply-To: <71d822e413ac457a530e1c367811cc24@cock.lu> User-Agent: Mutt/1.5.20 (2009-06-14) X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: bitcoin-dev@lists.linuxfoundation.org Subject: Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 May 2016 20:17:04 -0000 --61jdw2sOBCFtR2d/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I like this idea, but let's run some numbers... bfd--- via bitcoin-dev [bitcoin-dev@lists.linuxfoundation.org] wrote: > A Bloom Filter Digest is deterministically created of every block Bloom filters completely obfuscate the required size of the filter for a desired false-positive rate. But, an optimal filter is linear in the number of elements it contains for fixed false-positive rate, and logarithmic in the false-positive rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but that's not the only way) That is for N elements and false positive rate \epsilon: filter size = - N \log_2 \epsilon Given that the data that would be put into this particular filter is *already* hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a fixed false-positive rate, given expected wallet sizes. For Bloom filters, multiply the above formula by 1.44. To prevent light clients from downloading more blocks than necessary, the false-positive rate should be roughly less than 1/(block height). If we take the false positive rate to be 1e-6 for today's block height ~ 410000, this is about 20 bits per element. So for todays block's, this is a 30kb filter, for a 3% increase in block size, if blocks commit to the filter. Thus the required size of the filter commitment is roughly: filter size = N \log_2 H where H is the block height. If bitcoin had these filters from the beginning, a light client today would have to download about 12MB of data in filters. My personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth win, though it's definitely a win for computing load on full nodes. [1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf -- Cheers, Bob McElrath "For every complex problem, there is a solution that is simple, neat, and wrong." -- H. L. Mencken --61jdw2sOBCFtR2d/ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iEYEARECAAYFAlczkNgACgkQjwioWRGe9K0O9wCeMNWYnsG1xZ0XNnVulIVaISRS E9AAoJMYcm+ocbcCwl5kwbJa6B6Z22Ly =ArdB -----END PGP SIGNATURE----- --61jdw2sOBCFtR2d/--