From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 3D5ECCC2 for ; Mon, 21 May 2018 08:35:50 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-qt0-f193.google.com (mail-qt0-f193.google.com [209.85.216.193]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id F01C034E for ; Mon, 21 May 2018 08:35:48 +0000 (UTC) Received: by mail-qt0-f193.google.com with SMTP id m16-v6so17941051qtg.13 for ; Mon, 21 May 2018 01:35:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:in-reply-to:references:subject:message-id:date :mime-version; bh=L+hkmMzDaN5Sfu8aBsmkrw5Xeg8rQFCtqHikS7ZyMnU=; b=qyGDlHANLxFUgoXUVBj4Tu7yCG1gyuiAxvqTg3rIBc4Bxtzqc/LAauIUQPtYN6Yv0N igNufAJzQ8R5LtdCxb39EmsMQhy7OhPiQ/vaHzn5lWA+jAIVENUanSzz9+tqulGJF4hl tJE4sy0TUwJtBxGWcGaW+6KejoJp0mKB+dra9Cs52yMYSWhhiQcHdC/zYJDzC00sCeoi n1OUMC5wfaHYxonrY7ePEGylXU5cgczUOOrotWMf+X3jpWxwNMvtZaI3O0mkTs1va2Wl ilohLUTW9/RLZjdZ9C8swh6ZtiUBuT3gD5o7Nv5fBxC6pT2K3sO7/3v8IG5yZhU/CDHo jNzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:in-reply-to:references:subject :message-id:date:mime-version; bh=L+hkmMzDaN5Sfu8aBsmkrw5Xeg8rQFCtqHikS7ZyMnU=; b=kjZzYVbqG9NZiq1mRa7KGYkEvcZiXdYJg/+EryuRpdxFXZvlVzqaFGKtUDhrhaRjw0 XdcsKyfQhE/Hrn5+Iz+SSOxXiG4A3J29tiAYW1bdhO1KFzGWQl7lZJxPbBBNgpZ+AEcP eFaOI66G4aRt1iHM09w2apaW7NDUhvJAJEqELAz+xBQ6IOMW32KR10f9X0fxeUOm0Grn ejYeWD0RzRikUjbipIdmrbuZIJ0yX6xLnqgWMm4J60GyaK9tzs2RP3DugveqBUQG5mRW fW4EfoqtePfmCf6NLlXOEnfuYZXKph8x/rKaBSW7uLrNCWMem209G6mx7nIPbZvDdDNV 3p0w== X-Gm-Message-State: ALKqPwc1ebPSxZ1OXWgPbgU5C2qgWRFO9aTSYG7AiTB/DPHZDdpvJG8K qymUwRxx0ZRnR7xjvJ5F0ij4cj/A X-Google-Smtp-Source: AB8JxZo5EMYphFlx7K1FOTv1O7WwvzwuRhMmXt7nzpcqAh2QwYZk6pem1GupV15BGhJzDWGZe4sDOA== X-Received: by 2002:ac8:1766:: with SMTP id u35-v6mr17690424qtk.209.1526891747609; Mon, 21 May 2018 01:35:47 -0700 (PDT) Received: from [127.0.0.1] (ec2-54-210-254-132.compute-1.amazonaws.com. [54.210.254.132]) by smtp.gmail.com with ESMTPSA id b79-v6sm10440515qka.57.2018.05.21.01.35.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 May 2018 01:35:47 -0700 (PDT) Content-Type: multipart/alternative; boundary="----sinikael-?=_1-15268917468760.3363917884416878" From: =?UTF-8?Q?Johan_Tor=C3=A5s_Halseth?= To: Olaoluwa Osuntokun In-Reply-To: References: <22d375c7-a032-8691-98dc-0e6ee87a4b08@mattcorallo.com> Message-Id: Date: Mon, 21 May 2018 10:35:28 +0200 X-Cm-Message-Id: 1526891729122305aed4c4dc8db302de96b91f53fea313b95b0284d11de711223906206 X-Cm-Draft-Id: WyJhIiwyLCJkcmFmdF9pZCIsIjE1MjY4OTE3MjgyMzkiLCJ2IiwxXQ== X-Cm-Tracking-Code: 2.0/1526891728/b33cfe15f93c4412b9d3504fec672697/2/b53988f48cda345112c1586d420d31cf/076962fa5a1f7737d8541e3d8681273b/0f75828a92f1cf543e26ab700e98c693 X-Mailer: Newton MIME-Version: 1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Mon, 21 May 2018 13:08:31 +0000 Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2018 08:35:50 -0000 ------sinikael-?=_1-15268917468760.3363917884416878 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi all, Most light wallets will want to download the minimum amount of data= required to operate, which means they would ideally download the smallest = possible filters containing the subset of elements they need. What if instead of trying to decide up front which subset of elements will = be most useful to include in the filters, and the size tradeoff, we let the= full-node decide which subsets of elements it serves filters for? For instance, a full node would advertise that it could serve filters for = the subsets 110 (txid+script+outpoint), 100 (txid only), 011 ( = script+outpoint) etc. A light client could then choose to download the = minimal filter type covering its needs. The obvious benefit of this would = be minimal bandwidth usage for the light client, but there are also some = less obvious ones. We wouldn=E2=80=99t have to decide up front what each = filter type should contain, only the possible elements a filter can contain= (more can be added later without breaking existing clients). This, I think= , would let the most served filter types grow organically, with full-node = implementations coming with sane defaults for served filter types (maybe = even all possible types as long as the number of elements is small), = letting their operator add/remove types at will. The main disadvantage of = this as I see it, is that there=E2=80=99s an exponential blowup in the = number of possible filter types in the number of element types. However, = this would let us start out small with only the elements we need, and in = the worst case the node operators just choose to serve the subsets = corresponding to what now is called =E2=80=9Cregular=E2=80=9D + = =E2=80=9Cextended=E2=80=9D filters anyway, requiring no more resources. This would also give us some data on what is the most widely used filter = types, which could be useful in making the decision on what should be part = of filters to eventually commit to in blocks. - Johan On Sat, May 19, 2018 = at 5:12, Olaoluwa Osuntokun via bitcoin-dev wrote: On Thu, May 17, 2018 at 2:44 PM Jim Posen via = bitcoin-dev
Hi all,

Mos= t light wallets will want to download the minimum amount of data required = to operate, which means they would ideally download the smallest possible = filters containing the subset of elements they need.=  

What if instead = of trying to decide up front which subset of elements will be most useful = to include in the filters, and the size tradeoff, we let the full-node = decide which subsets of elements it serves filters for?=

For instance, a full node would advertise that it = could serve filters for the subsets 110 (txid+script+outpoint), 100 (txid = only), 011 (script+outpoint) etc. A light client could then choose = to download the minimal filter type covering its needs.=  

The obvious benefit of this would be = minimal bandwidth usage for the light client, but there are also some less = obvious ones. We wouldn=E2=80=99t have to decide up front what each filter = type should contain, only the possible elements a filter can contain (more = can be added later without breaking existing clients). This, I think, would= let the most served filter types grow organically, with full-node = implementations coming with sane defaults for served filter types (maybe = even all possible types as long as the number of elements is small), = letting their operator add/remove types at will.

T= he main disadvantage of this as I see it, is that there=E2=80=99s an = exponential blowup in the number of possible filter types in the number of = element types. However, this would let us start out small with only the = elements we need, and in the worst case the node operators just choose to = serve the subsets corresponding to what now is called = =E2=80=9Cregular=E2=80=9D + =E2=80=9Cextended=E2=80=9D filters anyway, = requiring no more resources.

This would also give = us some data on what is the most widely used filter types, which could be = useful in making the decision on what should be part of filters to = eventually commit to in blocks.

- = Johan
On Sat, May 19, 2018 at = 5:12, Olaoluwa Osuntokun via bitcoin-dev <bitcoin-dev@lists.= linuxfoundation.org> wrote:
On Thu, May 17, 2018 at 2:44 PM Jim = Posen via bitcoin-dev <bitcoin-
Monitoring inputs by scriptPubkey vs input-txid = also has a massive
advantage for parallel filtering: You can usually = known your pubkeys
well in advance, but if you have to change what = you're watching block
N+1 for based on the txids that paid you in N = you can't filter them
in parallel.

Yes, I'll grant that this is a benefit of your = suggestion.

Yeah = parallel filtering would be pretty nice. We've implemented a serial = filtering for btcwallet [1] for the use-case of rescanning after a seed = phrase import. Parallel filtering would help here, but also we don't yet = take advantage of batch querying for the filters themselves. This would = speed up the scanning by quite a bit.

I really = like the filtering model though, it really simplifies the code, and we can = leverage identical logic for btcd (which has RPCs to fetch the filters) as = well.


_______________________________________________ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
------sinikael-?=_1-15268917468760.3363917884416878--