From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 40C697A8 for ; Sat, 19 May 2018 03:06:24 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-oi0-f45.google.com (mail-oi0-f45.google.com [209.85.218.45]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 9A736B0 for ; Sat, 19 May 2018 03:06:23 +0000 (UTC) Received: by mail-oi0-f45.google.com with SMTP id b130-v6so8774566oif.12 for ; Fri, 18 May 2018 20:06:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FR6J0tAYy2F3YucOEtAkMRzrocede2dBsefHwIFMjl8=; b=RGCKVwu7xYXUhXmkPn3VXtO1Hlwt88tXl0Vi31GwTFUoNnV9mVDcGzNLRJxlqjs621 hGWtRPyC1hJ3yEA1T4eWsvL17FELApp5gbxh4WqtLs8SeiN4+CYjdnTA/eRsBNr33jrl D7nNj/1PS8R4QejwjUcF+HV691atxiksLKieEMOqGRMGPmyaTATOP5tDzX5CE/NI9wLz 0PHpR3XJa+u/Ugxry47QxUQlzBCP1d1LtNFA5ahHtIdTNVydVjLULNBlT40lGLEIV+eI ZgehYhziUum/hEq0+JFBfkbL39gntl6ZMFFa9qRBoqGiIC0hI8wY/PRNLGk5Q5cClo4K irgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FR6J0tAYy2F3YucOEtAkMRzrocede2dBsefHwIFMjl8=; b=UdB0H7NP41dFcC27DeQlbo7ZI+C4xuLvW8FKzM5bKJewO35RWj9V9eZA/IUiiD8db5 Z9FMVIubI9s3BTaEf29Xzyt1k2maMZjs53xD5307Rb2yRPoiFy3u9gSpP8XIgYVRHEvk kFzCXq9zey9pbG5Oli7Tv5HA84BwmFXQsTCvIvSErKytp1IKKRLsp0F8GQadqc10eLQy 85tO31UXI4mN+AuFZSnWQsr2BzNpXG09/zZ5GoCEt8UHRrcLwzJoGDsCG9TT2o1jOtk+ UFqSU/SW4oXcTjUJ0Q62oIw7VUhzr/LPx2RMM79h7ep4/zzRsbh3vOQ00OhJrbkncSgJ rwhQ== X-Gm-Message-State: ALKqPwdBPBGJx/wCHbuu4+MyY18cBU3qO6G0pyq0tejlyPMs0+qIo+xf m3qjFm1HJxt3JJDMsgQI9k3f9zH8sW727N6ozRk= X-Google-Smtp-Source: AB8JxZoAZSHEIPBFfmK8nXsJsG3qAczJlJedRLr1MeVZa4X/5SN64J1PkOiYbOk9JfQR4jiZTcHraj5+VKw+xvVpZeU= X-Received: by 2002:aca:f12:: with SMTP id 18-v6mr7377507oip.46.1526699182757; Fri, 18 May 2018 20:06:22 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Pieter Wuille Date: Fri, 18 May 2018 20:06:10 -0700 Message-ID: To: Olaoluwa Osuntokun , Bitcoin Dev Content-Type: multipart/alternative; boundary="00000000000077a485056c865a77" X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 May 2018 03:06:24 -0000 --00000000000077a485056c865a77 Content-Type: text/plain; charset="UTF-8" On Fri, May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Greg wrote: > > What about also making input prevouts filter based on the scriptpubkey > being > > _spent_? Layering wise in the processing it's a bit ugly, but if you > > validated the block you have the data needed. > > AFAICT, this would mean that in order for a new node to catch up the filter > index (index all historical blocks), they'd either need to: build up a > utxo-set in memory during indexing, or would require a txindex in order to > look up the prev out's script. The first option increases the memory load > during indexing, and the second requires nodes to have a transaction index > (and would also add considerable I/O load). When proceeding from tip, this > doesn't add any additional load assuming that your synchronously index the > block as you validate it, otherwise the utxo set will already have been > updated (the spent scripts removed). > I was wondering about that too, but it turns out that isn't necessary. At least in Bitcoin Core, all the data needed for such a filter is in the block + undo files (the latter contain the scriptPubKeys of the outputs being spent). I have a script running to compare the filter sizes assuming the regular > filter switches to include the prev out's script rather than the prev > outpoint itself. The script hasn't yet finished (due to the increased I/O > load to look up the scripts when indexing), but I'll report back once it's > finished. > That's very helpful, thank you. Cheers, -- Pieter --00000000000077a485056c865a77 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Fri, = May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.= org> wrote:
Greg wrote:
> What about also making input prevouts filt= er based on the scriptpubkey being
> _spent_?=C2=A0 Layering w= ise in the processing it's a bit ugly, but if you
> valida= ted the block you have the data needed.

AFAICT, th= is would mean that in order for a new node to catch up the filter
index (index all historical blocks), they'd either need to: build up a=
utxo-set in memory during indexing, or would require a txindex i= n order to
look up the prev out's script. The first option in= creases the memory load
during indexing, and the second requires = nodes to have a transaction index
(and would also add considerabl= e I/O load). When proceeding from tip, this
doesn't add any a= dditional load assuming that your synchronously index the
block a= s you validate it, otherwise the utxo set will already have been
= updated (the spent scripts removed).

I was wondering about that too,= but it turns out that isn't necessary. At least in Bitcoin Core, all t= he data needed for such a filter is in the block + undo files (the latter c= ontain the scriptPubKeys of the outputs being spent).

I have a script running to compare the = filter sizes assuming the regular
filter switches to include the = prev out's script rather than the prev
outpoint itself. The s= cript hasn't yet finished (due to the increased I/O
load to l= ook up the scripts when indexing), but I'll report back once it's
finished.
That's very helpful, thank you.

Cheers,

--=C2=A0
Pieter

--00000000000077a485056c865a77--