[bitcoin-dev] BloomFilter issue with segwit addresses

[bitcoin-dev] BloomFilter issue with segwit addresses

[Andreas Schildbach]

Anton, a developer on the bitcoinj maiing list, recently made me aware
[1] of a compatibility issue between segwit and BIP37 (Bloom Filtering).

The issue affects only P2WPKH and the special case of transactions
without change outputs (such as when emptying a wallet). In this case,
neither inputs not outputs contain any data elements that would cause a
match for the filter. The public key, which would match, goes to the
witness but not to the input.

My suggestion was to include an OP_RETURN output with a matching public
key in such transactions. Anton confirmed that this workaround is indeed
working. But of course it nullifies some of the segwit's size improvements.

I wonder if Bitcoin Core would be willing to extend the BIP37 matching
rules such that data elements in the witness are also matched against?


[1] https://groups.google.com/d/msg/bitcoinj/SJpLgjowc1I/V7u2BavvAwAJ

Re: [bitcoin-dev] BloomFilter issue with segwit addresses

[Jonas Schnelli]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

Hi Andreas

Thanks for bringing this up and this seems indeed to be suboptimal.

> I wonder if Bitcoin Core would be willing to extend the BIP37 matching
> rules such that data elements in the witness are also matched against?

Bitcoin Core is not an identity that can be „willing to extend“ (or reject) a feature.
Someone needs to come up with a proposal (pull request).

Maybe an extension for BIP37 would make sense (*meh*).
Just inserting the witness data into the bloom filter seems to be an easy solution (CBloomFilter::IsRelevantAndUpdate())

/jonas

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [bitcoin-dev] BloomFilter issue with segwit addresses

[Jim Posen]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

Why not add the outpoints owned by the wallet to the filter and watch for
those instead of elements in the input script or witness data?

On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi Andreas
>
> Thanks for bringing this up and this seems indeed to be suboptimal.
>
> > I wonder if Bitcoin Core would be willing to extend the BIP37 matching
> > rules such that data elements in the witness are also matched against?
>
> Bitcoin Core is not an identity that can be „willing to extend“ (or
> reject) a feature.
> Someone needs to come up with a proposal (pull request).
>
> Maybe an extension for BIP37 would make sense (*meh*).
> Just inserting the witness data into the bloom filter seems to be an easy
> solution (CBloomFilter::IsRelevantAndUpdate())
>
> /jonas
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 1618 bytes --]

Re: [bitcoin-dev] BloomFilter issue with segwit addresses

[Christian Decker]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

Note that this would compound the privacy leak that Jonas Nick used to
identify address clusters via the bloom filters in one of his publications.
By reducing the false positives when matching you can get very detailed
clusters. Then again we know that bloom filters aren't good for privacy
anyway, so this might be a non-issue.

On Sat, Apr 14, 2018, 00:17 Jim Posen via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Why not add the outpoints owned by the wallet to the filter and watch for
> those instead of elements in the input script or witness data?
>
> On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Hi Andreas
>>
>> Thanks for bringing this up and this seems indeed to be suboptimal.
>>
>> > I wonder if Bitcoin Core would be willing to extend the BIP37 matching
>> > rules such that data elements in the witness are also matched against?
>>
>> Bitcoin Core is not an identity that can be „willing to extend“ (or
>> reject) a feature.
>> Someone needs to come up with a proposal (pull request).
>>
>> Maybe an extension for BIP37 would make sense (*meh*).
>> Just inserting the witness data into the bloom filter seems to be an easy
>> solution (CBloomFilter::IsRelevantAndUpdate())
>>
>> /jonas
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 2748 bytes --]

Re: [bitcoin-dev] BloomFilter issue with segwit addresses

[Jim Posen]

[-- Attachment #1: Type: text/plain, Size: 2058 bytes --]

To Christian's point about privacy, I'll take this opportunity to
shamelessly review beg on https://github.com/bitcoin/bitcoin/pull/12254,
the PR for BIP 158 implementation (but not 157).

On Sat, Apr 14, 2018 at 9:14 AM, Christian Decker <
decker.christian@gmail.com> wrote:

> Note that this would compound the privacy leak that Jonas Nick used to
> identify address clusters via the bloom filters in one of his publications.
> By reducing the false positives when matching you can get very detailed
> clusters. Then again we know that bloom filters aren't good for privacy
> anyway, so this might be a non-issue.
>
> On Sat, Apr 14, 2018, 00:17 Jim Posen via bitcoin-dev <bitcoin-dev@lists.
> linuxfoundation.org> wrote:
>
>> Why not add the outpoints owned by the wallet to the filter and watch for
>> those instead of elements in the input script or witness data?
>>
>> On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev <
>> bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>>> Hi Andreas
>>>
>>> Thanks for bringing this up and this seems indeed to be suboptimal.
>>>
>>> > I wonder if Bitcoin Core would be willing to extend the BIP37 matching
>>> > rules such that data elements in the witness are also matched against?
>>>
>>> Bitcoin Core is not an identity that can be „willing to extend“ (or
>>> reject) a feature.
>>> Someone needs to come up with a proposal (pull request).
>>>
>>> Maybe an extension for BIP37 would make sense (*meh*).
>>> Just inserting the witness data into the bloom filter seems to be an
>>> easy solution (CBloomFilter::IsRelevantAndUpdate())
>>>
>>> /jonas
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

[-- Attachment #2: Type: text/html, Size: 3513 bytes --]

Re: [bitcoin-dev] BloomFilter issue with segwit addresses

[Andreas Schildbach]

Yes, I guess the quicker filter exhaustion must be the reason why
bitcoinj doesn't make use of outpoints in filters for standard
transactions. I'll look into if I can change that.


On 04/14/2018 06:14 PM, Christian Decker via bitcoin-dev wrote:
> Note that this would compound the privacy leak that Jonas Nick used to
> identify address clusters via the bloom filters in one of his
> publications. By reducing the false positives when matching you can get
> very detailed clusters. Then again we know that bloom filters aren't
> good for privacy anyway, so this might be a non-issue.
> 
> On Sat, Apr 14, 2018, 00:17 Jim Posen via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org
> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
> 
>     Why not add the outpoints owned by the wallet to the filter and
>     watch for those instead of elements in the input script or witness data?
> 
>     On Fri, Apr 13, 2018 at 12:12 PM, Jonas Schnelli via bitcoin-dev
>     <bitcoin-dev@lists.linuxfoundation.org
>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
> 
>         Hi Andreas
> 
>         Thanks for bringing this up and this seems indeed to be suboptimal.
> 
>         > I wonder if Bitcoin Core would be willing to extend the BIP37 matching
>         > rules such that data elements in the witness are also matched against?
> 
>         Bitcoin Core is not an identity that can be „willing to extend“
>         (or reject) a feature.
>         Someone needs to come up with a proposal (pull request).
> 
>         Maybe an extension for BIP37 would make sense (*meh*).
>         Just inserting the witness data into the bloom filter seems to
>         be an easy solution (CBloomFilter::IsRelevantAndUpdate())
> 
>         /jonas
> 
>         _______________________________________________
>         bitcoin-dev mailing list
>         bitcoin-dev@lists.linuxfoundation.org
>         <mailto:bitcoin-dev@lists.linuxfoundation.org>
>         https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> 
>     _______________________________________________
>     bitcoin-dev mailing list
>     bitcoin-dev@lists.linuxfoundation.org
>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>     https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> 
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

Re: [bitcoin-dev] BloomFilter issue with segwit addresses

[Luke Dashjr]

As I understand it, the plan is to deprecated and remove BIP37 entirely once 
BIP158 is implemented and deployed.

In the meantime, Bitcoin Knots supports the MSG_FILTERED_WITNESS_BLOCK 
extension to download witness data. (Note that light clients currently have no 
way to verify the witness data is correct.)

As far as matching goes, why not look for the specific COutPoints? That should 
work already with standard BIP37.

Luke


On Friday 13 April 2018 3:32:15 PM Andreas Schildbach via bitcoin-dev wrote:
> Anton, a developer on the bitcoinj maiing list, recently made me aware
> [1] of a compatibility issue between segwit and BIP37 (Bloom Filtering).
> 
> The issue affects only P2WPKH and the special case of transactions
> without change outputs (such as when emptying a wallet). In this case,
> neither inputs not outputs contain any data elements that would cause a
> match for the filter. The public key, which would match, goes to the
> witness but not to the input.
> 
> My suggestion was to include an OP_RETURN output with a matching public
> key in such transactions. Anton confirmed that this workaround is indeed
> working. But of course it nullifies some of the segwit's size improvements.
> 
> I wonder if Bitcoin Core would be willing to extend the BIP37 matching
> rules such that data elements in the witness are also matched against?
> 
> 
> [1] https://groups.google.com/d/msg/bitcoinj/SJpLgjowc1I/V7u2BavvAwAJ
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev