Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Peter]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

Hi,

I believe two wallets. Andreas' Android Bitcoin wallet and BRD are
significant users of node_bloom.

Privacy is a matter of individual choice in the current protocol. Why not
let people provide this network service? I don't see why it should be
end-of-life if it provides value.

I believe there's a network security obtained by having a large quantity of
people following the Bitcoin headers based on longest weighted chain. As a
means of nullifying potential miner initiated hard forks (like S2X).

Respectfully,
Peter

[-- Attachment #2: Type: text/html, Size: 768 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Greg Sanders]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

People are allowed the choice, it's a change of default only.

On Mon, Jul 22, 2019 at 4:41 PM Peter via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi,
>
> I believe two wallets. Andreas' Android Bitcoin wallet and BRD are
> significant users of node_bloom.
>
> Privacy is a matter of individual choice in the current protocol. Why not
> let people provide this network service? I don't see why it should be
> end-of-life if it provides value.
>
> I believe there's a network security obtained by having a large quantity
> of people following the Bitcoin headers based on longest weighted chain. As
> a means of nullifying potential miner initiated hard forks (like S2X).
>
> Respectfully,
> Peter
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 1613 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Luke Dashjr]

On Monday 22 July 2019 18:52:10 Peter via bitcoin-dev wrote:
> Privacy is a matter of individual choice in the current protocol. Why not
> let people provide this network service? I don't see why it should be
> end-of-life if it provides value.

It's not EOL, just disabled by default. Anyone can provide it by choice.
Same as with Stratum/Electrum right now (except easier to enable).

> I believe there's a network security obtained by having a large quantity of
> people following the Bitcoin headers based on longest weighted chain. As a
> means of nullifying potential miner initiated hard forks (like S2X).

This is incorrect. Such wallets strictly degrade security, as they are blindly 
trusting miners. They make the network more VULNERABLE to 2X-like attacks.

Luke

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

On Mon, Jul 22, 2019 at 02:52:10PM -0400, Peter via bitcoin-dev wrote:
> Hi,
> 
> I believe two wallets. Andreas' Android Bitcoin wallet and BRD are
> significant users of node_bloom.
> 
> Privacy is a matter of individual choice in the current protocol. Why not
> let people provide this network service? I don't see why it should be
> end-of-life if it provides value.

With that patch people are still free to choose to provide bloom filtering
services by setting -peerbloomfilters=1

> I believe there's a network security obtained by having a large quantity of
> people following the Bitcoin headers based on longest weighted chain. As a
> means of nullifying potential miner initiated hard forks (like S2X).

There really isn't due to sybil attacks; we already have good reason to believe
that the Bitcoin network is subject to them by deanonymization/chainanalysis
services.

Indeed there's a good argument that creating services that are vulnerable to
sybil attacks encourages them by making them succesful at something. That
creates its own risks, for instance the risk that the sybil attacker will
themselves screw up and cause a bunch of nodes to go offline at once.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Matt Corallo]

Just a quick heads-up for those watching the list who may be using it -
in the next Bitcoin Core release bloom filter serving will be turned off
by default. This has been a long time coming, it's been an option for
many releases and has been a well-known DoS vector for some time.
As other DoS vectors have slowly been closed, this has become
increasingly an obvious low-hanging fruit. Those who are using it should
already have long been filtering for NODE_BLOOM-signaling nodes, and I
don't anticipate those being gone any time particularly soon.

See-also PR at https://github.com/bitcoin/bitcoin/pull/16152

The release notes will liekly read:

P2P Changes
-----------
- The default value for the -peerbloomfilters configuration option (and,
thus, NODE_BLOOM support) has been changed to false.
  This resolves well-known DoS vectors in Bitcoin Core, especially for
nodes with spinning disks. It is not anticipated that
  this will result in a significant lack of availability of
NODE_BLOOM-enabled nodes in the coming years, however, clients
  which rely on the availability of NODE_BLOOM-supporting nodes on the
P2P network should consider the process of migrating
  to a more modern (and less trustful and privacy-violating) alternative
over the coming years.

Matt

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Andreas Schildbach]

An estimated 10+ million wallets depend on that NODE_BLOOM to be
updated. So far, I haven't heard of an alternative, except reading all
transactions and full blocks.

It goes without saying pulling the rug under that many wallets is a
disastrous idea for the adoption of Bitcoin.

> well-known DoS vectors

I asked many people, even some "core developers" at meetings, but nobody
ever was able to explain the DoS vector. I think this is just a myth.

Yes, you can set an overly blurry filter and thus cause useless traffic,
but it never exceeds just drinking from the full firehose (which this
change doesn't prohibit). So where is the point? An attacker will just
switch filtering off, or in fact has never used it.

> It is not anticipated that
> this will result in a significant lack of availability of
> NODE_BLOOM-enabled nodes in the coming years

Why don't you anticipate that? People almost never change defaults,
especially if it's not for their own immediate benefit. At the same
time, release notes in general recommend updating to the latest version.
I *do* anticipate this will reduce the number of nodes usable by a large
enough amount so that the feature will become unstable.

> clients
> which rely on the availability of NODE_BLOOM-supporting nodes on the
> P2P network should consider the process of migrating
> to a more modern (and less trustful and privacy-violating) alternative
> over the coming years.

There is no such alternative.

I strongly recommend postponing this change until an alternative exists
and then give developers enough time to implement, test and roll out.

I also think as long as we don't have an alternative, we should improve
the current filtering for segwit. E.g. testing the scripts themselves
and each scriptPubKey spent by any input against the filter would do,
and it also fixes the main privacy issue with server-side filtering
(wallets have to add two items per address to the filter).

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Matt Corallo]

Hey Andreas,

I think maybe some of the comments here were misunderstood - I don't
anticipate that most people will change their defaults, indeed, but
given the general upgrade cycles we've seen on the network over the
entire course of Bitcoin's history, there's little reason to believe
that many nodes with NODE_BLOOM publicly accessible will be around for
at least three or four years to come, though obviously any conscious
effort by folks who need those services to run nodes could extend that
significantly.

As for the DoS issues, a super old Proof-of-Concept of the I/O variant
is here: https://github.com/petertodd/bloom-io-attack though CPU DoS
attacks are also possible that use high hash counts to fill a node's CPU
usage (you can pretty trivially see when a bloom-based peer connects to
you just by looking at top...).

Finally, regarding alternatives, the filter-generation code for BIP
157/158 has been in Bitcoin Core for some time, though the P2P serving
side of things appears to have lost any champions working on it. I
presume one of the Lightning folks will eventually, given they appear to
be requiring their users connect to a handful of their own servers right
now, but if you really need it, its likely not a ton of work to pipe
them through.

Matt

On 7/21/19 10:56 PM, Andreas Schildbach via bitcoin-dev wrote:
> An estimated 10+ million wallets depend on that NODE_BLOOM to be
> updated. So far, I haven't heard of an alternative, except reading all
> transactions and full blocks.
> 
> It goes without saying pulling the rug under that many wallets is a
> disastrous idea for the adoption of Bitcoin.
> 
>> well-known DoS vectors
> 
> I asked many people, even some "core developers" at meetings, but nobody
> ever was able to explain the DoS vector. I think this is just a myth.
> 
> Yes, you can set an overly blurry filter and thus cause useless traffic,
> but it never exceeds just drinking from the full firehose (which this
> change doesn't prohibit). So where is the point? An attacker will just
> switch filtering off, or in fact has never used it.
> 
>> It is not anticipated that
>> this will result in a significant lack of availability of
>> NODE_BLOOM-enabled nodes in the coming years
> 
> Why don't you anticipate that? People almost never change defaults,
> especially if it's not for their own immediate benefit. At the same
> time, release notes in general recommend updating to the latest version.
> I *do* anticipate this will reduce the number of nodes usable by a large
> enough amount so that the feature will become unstable.
> 
>> clients
>> which rely on the availability of NODE_BLOOM-supporting nodes on the
>> P2P network should consider the process of migrating
>> to a more modern (and less trustful and privacy-violating) alternative
>> over the coming years.
> 
> There is no such alternative.
> 
> I strongly recommend postponing this change until an alternative exists
> and then give developers enough time to implement, test and roll out.
> 
> I also think as long as we don't have an alternative, we should improve
> the current filtering for segwit. E.g. testing the scripts themselves
> and each scriptPubKey spent by any input against the filter would do,
> and it also fixes the main privacy issue with server-side filtering
> (wallets have to add two items per address to the filter).
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Justus Ranvier]

[-- Attachment #1.1.1: Type: text/plain, Size: 806 bytes --]

On 7/22/19 12:01 AM, Matt Corallo via bitcoin-dev wrote:
> Finally, regarding alternatives, the filter-generation code for BIP
> 157/158 has been in Bitcoin Core for some time, though the P2P serving
> side of things appears to have lost any champions working on it. I
> presume one of the Lightning folks will eventually, given they appear to
> be requiring their users connect to a handful of their own servers right
> now, but if you really need it, its likely not a ton of work to pipe
> them through.

If you want projects to adopt BIP-157/158, you'd do well to fix the
numerous errors in the specification.

As it stands right now it is impossible to implement the protocol using
the specification because he code examples are broken to the point of
appearing intentionally sabotaged.

[-- Attachment #1.1.2: 0x9B2D74E2A30A85B9.asc --]
[-- Type: application/pgp-keys, Size: 29805 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Tamas Blummer]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

Hi Justus,

It might be helpful to consult the Rust implementation  of BIP158 here:
https://github.com/rust-bitcoin/rust-bitcoin/blob/master/src/util/bip158.rs

It has a cleaner structure than Core or Neutrino, includes server and client side
and passes Core's test vectors.

Regards,

Tamas Blummer

> On Jul 22, 2019, at 17:58, Justus Ranvier via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> Signed PGP part
> On 7/22/19 12:01 AM, Matt Corallo via bitcoin-dev wrote:
>> Finally, regarding alternatives, the filter-generation code for BIP
>> 157/158 has been in Bitcoin Core for some time, though the P2P serving
>> side of things appears to have lost any champions working on it. I
>> presume one of the Lightning folks will eventually, given they appear to
>> be requiring their users connect to a handful of their own servers right
>> now, but if you really need it, its likely not a ton of work to pipe
>> them through.
> 
> If you want projects to adopt BIP-157/158, you'd do well to fix the
> numerous errors in the specification.
> 
> As it stands right now it is impossible to implement the protocol using
> the specification because he code examples are broken to the point of
> appearing intentionally sabotaged.
> 
> 
> 


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

On Mon, Jul 22, 2019 at 12:56:33AM +0200, Andreas Schildbach via bitcoin-dev wrote:
> An estimated 10+ million wallets depend on that NODE_BLOOM to be
> updated. So far, I haven't heard of an alternative, except reading all
> transactions and full blocks.

Can you specify exactly which wallets those are?

Secondly, this doesn't stop people from running NODE_BLOOM nodes, and the DNS
seed infrastructure among others can easily direct wallets to those nodes. This
is of course not very secure, but bloom filters have never been very secure.

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Jonas Schnelli]

[-- Attachment #1.1: Type: text/plain, Size: 4286 bytes --]

Hi Andreas

>> well-known DoS vectors
> 
> I asked many people, even some "core developers" at meetings, but nobody
> ever was able to explain the DoS vector. I think this is just a myth.

No. They are not a myth [1] [2] [3].

> Yes, you can set an overly blurry filter and thus cause useless traffic,
> but it never exceeds just drinking from the full firehose (which this
> change doesn't prohibit). So where is the point? An attacker will just
> switch filtering off, or in fact has never used it.

I guess it’s not about traffic DoS. It’s about asking a peer for extensive CPU and disk work. The NODE_BLOOM peers do provide this service for free while it’s not directly beneficial for the Bitcoin Network (pure consumed CPU/disk time).

> 
>> It is not anticipated that
>> this will result in a significant lack of availability of
>> NODE_BLOOM-enabled nodes in the coming years
> 
> Why don't you anticipate that? People almost never change defaults,
> especially if it's not for their own immediate benefit. At the same
> time, release notes in general recommend updating to the latest version.
> I *do* anticipate this will reduce the number of nodes usable by a large
> enough amount so that the feature will become unstable.

I guess this is speculation.
A quick lookup in my crawler databases shows me that there are more than 8’000 „good“ peers supporting NODE_BLOOM right now.
I don’t expect that this number drops rapidly, but maybe in the long run ("in years“, but again: speculation).

We eventually can’t expect - in the long run - that nodes provide disk/CPU intense services for free to clients not contributing back to the network.
However, sadly, due to the privacy leaks in BIP37, I expect that there will always be a wide range of peers offering NODE_BLOOM in return of collecting valuable data.

> 
>> clients
>> which rely on the availability of NODE_BLOOM-supporting nodes on the
>> P2P network should consider the process of migrating
>> to a more modern (and less trustful and privacy-violating) alternative
>> over the coming years.
> 
> There is no such alternative.
> 
> I strongly recommend postponing this change until an alternative exists
> and then give developers enough time to implement, test and roll out.

NODE_BLOOM was added in Core 0.12 [4].
BIP111 is from 2015 [2].
One who follows the protocol development should have known that defaulting NODE_BLOOM to off was something anticipated by most developers.

I would say that there are possible alternatives, like BIP158 (though BIP157 is not widely available on the network yet).
Client side filtering works also by collecting the filter form a centralised service by the wallet provider(s) or a CDN.
You may miss transactions by a dishonest filter-packet, so may you by talking to a dishonest NODE_BLOOM peer.
Of course BIP158 is still young and – who knows – eventually once committed to consensus layer (coinbase).


> I also think as long as we don't have an alternative, we should improve
> the current filtering for segwit. E.g. testing the scripts themselves
> and each scriptPubKey spent by any input against the filter would do,
> and it also fixes the main privacy issue with server-side filtering
> (wallets have to add two items per address to the filter).

I think the consensus among protocol developers is (please speak up), that BIP37 (public server based tx filtering) – in general – was a conceptual mistake.
Maybe extending it further is the wrong step, especially when promising alternatives like BIP158 (neutrino) are around.
The fact that nobody cared about extending it for SW may also underline that BIP37 is seen as a conceptual mistake and/or "low interest in further extensions“.


/jonas

[1] https://github.com/petertodd/bloom-io-attack <https://github.com/petertodd/bloom-io-attack>
[2] https://github.com/bitcoin/bips/blob/master/bip-0111.mediawiki <https://github.com/bitcoin/bips/blob/master/bip-0111.mediawiki>
[3] https://github.com/bitcoin/bitcoin/pull/9238 <https://github.com/bitcoin/bitcoin/pull/9238>
[4] https://github.com/bitcoin/bitcoin/blob/47d981e8273804a040d71665a4cb16038d6717e1/doc/release-notes/release-notes-0.12.0.md#node_bloom-service-bit


[-- Attachment #1.2: Type: text/html, Size: 6236 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Luke Dashjr]

On Monday 22 July 2019 13:25:25 Jonas Schnelli via bitcoin-dev wrote:
> > I also think as long as we don't have an alternative, we should improve
> > the current filtering for segwit. E.g. testing the scripts themselves
> > and each scriptPubKey spent by any input against the filter would do,
> > and it also fixes the main privacy issue with server-side filtering
> > (wallets have to add two items per address to the filter).
>
> I think the consensus among protocol developers is (please speak up), that
> BIP37 (public server based tx filtering) – in general – was a conceptual
> mistake. Maybe extending it further is the wrong step, especially when
> promising alternatives like BIP158 (neutrino) are around.

Neutrino is very controversial, and NOT less trustful than bloom filters.
It also uses significantly more bandwidth.

It seems a better approach is to add Stratum (Electrum) support, and to limit 
usage of all pseudo-SPV protocols to trusted peers.

> The fact that nobody cared about extending it for SW may also underline that
> BIP37 is seen as a conceptual mistake and/or "low interest in further
> extensions“. 

Eric Lombrozo added segwit support. While it was never reviewed for Core, it 
has been included and supported in Knots since v0.15.1. As I understand it, 
his mSIGNA wallet also makes usage of the feature.

Luke

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Luke Dashjr]

On Sunday 21 July 2019 22:56:33 Andreas Schildbach via bitcoin-dev wrote:
> An estimated 10+ million wallets depend on that NODE_BLOOM to be
> updated.

Where do you see this number? I think it would be useful to chart.

> So far, I haven't heard of an alternative, except reading all 
> transactions and full blocks.

Electrum has a JSON-based protocol that can provide information much more 
efficiently.

Currently, this does require nodes that run additional software and/or 
indexes, but there are plenty of them available, and there are steps that can 
be taken to improve that situation.

> > It is not anticipated that
> > this will result in a significant lack of availability of
> > NODE_BLOOM-enabled nodes in the coming years
>
> Why don't you anticipate that? People almost never change defaults,
> especially if it's not for their own immediate benefit. At the same
> time, release notes in general recommend updating to the latest version.
> I *do* anticipate this will reduce the number of nodes usable by a large
> enough amount so that the feature will become unstable.

Many users run older versions, and do not update immediately. Today, only 42% 
of listening nodes are using 0.18.

(If it helps ease the concern, we can keep bloom enabled by default in Knots 
longer.)

> I strongly recommend postponing this change until an alternative exists
> and then give developers enough time to implement, test and roll out.

There will be time to do so, since the functionality won't disappear 
overnight.

Luke

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Andreas Schildbach]

(Rather than replying individually, I try to address questions and add
my remarks in one post.)

> Finally, regarding alternatives, the filter-generation code for
> BIP 157/158 has been in Bitcoin Core for some time, though the
> P2P serving side of things appears to have lost any champions
> working on it.

BIP 157/158 is not an alternative to BIP 37:

1) It causes way too much traffic for mobile users, and likely even too
much traffic for fixed lines in not so developed parts of the world.

2) It filters blocks only. It doesn't address unconfirmed transactions.

3) Afaik, it enforces/encourages address re-use. This stems from the
fact that the server decides on the filter and in particular on the
false positive rate. On wallets with many addresses, a hardcoded filter
will be too blurry and thus each block will be matched. So wallets that
follow the "one address per incoming payment" pattern (e.g. HD wallets)
at some point will be forced to wrap their key chains back to the
beginning. If I'm wrong on this one please let me know.

4) The filters are not yet committed to the blockchain. Until that
happens we'd have to trust a server to provide correct filters.

> Can you specify exactly which wallets those are?

Bitcoin Wallet, Breadwallet, BISQ and many smaller ones.

> the DNS seed infrastructure among others can easily direct
> wallets to those nodes

Last I checked none of the seeds did. But I agree this would be nice to
have.

> We eventually can’t expect - in the long run - that nodes provide
> disk/CPU intense services for free to clients not contributing back
> to the network.

I'm disappointed to learn that supporting 10M wallets gets discredited
down to "no contribution". Each node of course supports just a number of
them, but still...

> The fact that nobody cared about extending it for SW may also
> underline that BIP37 is seen as a conceptual mistake and/or "low
> interest in further extensions“.

I suspect this is a fallacy. BIP37 doesn't need to be extended for
SegWit. See Bitcoin Wallet and Breadwallet, both support SegWit and use
the original BIP 37.

It's true however that BIP 37 could be made simpler to work with, more
future-proof and more private by simply matching output scripts.

> [1] https://github.com/petertodd/bloom-io-attack

This claims to be a proof of concept, but it's missing the concept. I
could not find a description of the attack and why is it likely to be
exploited (and why it hasn't been exploited yet).

> CPU DoS attacks are also possible

If such an attack exists, it should be easy to defend against. Filter is
using too much CPU time → disconnect and blacklist the IP for some time.
I vaguely remember the infrastructure for misbehaving peers is already
present in bitcoind.


As a side note, Coinbase just announced their 30M'th wallet. I'm
convinced this massive run into custodial wallets is caused by the
public realizing around Dec 2017 that Bitcoin fails to scale. IMHO, BIP
37 is the only scaling technology that has proven successful and could –
if supported and improved rather than choked – continue to help holding
against the bitbanks trend.

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Justus Ranvier]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On 7/23/19 9:47 AM, Andreas Schildbach via bitcoin-dev wrote:
> BIP 157/158 is not an alternative to BIP 37:

They complement each other pretty well though.

Wallets can save the deterministic GCS filters in the same way as
headers, which means blocks can be re-scanned if necessary (importing
new keys, etc) offline.

Bloom filters are good for limiting mempool bandwidth as well as
controlling the fraction of each block that is downloaded.

On the BTC chain that last point doesn't matter as so much, but on
chains with larger blocks I expect that clients will ultimately end up
using both filter schemes for exactly that reason.

[-- Attachment #2: 0x9B2D74E2A30A85B9.asc --]
[-- Type: application/pgp-keys, Size: 29350 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Luke Dashjr]

On Tuesday 23 July 2019 14:47:18 Andreas Schildbach via bitcoin-dev wrote:
> > the DNS seed infrastructure among others can easily direct
> > wallets to those nodes
>
> Last I checked none of the seeds did. But I agree this would be nice to
> have.

It's supported by default in sipa's bitcoin-seeder, which many DNS seeds use.

My seed also currently requires NODE_BLOOM for *any* answers returned.
(But I don't guarantee that will remain forever.)

> As a side note, Coinbase just announced their 30M'th wallet. I'm
> convinced this massive run into custodial wallets...

Light wallets are just as bad for the network as custodial wallets.

> IMHO, BIP 37 is the only scaling technology that has proven successful and
> could – if supported and improved rather than choked – continue to help
> holding against the bitbanks trend.

Light wallets are not Bitcoin scaling. They are just trusting anonymous 3rd 
parties, which is harmful to Bitcoin.

Luke

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Jonas Schnelli]

[-- Attachment #1.1: Type: text/plain, Size: 4990 bytes --]


> 1) It causes way too much traffic for mobile users, and likely even too
> much traffic for fixed lines in not so developed parts of the world.

Yes. It causes more traffic than BIP37.
Basic block filters for current last ~7 days (1008 blocks) are about 19MB (just the filters).
On top, you will probably fetch a handful of irrelevant blocks due to the FPs and due to true relevant txns.
A over-the-thumb estimation: ~25MB per week of catch-up.
If you where offline for a month: ~108MB

Thats certainly more then BIP37 BF (measured 1.6MB total traffic with android schildbach wallet restore blockchain for 8 week [7 weeks headers, 1week merkleblocks]).

But lets look at it like this: for an additional, say 25MB per week (maybe a bit more), you get the ability to filter blocks without depending on serving peers who may compromise your financial privacy.
Also, if you keep the filters, further rescans do consume the same or less bandwidth than BF BIP37.
In other words: you have the chance to potentially increase privacy by consuming bandwidth in the range of a single audio podcast per week.

I would say the job of protocol developers is protect users privacy where it’s possible (as a default).
It’s probably a debatable point wether 25MB per week of traffic is worth a potential increase in privacy, though I absolutely think 25MB/week is an acceptable tradeoff.
Saving traffic is possible by using BIP37 or stratum/electrum… but developers should make sure users are __warned about the consequences__!

Additionally, it looks like, peer operators are not endless being willing to serve – for free – a CPU/disk intense service with no benefits for the network. I would question wether a decentralised form of BIP37 is sustainable in the long run (if SPV wallet provider bootstrap a net range of NODE_BLOOM peers to make it more reliable on the network would be snake-oil).


> 
> 2) It filters blocks only. It doesn't address unconfirmed transactions.

Well, unconfirmed transaction are uncertain for various reasons.

BIP158 won't allow you to filter the mempool.
But as soon as you are connected to the network, you may fetch tx with inv/getdata and pick out the relevant ones (causes also traffic).
Unclear and probably impossible with the current BIP158 specs to fetch transactions that are not in active relay and are not in a block (mempool txns, at least this is true with the current observed relay tactics).


> 3) Afaik, it enforces/encourages address re-use. This stems from the
> fact that the server decides on the filter and in particular on the
> false positive rate. On wallets with many addresses, a hardcoded filter
> will be too blurry and thus each block will be matched. So wallets that
> follow the "one address per incoming payment" pattern (e.g. HD wallets)
> at some point will be forced to wrap their key chains back to the
> beginning. If I'm wrong on this one please let me know.

I’m probably the wrong guy to ask (haven’t made the numbers) but last time I rescanned a Core wallet (in my dev branch) with block filters (and a Core wallet has >2000 addresses by default) it fetched a low and acceptable amount of false positive blocks.
(Maybe someone who made the numbers step in here.)

Though, large wallets – AFAIK – also operate badly with BIP37.

> 
> 4) The filters are not yet committed to the blockchain. Until that
> happens we'd have to trust a server to provide correct filters.

I wouldn’t say so. It’s on a similar level than BIP37.
BIP37 is not – and can not – be committed to the blockchain.
You fully trust the peer that it won’t…
a) create fake unconfirmed transactions (would be the same if a BIP158 wallet would show you unconfirmed transaction)
b) lies by omission (you will miss relevant transactions, eventually swipe your wallet and loose coins)

IMO, the point b) is true for BIP37 and BIP158 (as long as not commited).
In both cases, you can reduce the trust by comparing between peers / filter-providers.

b) is conceptually solvable with a soft-fork (commitment) in BIP158 (not with BIP37).

Additionally, block-filters will, very likely, be useful for other features (load/rescan an [old] wallet on a prune peer that has the filters constructed).



There is probably no clear answer like „X is better than Y“.

Personally I would like to see developers being more honest/transparent to users about the implications of the used filtering,... and giving them choices.
Imagine a user can choose between „Electrum / BIP37 / BIP158“ depending on his needs for privacy and availability of bandwidth. Eventually also taking the future usage of this wallet (will he load old private keys, will he receive money daily, etc.) into account.

Plus, full node hardware appliances that run at home (or in a trusted environment) are solving many of these issues plus adding a bunch of great features – if done right.

/jonas

[-- Attachment #1.2: Type: text/html, Size: 15526 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Matt Corallo]

[-- Attachment #1: Type: text/plain, Size: 6888 bytes --]

This conversation went off the rails somewhat. I don't think there's any immediate risk of NODE_BLOOM peers being unavailable. This is a defaults change, not a removal of the code to serve BIP 37 peers (nor would I suggest removing said code while people still want to use them - the maintenance burden isn't much). Looking at historical upgrade cycles, ignoring any other factors, there will be a large number of nodes serving NODE_BLOOM for many years.

Even more importantly, if you need them, run a node or two. As long as no one is exploiting the issues with them such a node isn't *too* expensive. Or don't, I guarantee you chainanalysis or some competitor of theirs will very very happily serve bloom-filtered clients as long as such clients want to deanonymize themselves. We already see a plurality of nodes on the network are clearly not run-of-the-mill Core nodes, many of which are likely deanonimization efforts.

In some cases BIP 137 is a replacement, in some cases, indeed, it is not. I agree at a protocol level we shouldn't be passing judgement about how users wish to interact with the Bitcoin system (aside from not putting our own, personal, effort into building such things) but that isn't what's happening here. This is an important DoS fix for the average node, and I don't really understand the argument that this is going to break existing BIP 37 wallets, but if it makes you feel any better I can run some beefy BIP 37 nodes.

Matt

> On Jul 26, 2019, at 06:04, Jonas Schnelli via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> 
>> 1) It causes way too much traffic for mobile users, and likely even too
>> much traffic for fixed lines in not so developed parts of the world.
> 
> Yes. It causes more traffic than BIP37.
> Basic block filters for current last ~7 days (1008 blocks) are about 19MB (just the filters).
> On top, you will probably fetch a handful of irrelevant blocks due to the FPs and due to true relevant txns.
> A over-the-thumb estimation: ~25MB per week of catch-up.
> If you where offline for a month: ~108MB
> 
> Thats certainly more then BIP37 BF (measured 1.6MB total traffic with android schildbach wallet restore blockchain for 8 week [7 weeks headers, 1week merkleblocks]).
> 
> But lets look at it like this: for an additional, say 25MB per week (maybe a bit more), you get the ability to filter blocks without depending on serving peers who may compromise your financial privacy.
> Also, if you keep the filters, further rescans do consume the same or less bandwidth than BF BIP37.
> In other words: you have the chance to potentially increase privacy by consuming bandwidth in the range of a single audio podcast per week.
> 
> I would say the job of protocol developers is protect users privacy where it’s possible (as a default).
> It’s probably a debatable point wether 25MB per week of traffic is worth a potential increase in privacy, though I absolutely think 25MB/week is an acceptable tradeoff.
> Saving traffic is possible by using BIP37 or stratum/electrum… but developers should make sure users are __warned about the consequences__!
> 
> Additionally, it looks like, peer operators are not endless being willing to serve – for free – a CPU/disk intense service with no benefits for the network. I would question wether a decentralised form of BIP37 is sustainable in the long run (if SPV wallet provider bootstrap a net range of NODE_BLOOM peers to make it more reliable on the network would be snake-oil).
> 
> 
>> 
>> 2) It filters blocks only. It doesn't address unconfirmed transactions.
> 
> Well, unconfirmed transaction are uncertain for various reasons.
> 
> BIP158 won't allow you to filter the mempool.
> But as soon as you are connected to the network, you may fetch tx with inv/getdata and pick out the relevant ones (causes also traffic).
> Unclear and probably impossible with the current BIP158 specs to fetch transactions that are not in active relay and are not in a block (mempool txns, at least this is true with the current observed relay tactics).
> 
> 
>> 3) Afaik, it enforces/encourages address re-use. This stems from the
>> fact that the server decides on the filter and in particular on the
>> false positive rate. On wallets with many addresses, a hardcoded filter
>> will be too blurry and thus each block will be matched. So wallets that
>> follow the "one address per incoming payment" pattern (e.g. HD wallets)
>> at some point will be forced to wrap their key chains back to the
>> beginning. If I'm wrong on this one please let me know.
> 
> I’m probably the wrong guy to ask (haven’t made the numbers) but last time I rescanned a Core wallet (in my dev branch) with block filters (and a Core wallet has >2000 addresses by default) it fetched a low and acceptable amount of false positive blocks.
> (Maybe someone who made the numbers step in here.)
> 
> Though, large wallets – AFAIK – also operate badly with BIP37.
> 
>> 
>> 4) The filters are not yet committed to the blockchain. Until that
>> happens we'd have to trust a server to provide correct filters.
> 
> I wouldn’t say so. It’s on a similar level than BIP37.
> BIP37 is not – and can not – be committed to the blockchain.
> You fully trust the peer that it won’t…
> a) create fake unconfirmed transactions (would be the same if a BIP158 wallet would show you unconfirmed transaction)
> b) lies by omission (you will miss relevant transactions, eventually swipe your wallet and loose coins)
> 
> IMO, the point b) is true for BIP37 and BIP158 (as long as not commited).
> In both cases, you can reduce the trust by comparing between peers / filter-providers.
> 
> b) is conceptually solvable with a soft-fork (commitment) in BIP158 (not with BIP37).
> 
> Additionally, block-filters will, very likely, be useful for other features (load/rescan an [old] wallet on a prune peer that has the filters constructed).
> 
> 
> 
> There is probably no clear answer like „X is better than Y“.
> 
> Personally I would like to see developers being more honest/transparent to users about the implications of the used filtering,... and giving them choices.
> Imagine a user can choose between „Electrum / BIP37 / BIP158“ depending on his needs for privacy and availability of bandwidth. Eventually also taking the future usage of this wallet (will he load old private keys, will he receive money daily, etc.) into account.
> 
> Plus, full node hardware appliances that run at home (or in a trusted environment) are solving many of these issues plus adding a bunch of great features – if done right.
> 
> /jonas
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 17831 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Chris]

On 7/23/19 10:47 AM, Andreas Schildbach via bitcoin-dev wrote:

> 3) Afaik, it enforces/encourages address re-use. This stems from the
> fact that the server decides on the filter and in particular on the
> false positive rate. On wallets with many addresses, a hardcoded filter
> will be too blurry and thus each block will be matched. So wallets that
> follow the "one address per incoming payment" pattern (e.g. HD wallets)
> at some point will be forced to wrap their key chains back to the
> beginning. If I'm wrong on this one please let me know.

Maybe someone who knows better can confirm but I thought I read the the 
fp rate on the filter was chosen assuming a wallet would average a 
certain number of addresses (ie, they assumed use of an HD wallet). Your 
criticism might be valid for wallets that well exceed the average.

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Luke Dashjr]

On Tuesday 23 July 2019 14:47:18 Andreas Schildbach via bitcoin-dev wrote:
> 3) Afaik, it enforces/encourages address re-use. This stems from the
> fact that the server decides on the filter and in particular on the
> false positive rate. On wallets with many addresses, a hardcoded filter
> will be too blurry and thus each block will be matched. So wallets that
> follow the "one address per incoming payment" pattern (e.g. HD wallets)
> at some point will be forced to wrap their key chains back to the
> beginning. If I'm wrong on this one please let me know.

BTW, you are indeed wrong on this. You don't need to match every single 
address the wallet has ever used, only outstanding addresses that haven't 
been paid. ;)

Luke

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Tom Harding]

On 7/20/19 10:46 AM, Matt Corallo via bitcoin-dev wrote:
> (less trustful and privacy-violating) alternative
> over the coming years.

The same paper that established the 'privacy-violating' conventional 
wisdom presented mitigations which have seen little exploration. 
https://eprint.iacr.org/2014/763.pdf

Meanwhile we have custodial LN, the L-BTC altcoin and, today, a massive 
push into infrastructure for fully custodial accounts.

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Dustin Dettmer]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

Has someone built an analysis of how much extra bandwidth CFB uses over
bloom filters?

Obviously an active merchant in an impoverished country paying data rates
per MB will never be able to afford CFB — so those people are being cut out
of Bitcoin entirely. I suppose the plan is they will rely on custodial
services now?

But if someone receives say, 5 tx a day, how much more bandwidth precisely
will CFB require over bloom?

On Mon, Jul 22, 2019 at 8:10 AM Tom Harding via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On 7/20/19 10:46 AM, Matt Corallo via bitcoin-dev wrote:
> > (less trustful and privacy-violating) alternative
> > over the coming years.
>
> The same paper that established the 'privacy-violating' conventional
> wisdom presented mitigations which have seen little exploration.
> https://eprint.iacr.org/2014/763.pdf
>
> Meanwhile we have custodial LN, the L-BTC altcoin and, today, a massive
> push into infrastructure for fully custodial accounts.
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 1908 bytes --]

Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default

[Matt Corallo]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

You very clearly didn't bother to read other mails in this thread. To make it easy for you, here's a few links:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-July/017147.html
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-July/017175.html

Matt

> On Aug 13, 2019, at 23:05, Will Madden <will.madden@bridge21.com> wrote:
> 
> For the record, strong NACK. My understanding is that this breaks several established SPV implementations (such as early breadwallet for sure and possibly current BRD wallets) and I have yet to see quantitative prioritization or even a rational justification for this change.
> 
> Requiring SPV wallets to communicate with trusted nodes is centralization, and breaking functionality and implementations that enable this without a thoroughly researched rationale is highly suspect.
> 
>> On Jul 20, 2019, at 1:46 PM, Matt Corallo via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> 
>> Just a quick heads-up for those watching the list who may be using it -
>> in the next Bitcoin Core release bloom filter serving will be turned off
>> by default. This has been a long time coming, it's been an option for
>> many releases and has been a well-known DoS vector for some time.
>> As other DoS vectors have slowly been closed, this has become
>> increasingly an obvious low-hanging fruit. Those who are using it should
>> already have long been filtering for NODE_BLOOM-signaling nodes, and I
>> don't anticipate those being gone any time particularly soon.
>> 
>> See-also PR at https://github.com/bitcoin/bitcoin/pull/16152
>> 
>> The release notes will liekly read:
>> 
>> P2P Changes
>> -----------
>> - The default value for the -peerbloomfilters configuration option (and,
>> thus, NODE_BLOOM support) has been changed to false.
>> This resolves well-known DoS vectors in Bitcoin Core, especially for
>> nodes with spinning disks. It is not anticipated that
>> this will result in a significant lack of availability of
>> NODE_BLOOM-enabled nodes in the coming years, however, clients
>> which rely on the availability of NODE_BLOOM-supporting nodes on the
>> P2P network should consider the process of migrating
>> to a more modern (and less trustful and privacy-violating) alternative
>> over the coming years.
>> 
>> Matt
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 5016 bytes --]