[bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Marco Falke]

Bitcoin Core may send "reject" messages as response to "tx", "block" or
"version" messages from a network peer when the message could not be accepted.

This feature is toggled by the `-enablebip61` command line option and has been
disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
time of writing). Nodes on the network can not generally be trusted to send
valid ("reject") messages, so this should only ever be used when connected to a
trusted node. At this time, I am not aware of any software that requires this
feature, and I would like to remove if from Bitcoin Core to make the codebase
slimmer, easier to understand and maintain. Let us know if your application
relies on this feature and you can not use any of the recommended alternatives:

* Testing or debugging of implementations of the Bitcoin P2P network protocol
  should be done by inspecting the log messages that are produced by a recent
  version of Bitcoin Core. Bitcoin Core logs debug messages
  (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
  (`-debuglogfile=<debug.log>`).

* Testing the validity of a block can be achieved by specific RPCs:
  - `submitblock`
  - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
    potentially invalid POW

* Testing the validity of a transaction can be achieved by specific RPCs:
  - `sendrawtransaction`
  - `testmempoolaccept`

* Wallets should not use the absence of "reject" messages to indicate a
  transaction has propagated the network, nor should wallets use "reject"
  messages to set transaction fees. Wallets should rather use fee estimation
  to determine transaction fees and set replace-by-fee if desired. Thus, they
  could wait until the transaction has confirmed (taking into account the fee
  target they set (compare the RPC `estimatesmartfee`)) or listen for the
  transaction announcement by other network peers to check for propagation.

I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
valid concerns about its removal.

Marco

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Dustin Dettmer]

[-- Attachment #1: Type: text/plain, Size: 3327 bytes --]

The reject message is helpful for figuring out why a tx was rejected.

It’s not useful for determining success, yes. Particularly when doing
segwit / newer types of tx’s as there’s always one or more pesky nodes who
still don’t support it and send a reject message for perfectly good tx’s.

But after a delay where you haven’t seen your tx propagated on the network,
it’s useful to know *why* it failed.

What would be nice is actually expanding this error message. Currently with
RBF tx’s “fee too small” is sent for both original transactions as well as
replacement transactions. So a bug accidentally sending spent txos
(currently in mempool) says “fee too small” instead of something more
appropriate like “fee too small to supersede existing unconfirmed
transaction.”

On Tue, Mar 5, 2019 at 7:26 PM Marco Falke via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be
> accepted.
>
> This feature is toggled by the `-enablebip61` command line option and has
> been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as
> of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected
> to a
> trusted node. At this time, I am not aware of any software that requires
> this
> feature, and I would like to remove if from Bitcoin Core to make the
> codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended
> alternatives:
>
> * Testing or debugging of implementations of the Bitcoin P2P network
> protocol
>   should be done by inspecting the log messages that are produced by a
> recent
>   version of Bitcoin Core. Bitcoin Core logs debug messages
>   (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>   (`-debuglogfile=<debug.log>`).
>
> * Testing the validity of a block can be achieved by specific RPCs:
>   - `submitblock`
>   - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>     potentially invalid POW
>
> * Testing the validity of a transaction can be achieved by specific RPCs:
>   - `sendrawtransaction`
>   - `testmempoolaccept`
>
> * Wallets should not use the absence of "reject" messages to indicate a
>   transaction has propagated the network, nor should wallets use "reject"
>   messages to set transaction fees. Wallets should rather use fee
> estimation
>   to determine transaction fees and set replace-by-fee if desired. Thus,
> they
>   could wait until the transaction has confirmed (taking into account the
> fee
>   target they set (compare the RPC `estimatesmartfee`)) or listen for the
>   transaction announcement by other network peers to check for propagation.
>
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless
> there are
> valid concerns about its removal.
>
> Marco
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 4106 bytes --]

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Andreas Schildbach]

Reject messages cannot be replaced for debugging user problems. At least
unless you plan to make RPC or bitcoind logfiles available via the P2P
protocol (both probably not a good idea).

The typical case is, I get mailed a wallet logfile with reject messages
and that's all I have. I cannot access the bitcoind logfile(s) of the
node(s) that generated the reject message in the first place. Nor can I
access their RPC interface.

I strongly suggest re-enabling reject messages by default before 0.18.


On 06/03/2019 01.53, Marco Falke via bitcoin-dev wrote:
> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be accepted.
> 
> This feature is toggled by the `-enablebip61` command line option and has been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected to a
> trusted node. At this time, I am not aware of any software that requires this
> feature, and I would like to remove if from Bitcoin Core to make the codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended alternatives:
> 
> * Testing or debugging of implementations of the Bitcoin P2P network protocol
>   should be done by inspecting the log messages that are produced by a recent
>   version of Bitcoin Core. Bitcoin Core logs debug messages
>   (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>   (`-debuglogfile=<debug.log>`).
> 
> * Testing the validity of a block can be achieved by specific RPCs:
>   - `submitblock`
>   - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>     potentially invalid POW
> 
> * Testing the validity of a transaction can be achieved by specific RPCs:
>   - `sendrawtransaction`
>   - `testmempoolaccept`
> 
> * Wallets should not use the absence of "reject" messages to indicate a
>   transaction has propagated the network, nor should wallets use "reject"
>   messages to set transaction fees. Wallets should rather use fee estimation
>   to determine transaction fees and set replace-by-fee if desired. Thus, they
>   could wait until the transaction has confirmed (taking into account the fee
>   target they set (compare the RPC `estimatesmartfee`)) or listen for the
>   transaction announcement by other network peers to check for propagation.
> 
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
> valid concerns about its removal.
> 
> Marco
> 

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Sjors Provoost]

Can you elaborate a bit on what kind of reject messages your users are getting? I assume the users wallet connects directly to the Bitcoin p2p network?

What does the wallet do when a transaction is rejected? Does it forget about it (that seems unsafe) or compose another one (with overlapping inputs)?

Sjors

> Op 6 mrt. 2019, om 17:49 heeft Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> het volgende geschreven:
> 
> Reject messages cannot be replaced for debugging user problems. At least
> unless you plan to make RPC or bitcoind logfiles available via the P2P
> protocol (both probably not a good idea).
> 
> The typical case is, I get mailed a wallet logfile with reject messages
> and that's all I have. I cannot access the bitcoind logfile(s) of the
> node(s) that generated the reject message in the first place. Nor can I
> access their RPC interface.
> 
> I strongly suggest re-enabling reject messages by default before 0.18.
> 
> 
> On 06/03/2019 01.53, Marco Falke via bitcoin-dev wrote:
>> Bitcoin Core may send "reject" messages as response to "tx", "block" or
>> "version" messages from a network peer when the message could not be accepted.
>> 
>> This feature is toggled by the `-enablebip61` command line option and has been
>> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
>> time of writing). Nodes on the network can not generally be trusted to send
>> valid ("reject") messages, so this should only ever be used when connected to a
>> trusted node. At this time, I am not aware of any software that requires this
>> feature, and I would like to remove if from Bitcoin Core to make the codebase
>> slimmer, easier to understand and maintain. Let us know if your application
>> relies on this feature and you can not use any of the recommended alternatives:
>> 
>> * Testing or debugging of implementations of the Bitcoin P2P network protocol
>>  should be done by inspecting the log messages that are produced by a recent
>>  version of Bitcoin Core. Bitcoin Core logs debug messages
>>  (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>>  (`-debuglogfile=<debug.log>`).
>> 
>> * Testing the validity of a block can be achieved by specific RPCs:
>>  - `submitblock`
>>  - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>>    potentially invalid POW
>> 
>> * Testing the validity of a transaction can be achieved by specific RPCs:
>>  - `sendrawtransaction`
>>  - `testmempoolaccept`
>> 
>> * Wallets should not use the absence of "reject" messages to indicate a
>>  transaction has propagated the network, nor should wallets use "reject"
>>  messages to set transaction fees. Wallets should rather use fee estimation
>>  to determine transaction fees and set replace-by-fee if desired. Thus, they
>>  could wait until the transaction has confirmed (taking into account the fee
>>  target they set (compare the RPC `estimatesmartfee`)) or listen for the
>>  transaction announcement by other network peers to check for propagation.
>> 
>> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
>> valid concerns about its removal.
>> 
>> Marco
>> 
> 
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Andreas Schildbach]

Yes, I'm talking about P2P connections.

First and foremost, reject messages are an indication that the
transaction isn't going to confirm. Without these messages, we'd need to
revert to pre-BIP61 behaviour of using a timeout for reception of
network confirmations.

Regarding the content, these cases are useful to distinguish:

- Not enough fee
- UTXO already spent
- Tx validity/standardness (e.g. invalid signature)

While the last one in theority wouldn't be necessary if you produced
your software bug-free to begin with, this just isn't how software
development works. Developers need any indication they can get.

The first two happen even in the ideal case. Fees are impossible to
predict, and unintentional double spends happen because users clone
their wallet state.


On 07/03/2019 14.59, Sjors Provoost via bitcoin-dev wrote:
> Can you elaborate a bit on what kind of reject messages your users are getting? I assume the users wallet connects directly to the Bitcoin p2p network?
> 
> What does the wallet do when a transaction is rejected? Does it forget about it (that seems unsafe) or compose another one (with overlapping inputs)?
> 
> Sjors
> 
>> Op 6 mrt. 2019, om 17:49 heeft Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> het volgende geschreven:
>>
>> Reject messages cannot be replaced for debugging user problems. At least
>> unless you plan to make RPC or bitcoind logfiles available via the P2P
>> protocol (both probably not a good idea).
>>
>> The typical case is, I get mailed a wallet logfile with reject messages
>> and that's all I have. I cannot access the bitcoind logfile(s) of the
>> node(s) that generated the reject message in the first place. Nor can I
>> access their RPC interface.
>>
>> I strongly suggest re-enabling reject messages by default before 0.18.
>>
>>
>> On 06/03/2019 01.53, Marco Falke via bitcoin-dev wrote:
>>> Bitcoin Core may send "reject" messages as response to "tx", "block" or
>>> "version" messages from a network peer when the message could not be accepted.
>>>
>>> This feature is toggled by the `-enablebip61` command line option and has been
>>> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
>>> time of writing). Nodes on the network can not generally be trusted to send
>>> valid ("reject") messages, so this should only ever be used when connected to a
>>> trusted node. At this time, I am not aware of any software that requires this
>>> feature, and I would like to remove if from Bitcoin Core to make the codebase
>>> slimmer, easier to understand and maintain. Let us know if your application
>>> relies on this feature and you can not use any of the recommended alternatives:
>>>
>>> * Testing or debugging of implementations of the Bitcoin P2P network protocol
>>>  should be done by inspecting the log messages that are produced by a recent
>>>  version of Bitcoin Core. Bitcoin Core logs debug messages
>>>  (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>>>  (`-debuglogfile=<debug.log>`).
>>>
>>> * Testing the validity of a block can be achieved by specific RPCs:
>>>  - `submitblock`
>>>  - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>>>    potentially invalid POW
>>>
>>> * Testing the validity of a transaction can be achieved by specific RPCs:
>>>  - `sendrawtransaction`
>>>  - `testmempoolaccept`
>>>
>>> * Wallets should not use the absence of "reject" messages to indicate a
>>>  transaction has propagated the network, nor should wallets use "reject"
>>>  messages to set transaction fees. Wallets should rather use fee estimation
>>>  to determine transaction fees and set replace-by-fee if desired. Thus, they
>>>  could wait until the transaction has confirmed (taking into account the fee
>>>  target they set (compare the RPC `estimatesmartfee`)) or listen for the
>>>  transaction announcement by other network peers to check for propagation.
>>>
>>> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
>>> valid concerns about its removal.
>>>
>>> Marco
>>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Aymeric Vitte]

Bitcoin-transactions did use this "feature", but does not rely on it any
longer since I observed some strange behavior sometimes (no reject
message for bad tx, with suprnova for example as far as I remember),
then it doublechecks using getdata to see if the tx is in mempool

Indeed you can't trust what a node tells you with or without reject
(idem for getdata but more difficult to fake and better than nothing)

Then I don't see any problem to remove it, taking into account also that
the error message is too vague to be really helpful
https://github.com/bitcoin/bitcoin/issues/11891

Le 06/03/2019 à 01:53, Marco Falke via bitcoin-dev a écrit :
> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be accepted.
>
> This feature is toggled by the `-enablebip61` command line option and has been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected to a
> trusted node. At this time, I am not aware of any software that requires this
> feature, and I would like to remove if from Bitcoin Core to make the codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended alternatives:
>
> * Testing or debugging of implementations of the Bitcoin P2P network protocol
>   should be done by inspecting the log messages that are produced by a recent
>   version of Bitcoin Core. Bitcoin Core logs debug messages
>   (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>   (`-debuglogfile=<debug.log>`).
>
> * Testing the validity of a block can be achieved by specific RPCs:
>   - `submitblock`
>   - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>     potentially invalid POW
>
> * Testing the validity of a transaction can be achieved by specific RPCs:
>   - `sendrawtransaction`
>   - `testmempoolaccept`
>
> * Wallets should not use the absence of "reject" messages to indicate a
>   transaction has propagated the network, nor should wallets use "reject"
>   messages to set transaction fees. Wallets should rather use fee estimation
>   to determine transaction fees and set replace-by-fee if desired. Thus, they
>   could wait until the transaction has confirmed (taking into account the fee
>   target they set (compare the RPC `estimatesmartfee`)) or listen for the
>   transaction announcement by other network peers to check for propagation.
>
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless there are
> valid concerns about its removal.
>
> Marco
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

-- 
Move your coins by yourself (browser version): https://peersm.com/wallet
Bitcoin transactions made simple: https://github.com/Ayms/bitcoin-transactions
Zcash wallets made simple: https://github.com/Ayms/zcash-wallets
Bitcoin wallets made simple: https://github.com/Ayms/bitcoin-wallets
Get the torrent dynamic blocklist: http://peersm.com/getblocklist
Check the 10 M passwords list: http://peersm.com/findmyass
Anti-spies and private torrents, dynamic blocklist: http://torrent-live.org
Peersm : http://www.peersm.com
torrent-live: https://github.com/Ayms/torrent-live
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Wilmer Paulino]

Hi Marco,

> At this time, I am not aware of any software that requires this feature, and I
> would like to remove if from Bitcoin Core to make the codebase slimmer, easier
> to understand and maintain.

Neutrino[1], a light client implementation that uses BIPs 157 and 158,
currently relies on receiving reject messages from its peers when broadcasting
a transaction to the network. I've personally gone through the relevant parts
of the Bitcoin Core codebase involving reject messages and respectfully
disagree that removing it would help much in terms of comprehension and
maintainability. IMO, the benefits outweigh this small cost.

> Nodes on the network can not generally be trusted to send valid ("reject")
> messages, so this should only ever be used when connected to a trusted node.

Nodes in the network generally rely on the assumption that they are connected
to at least one honest peer, so we can actually converge on the set of honest
peers and ban/disconnect any who send an invalid reject message for a valid
transaction.

> Let us know if your application relies on this feature and you can not use any
> of the recommended alternatives:

Unfortunately, none of the recommended alternatives work for our use case. The
main thing we want to identify when broadcasting a transaction is whether is
it invalid or not. As long as it is valid, reject messages aren't required as
the light client can just rebroadcast the transaction upon every new block to
ensure it is eventually included in the chain. It can then stop rebroadcasting
it once it detects it has confirmed on-chain through its filters. However, if
it is invalid, some of the validity checks required cannot be performed by
light clients as they do not have a mempool and/or UTXO set.

Reject messages also useful when developing new light clients, as we can get
some feedback from the network on why a transaction was rejected, which helps
identify potential bugs in their transaction crafting logic. I understand that
this can be done by setting up test nodes with the flag enabled, but this
justifies that the feature should at least exist and not be completely
removed.

> * Testing the validity of a transaction can be achieved by specific RPCs:
>  - `sendrawtransaction`
>  - `testmempoolaccept`

These RPCs are not helpful for light clients. Even for full nodes, in the case
of `testmempoolaccept`, mempool conditions can quickly change and cause a
transaction to be rejected after the fact. One alternative would be for a
third-party to set up an endpoint where users can submit their transactions
to, but now you're placing your trust solely on them, rather than the network,
which doesn't seem like a reasonable or comparable compromise.

With that said, I believe the feature should remain enabled by default in
order to aid the light clients of the network. If we disable them by default,
no one will bother to enable them manually, and light clients won't be able to
realize they are broadcasting invalid transactions.

[1] https://github.com/lightninglabs/neutrino

- Wilmer

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Eric Voskuil]

>> On Mar 7, 2019, at 19:09, Wilmer Paulino via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> ...
>> Nodes on the network can not generally be trusted to send valid ("reject") messages, so this should only ever be used when connected to a trusted node.
> 
> Nodes in the network generally rely on the assumption that they are connected to at least one honest peer, so we can actually converge on the set of honest peers and ban/disconnect any who send an invalid reject message for a valid transaction.

This implies the reject message is valid only when it is expected (i.e. the sender is knowingly sending invalid transactions), which is presumably useful only in a local development environment.

e

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Gregory Maxwell]

On Thu, Mar 7, 2019 at 11:46 PM Andreas Schildbach via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> First and foremost, reject messages are an indication that the
> transaction isn't going to confirm. Without these messages, we'd need to
> revert to pre-BIP61 behaviour of using a timeout for reception of
> network confirmations.

That is already required because even in the presence of perfectly
honest and cooperative hosts reject messages at most can only tell you
about first-hop behaviour. It won't even tell you if the transaction
was ever even attempted to be sent to a next hop.  So alternative
handling must be provided and must be reliable for the software to
work at all regardless of reject messages.

> - Not enough fee

Rejection on low fee (over the static minimum feerate) only happens at
the point where the nodes mempool is full, which is already at a point
where you might be waiting weeks for confirmation.

Rejection causes were also not stable or reliable because the validity
criteria cannot generally be tested independently. For example, if a
transaction is queued due to missing a parent it isn't rejected
because missing the parent is often a temporary issue, but its feerate
cannot be measured without the parent. Later, when the parent is
obtained, the transaction can then be rejected due to feerate-- but no
reject is sent then.

Output already spend is often completely indistinguishable from a
missing parent and can't get rejects generated for it generally.

Similarly, the error state detected for things like invalid signatures
are often not very useful. The software knows that script execution
returned false, but in the general case _why_ it returned false is not
clear, and a straightforward high performance validation
implementation doesn't necessarily yield a good way of figuring out
and propagating up that information.  (I think invalid signatures end
up returning a stack-nonempty state from validation currently, as an
example of that).

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Andreas Schildbach]

(Posting again, since my previous reply didn't appear)


On 08/03/2019 01.52, Gregory Maxwell via bitcoin-dev wrote:

> That is already required because even in the presence of perfectly
> honest and cooperative hosts reject messages at most can only tell you
> about first-hop behaviour. It won't even tell you if the transaction
> was ever even attempted to be sent to a next hop.  So alternative
> handling must be provided and must be reliable for the software to
> work at all regardless of reject messages.
>
> Rejection causes were also not stable or reliable because the validity
> criteria cannot generally be tested independently. For example, if a
> transaction is queued due to missing a parent it isn't rejected
> because missing the parent is often a temporary issue, but its feerate
> cannot be measured without the parent. Later, when the parent is
> obtained, the transaction can then be rejected due to feerate-- but no
> reject is sent then.

These two cases are understood and handled by current code. Generally
the idea is take reject messages serious, but don't overrate the lack
of. Luckily, network confirmations fill the gap. (Yes, a timeout is
still useful. But at least it almost never happens.)

> Similarly, the error state detected for things like invalid signatures
> are often not very useful. The software knows that script execution
> returned false, but in the general case _why_ it returned false is not
> clear, and a straightforward high performance validation
> implementation doesn't necessarily yield a good way of figuring out
> and propagating up that information.  (I think invalid signatures end
> up returning a stack-nonempty state from validation currently, as an
> example of that).

Nevertheless, it has been proven as useful in debugging (just recently
when I implemented the witness signature hash in bitcoinj). I think
Wilmer Paulino summed up this point quite nicely in his reply to this
thread.

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Gregory Maxwell]

On Tue, Mar 12, 2019 at 7:45 PM Andreas Schildbach via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> These two cases are understood and handled by current code. Generally
> the idea is take reject messages serious, but don't overrate the lack
> of. Luckily, network confirmations fill the gap. (Yes, a timeout is

I'd like to better understand this, but it would be easier to just
read the code than ask a bunch of questions. I tried looking for the
handling of reject messages in Android  Bitcoin Wallet and BitcoinJ
and didn't really find and handling other than logging exceptions.
Would you mind giving me a couple pointers to where in the code
they're handled?

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Andreas Schildbach]

On 12/03/2019 23.14, Gregory Maxwell via bitcoin-dev wrote:
> On Tue, Mar 12, 2019 at 7:45 PM Andreas Schildbach via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> These two cases are understood and handled by current code. Generally
>> the idea is take reject messages serious, but don't overrate the lack
>> of. Luckily, network confirmations fill the gap. (Yes, a timeout is
> 
> I'd like to better understand this, but it would be easier to just
> read the code than ask a bunch of questions. I tried looking for the
> handling of reject messages in Android  Bitcoin Wallet and BitcoinJ
> and didn't really find and handling other than logging exceptions.
> Would you mind giving me a couple pointers to where in the code
> they're handled?

It's implemented in bitcoinj's TransactionBroadcast class. Received
reject messages are collected and -- if a certain consensus (currently:
half of connected peers) is reached -- a RejectedTransactionException is
raised.

The handling of that exception in Bitcoin Wallet is extremely
rudimentary. I think it still only shows the exception message. But
certainly I was hoping to improve on this soon.

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Oscar Guindzberg]

> I'd like to better understand this, but it would be easier to just
> read the code than ask a bunch of questions. I tried looking for the
> handling of reject messages in Android  Bitcoin Wallet and BitcoinJ
> and didn't really find and handling other than logging exceptions.
> Would you mind giving me a couple pointers to where in the code
> they're handled?

https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/core/TransactionBroadcast.java#L93-L108

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Dustin Dettmer]

[-- Attachment #1: Type: text/plain, Size: 2177 bytes --]

I’ve solved the same problem in a different way.

1) Submit a transaction
2) Collect all reject messages (that have matching txid in the reject data)
3) Wait 16 seconds after first error message received (chosen semirandomly
from trial and error) before processing errors
4) Wait for our txid to be submitted back to us through the mempool, if we
get it notify success and delete all pending error events
5) Signal failure with the given reject code if present (after the 16
seconds have elapsed)
6) If no error or success after 20 seconds, signal timeout failure

This works fairly well in testing. Newer transaction types seem to generate
reject codes 100% of the time (from at least one node when sending to 4
nodes) so this culling / time delay approach is essentially required.

On a related note: One issue is that RBF attempts with too small a fee and
accidental double spends (with enough fee for 1 tx but not a RBF) both
generate the same reject code: not enough fee.

A new reject code for RBF based too small of fee would definitely make for
a better user experience as I’ve seen this exact problem create confusion
for users.

Removing reject codes would make for a much worse user experience. “Your tx
failed and we have no idea why” would be the only message and it would
require waiting for a full timeout.

On Wed, Mar 13, 2019 at 3:16 PM Oscar Guindzberg via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> > I'd like to better understand this, but it would be easier to just
> > read the code than ask a bunch of questions. I tried looking for the
> > handling of reject messages in Android  Bitcoin Wallet and BitcoinJ
> > and didn't really find and handling other than logging exceptions.
> > Would you mind giving me a couple pointers to where in the code
> > they're handled?
>
>
> https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/core/TransactionBroadcast.java#L93-L108
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 3184 bytes --]

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Aymeric Vitte]

[-- Attachment #1: Type: text/plain, Size: 4114 bytes --]

Apparently I don't have the same experience than others here, what I
encountered is no reject message received for wrong txs, but from what I
understand here it's not unusual to receive reject message for valid
txs, then I don't see how it can be really helpful/relied, given also
that the reject messages are unclear and even can be misleading

As it was written already I found it useful only for debugging purposes,
at least it can give some kind of ideas about what happened,
bitcoin-transactions is implementing the bitcoin protocol but does not
act as a node and does not pretend to fake a node behavior waiting for
example to get the tx back, is the method of sending a getdata for a
given tx to see if it was accepted by a node wrong ? It can't guarantee
100% that it was successful and will propagate but I see that you are
doing completely different things

Le 13/03/2019 à 23:30, Dustin Dettmer via bitcoin-dev a écrit :
> I’ve solved the same problem in a different way.
>
> 1) Submit a transaction
> 2) Collect all reject messages (that have matching txid in the reject
> data)
> 3) Wait 16 seconds after first error message received (chosen
> semirandomly from trial and error) before processing errors
> 4) Wait for our txid to be submitted back to us through the mempool,
> if we get it notify success and delete all pending error events
> 5) Signal failure with the given reject code if present (after the 16
> seconds have elapsed)
> 6) If no error or success after 20 seconds, signal timeout failure
>
> This works fairly well in testing. Newer transaction types seem to
> generate reject codes 100% of the time (from at least one node when
> sending to 4 nodes) so this culling / time delay approach is
> essentially required.
>
> On a related note: One issue is that RBF attempts with too small a fee
> and accidental double spends (with enough fee for 1 tx but not a RBF)
> both generate the same reject code: not enough fee.
>
> A new reject code for RBF based too small of fee would definitely make
> for a better user experience as I’ve seen this exact problem create
> confusion for users.
>
> Removing reject codes would make for a much worse user experience.
> “Your tx failed and we have no idea why” would be the only message and
> it would require waiting for a full timeout.
>
> On Wed, Mar 13, 2019 at 3:16 PM Oscar Guindzberg via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org
> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>
>     > I'd like to better understand this, but it would be easier to just
>     > read the code than ask a bunch of questions. I tried looking for the
>     > handling of reject messages in Android  Bitcoin Wallet and BitcoinJ
>     > and didn't really find and handling other than logging exceptions.
>     > Would you mind giving me a couple pointers to where in the code
>     > they're handled?
>
>     https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/core/TransactionBroadcast.java#L93-L108
>     _______________________________________________
>     bitcoin-dev mailing list
>     bitcoin-dev@lists.linuxfoundation.org
>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>     https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

-- 
Move your coins by yourself (browser version): https://peersm.com/wallet
Bitcoin transactions made simple: https://github.com/Ayms/bitcoin-transactions
Zcash wallets made simple: https://github.com/Ayms/zcash-wallets
Bitcoin wallets made simple: https://github.com/Ayms/bitcoin-wallets
Get the torrent dynamic blocklist: http://peersm.com/getblocklist
Check the 10 M passwords list: http://peersm.com/findmyass
Anti-spies and private torrents, dynamic blocklist: http://torrent-live.org
Peersm : http://www.peersm.com
torrent-live: https://github.com/Ayms/torrent-live
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms


[-- Attachment #2: Type: text/html, Size: 7434 bytes --]

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[John Newbery]

[-- Attachment #1: Type: text/plain, Size: 3399 bytes --]

Following discussion on this mailing list, support for BIP 61 REJECT
messages was not removed from Bitcoin Core in V0.19. The behaviour in that
upcoming release is that REJECT messages are disabled by default and can be
enabled using the `-enablebip61` command line option.

Support for REJECT messages will be removed entirely in Bitcoin Core V0.20,
expected for release in mid 2020. The PR to remove support was merged into
Bitcoin Core's master branch this week.

Adoption of new Bitcoin Core versions across reachable nodes generally
takes several months. https://bitnodes.earn.com/dashboard/?days=365 shows
that although v0.18 was released in May 2019, there are still several
hundred reachable nodes on V0.17, V0.16, V0.15 and earlier software.
Software that currently use REJECT messages from public nodes for
troubleshooting issues therefore have plenty of time to transition to one
of the methods listed by Marco in the email above.

John

On Tue, Mar 5, 2019 at 10:28 PM Marco Falke via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Bitcoin Core may send "reject" messages as response to "tx", "block" or
> "version" messages from a network peer when the message could not be
> accepted.
>
> This feature is toggled by the `-enablebip61` command line option and has
> been
> disabled by default since Bitcoin Core version 0.18.0 (not yet released as
> of
> time of writing). Nodes on the network can not generally be trusted to send
> valid ("reject") messages, so this should only ever be used when connected
> to a
> trusted node. At this time, I am not aware of any software that requires
> this
> feature, and I would like to remove if from Bitcoin Core to make the
> codebase
> slimmer, easier to understand and maintain. Let us know if your application
> relies on this feature and you can not use any of the recommended
> alternatives:
>
> * Testing or debugging of implementations of the Bitcoin P2P network
> protocol
>   should be done by inspecting the log messages that are produced by a
> recent
>   version of Bitcoin Core. Bitcoin Core logs debug messages
>   (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>   (`-debuglogfile=<debug.log>`).
>
> * Testing the validity of a block can be achieved by specific RPCs:
>   - `submitblock`
>   - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>     potentially invalid POW
>
> * Testing the validity of a transaction can be achieved by specific RPCs:
>   - `sendrawtransaction`
>   - `testmempoolaccept`
>
> * Wallets should not use the absence of "reject" messages to indicate a
>   transaction has propagated the network, nor should wallets use "reject"
>   messages to set transaction fees. Wallets should rather use fee
> estimation
>   to determine transaction fees and set replace-by-fee if desired. Thus,
> they
>   could wait until the transaction has confirmed (taking into account the
> fee
>   target they set (compare the RPC `estimatesmartfee`)) or listen for the
>   transaction announcement by other network peers to check for propagation.
>
> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless
> there are
> valid concerns about its removal.
>
> Marco
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 4243 bytes --]

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Andreas Schildbach]

On 16/10/2019 18.43, John Newbery via bitcoin-dev wrote:

> Following discussion on this mailing list, support for BIP 61 REJECT
> messages was not removed from Bitcoin Core in V0.19. The behaviour in
> that upcoming release is that REJECT messages are disabled by default
> and can be enabled using the `-enablebip61` command line option.

Is there a NODE_* bit we can use to pick peers that support this
(useful!) feature?

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Eric Voskuil]

As this is a P2P protocol change it should be exposed as a version increment (and a BIP), not just as a conditional service. If the intent is to retain this protocol indefinitely, exposing it conditionally, then a service bit would make sense, but it remains a protocol change.

BIP61 is explicit:

“All implementations of the P2P protocol version 70,002 and later should support the reject message.“

e

> On Oct 17, 2019, at 12:54, Andreas Schildbach via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
> 
> On 16/10/2019 18.43, John Newbery via bitcoin-dev wrote:
> 
>> Following discussion on this mailing list, support for BIP 61 REJECT
>> messages was not removed from Bitcoin Core in V0.19. The behaviour in
>> that upcoming release is that REJECT messages are disabled by default
>> and can be enabled using the `-enablebip61` command line option.
> 
> Is there a NODE_* bit we can use to pick peers that support this
> (useful!) feature?
> 
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[John Newbery]

[-- Attachment #1: Type: text/plain, Size: 3754 bytes --]

> Is there a NODE_* bit we can use to pick peers that support this (useful!)
feature?

No. BIP 61 has no mechanism for advertising that a node will send REJECT
messages.

On Wed, Oct 16, 2019 at 12:43 PM John Newbery <john@johnnewbery.com> wrote:

> Following discussion on this mailing list, support for BIP 61 REJECT
> messages was not removed from Bitcoin Core in V0.19. The behaviour in that
> upcoming release is that REJECT messages are disabled by default and can be
> enabled using the `-enablebip61` command line option.
>
> Support for REJECT messages will be removed entirely in Bitcoin Core
> V0.20, expected for release in mid 2020. The PR to remove support was
> merged into Bitcoin Core's master branch this week.
>
> Adoption of new Bitcoin Core versions across reachable nodes generally
> takes several months. https://bitnodes.earn.com/dashboard/?days=365 shows
> that although v0.18 was released in May 2019, there are still several
> hundred reachable nodes on V0.17, V0.16, V0.15 and earlier software.
> Software that currently use REJECT messages from public nodes for
> troubleshooting issues therefore have plenty of time to transition to one
> of the methods listed by Marco in the email above.
>
> John
>
> On Tue, Mar 5, 2019 at 10:28 PM Marco Falke via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> Bitcoin Core may send "reject" messages as response to "tx", "block" or
>> "version" messages from a network peer when the message could not be
>> accepted.
>>
>> This feature is toggled by the `-enablebip61` command line option and has
>> been
>> disabled by default since Bitcoin Core version 0.18.0 (not yet released
>> as of
>> time of writing). Nodes on the network can not generally be trusted to
>> send
>> valid ("reject") messages, so this should only ever be used when
>> connected to a
>> trusted node. At this time, I am not aware of any software that requires
>> this
>> feature, and I would like to remove if from Bitcoin Core to make the
>> codebase
>> slimmer, easier to understand and maintain. Let us know if your
>> application
>> relies on this feature and you can not use any of the recommended
>> alternatives:
>>
>> * Testing or debugging of implementations of the Bitcoin P2P network
>> protocol
>>   should be done by inspecting the log messages that are produced by a
>> recent
>>   version of Bitcoin Core. Bitcoin Core logs debug messages
>>   (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>>   (`-debuglogfile=<debug.log>`).
>>
>> * Testing the validity of a block can be achieved by specific RPCs:
>>   - `submitblock`
>>   - `getblocktemplate` with `'mode'` set to `'proposal'` for blocks with
>>     potentially invalid POW
>>
>> * Testing the validity of a transaction can be achieved by specific RPCs:
>>   - `sendrawtransaction`
>>   - `testmempoolaccept`
>>
>> * Wallets should not use the absence of "reject" messages to indicate a
>>   transaction has propagated the network, nor should wallets use "reject"
>>   messages to set transaction fees. Wallets should rather use fee
>> estimation
>>   to determine transaction fees and set replace-by-fee if desired. Thus,
>> they
>>   could wait until the transaction has confirmed (taking into account the
>> fee
>>   target they set (compare the RPC `estimatesmartfee`)) or listen for the
>>   transaction announcement by other network peers to check for
>> propagation.
>>
>> I propose to remove "reject" messages from Bitcoin Core 0.19.0 unless
>> there are
>> valid concerns about its removal.
>>
>> Marco
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

[-- Attachment #2: Type: text/html, Size: 5093 bytes --]

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[David A. Harding]

On Thu, Oct 17, 2019 at 01:16:47PM -0700, Eric Voskuil via bitcoin-dev wrote:
> As this is a P2P protocol change it should be exposed as a version
> increment (and a BIP) [...]
>
> BIP61 is explicit:
> 
> “All implementations of the P2P protocol version 70,002 and later
> should support the reject message.“

I don't think a new BIP or a version number increment is necessary.

1. "Should support" isn't the same as "must support".  See
   https://tools.ietf.org/html/rfc2119 ; by that reading,
   implementations with protocol versions above 70,002 are not required
   to support the reject message.

2. If you don't implement a BIP, as Bitcoin Core explicitly doesn't any
   more for BIP61[1], you're not bound by its conditions.

-Dave

[1] https://github.com/bitcoin/bitcoin/blob/master/doc/bips.md  "BIP61
[...] Support was removed in v0.20.0"

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

I agree, thanks.

FWIW I’ve never been a fan of the ‘reject’ message, or its implementation.

https://github.com/bitcoin/bips/wiki/Comments:BIP-0061

e

> On Oct 18, 2019, at 18:46, David A. Harding <dave@dtrt.org> wrote:
> 
> On Thu, Oct 17, 2019 at 01:16:47PM -0700, Eric Voskuil via bitcoin-dev wrote:
>> As this is a P2P protocol change it should be exposed as a version
>> increment (and a BIP) [...]
>> 
>> BIP61 is explicit:
>> 
>> “All implementations of the P2P protocol version 70,002 and later
>> should support the reject message.“
> 
> I don't think a new BIP or a version number increment is necessary.
> 
> 1. "Should support" isn't the same as "must support".  See
>   https://tools.ietf.org/html/rfc2119 ; by that reading,
>   implementations with protocol versions above 70,002 are not required
>   to support the reject message.
> 
> 2. If you don't implement a BIP, as Bitcoin Core explicitly doesn't any
>   more for BIP61[1], you're not bound by its conditions.
> 
> -Dave
> 
> [1] https://github.com/bitcoin/bitcoin/blob/master/doc/bips.md  "BIP61
> [...] Support was removed in v0.20.0"

[-- Attachment #2: Type: text/html, Size: 2217 bytes --]

Re: [bitcoin-dev] Removal of reject network messages from Bitcoin Core (BIP61)

[Andreas Schildbach]

I guess then the best way to discover nodes that have reject messages
enabled is connecting/disconnecting to random nodes and send them
invalid transactions and keep the ones which reply with a reject message.


On 18/10/2019 22.53, John Newbery via bitcoin-dev wrote:
>> Is there a NODE_* bit we can use to pick peers that support this
> (useful!) feature?
> 
> No. BIP 61 has no mechanism for advertising that a node will send REJECT
> messages.
> 
> On Wed, Oct 16, 2019 at 12:43 PM John Newbery <john@johnnewbery.com
> <mailto:john@johnnewbery.com>> wrote:
> 
>     Following discussion on this mailing list, support for BIP 61 REJECT
>     messages was not removed from Bitcoin Core in V0.19. The behaviour
>     in that upcoming release is that REJECT messages are disabled by
>     default and can be enabled using the `-enablebip61` command line option.
> 
>     Support for REJECT messages will be removed entirely in Bitcoin Core
>     V0.20, expected for release in mid 2020. The PR to remove support
>     was merged into Bitcoin Core's master branch this week.
> 
>     Adoption of new Bitcoin Core versions across reachable nodes
>     generally takes several months.
>     https://bitnodes.earn.com/dashboard/?days=365 shows that although
>     v0.18 was released in May 2019, there are still several hundred
>     reachable nodes on V0.17, V0.16, V0.15 and earlier software.
>     Software that currently use REJECT messages from public nodes for
>     troubleshooting issues therefore have plenty of time to transition
>     to one of the methods listed by Marco in the email above.
> 
>     John
> 
>     On Tue, Mar 5, 2019 at 10:28 PM Marco Falke via bitcoin-dev
>     <bitcoin-dev@lists.linuxfoundation.org
>     <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
> 
>         Bitcoin Core may send "reject" messages as response to "tx",
>         "block" or
>         "version" messages from a network peer when the message could
>         not be accepted.
> 
>         This feature is toggled by the `-enablebip61` command line
>         option and has been
>         disabled by default since Bitcoin Core version 0.18.0 (not yet
>         released as of
>         time of writing). Nodes on the network can not generally be
>         trusted to send
>         valid ("reject") messages, so this should only ever be used when
>         connected to a
>         trusted node. At this time, I am not aware of any software that
>         requires this
>         feature, and I would like to remove if from Bitcoin Core to make
>         the codebase
>         slimmer, easier to understand and maintain. Let us know if your
>         application
>         relies on this feature and you can not use any of the
>         recommended alternatives:
> 
>         * Testing or debugging of implementations of the Bitcoin P2P
>         network protocol
>           should be done by inspecting the log messages that are
>         produced by a recent
>           version of Bitcoin Core. Bitcoin Core logs debug messages
>           (`-debug=<category>`) to a stream (`-printtoconsole`) or to a file
>           (`-debuglogfile=<debug.log>`).
> 
>         * Testing the validity of a block can be achieved by specific RPCs:
>           - `submitblock`
>           - `getblocktemplate` with `'mode'` set to `'proposal'` for
>         blocks with
>             potentially invalid POW
> 
>         * Testing the validity of a transaction can be achieved by
>         specific RPCs:
>           - `sendrawtransaction`
>           - `testmempoolaccept`
> 
>         * Wallets should not use the absence of "reject" messages to
>         indicate a
>           transaction has propagated the network, nor should wallets use
>         "reject"
>           messages to set transaction fees. Wallets should rather use
>         fee estimation
>           to determine transaction fees and set replace-by-fee if
>         desired. Thus, they
>           could wait until the transaction has confirmed (taking into
>         account the fee
>           target they set (compare the RPC `estimatesmartfee`)) or
>         listen for the
>           transaction announcement by other network peers to check for
>         propagation.
> 
>         I propose to remove "reject" messages from Bitcoin Core 0.19.0
>         unless there are
>         valid concerns about its removal.
> 
>         Marco
>         _______________________________________________
>         bitcoin-dev mailing list
>         bitcoin-dev@lists.linuxfoundation.org
>         <mailto:bitcoin-dev@lists.linuxfoundation.org>
>         https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>