[Bitcoin-development] Feedback requested: "reject" p2p message

[Bitcoin-development] Feedback requested: "reject" p2p message

[Gavin Andresen]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

Mike Hearn has been lobbying for an "error" message in the Bitcoin p2p
protocol for years (at least since the "ban peers if they send us garbage"
denial-of-service mitigation code was pull-requested). This came up again
with my proposed "smartfee" changes, which would drop low-priority or
low-fee transactions.

In short, giving peers feedback about why their blocks or transactions are
dropped or why they are being banned should help interoperability between
different implementations, and will give SPV (simplified payment
verification) clients feedback when their transactions are rejected due to
insufficient priority or fees.

See the gist for details, I'm looking for feedback and planning on
implementing this before circling back to finish the 'smart fee' work:

   https://gist.github.com/gavinandresen/7079034

-- 
--
Gavin Andresen

[-- Attachment #2: Type: text/html, Size: 1084 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Jean-Paul Kogelman]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]


Would it make sense to use either fixed length strings or maybe even enums?

On Oct 25, 2013, at 05:34 PM, Gavin Andresen <gavinandresen@gmail.com> wrote:
Mike Hearn has been lobbying for an "error" message in the Bitcoin p2p protocol for years (at least since the "ban peers if they send us garbage" denial-of-service mitigation code was pull-requested). This came up again with my proposed "smartfee" changes, which would drop low-priority or low-fee transactions.

[-- Attachment #2.1: Type: text/html, Size: 573 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Gavin]

On Oct 26, 2013, at 11:01 AM, Jean-Paul Kogelman <jeanpaulkogelman@me.com> wrote:

> 
> Would it make sense to use either fixed length strings or maybe even enums?

No. Enums or fixed length strings just make it harder to extend, for no benefit (bandwidth of 'reject' messages doesn't matter, they will be rare and are not relayed).

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[kjj]

The HTTP status code system seems to work well enough, and seems to give 
the best of both worlds.  A 3 digit numeric code that is 
machine-readable, and a freeform text note for humans.

The clever part about that system was in realizing that the numeric 
codes didn't need to account for every possible error. They just need to 
give the other node the most useful information, like "try that again 
later, I'm having a temporary problem" vs. "That is just plain wrong and 
it will still be wrong next time too, so don't bother to retry".

We can leave it to the humans to puzzle out the meaning of "403: values 
of txid gives rise to dom!"

Gavin wrote:
>
> On Oct 26, 2013, at 11:01 AM, Jean-Paul Kogelman <jeanpaulkogelman@me.com> wrote:
>
>> Would it make sense to use either fixed length strings or maybe even enums?
> No. Enums or fixed length strings just make it harder to extend, for no benefit (bandwidth of 'reject' messages doesn't matter, they will be rare and are not relayed).
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 3082 bytes --]

Yeah, something like HTTP would work well.

I'm really looking forward to this. Currently bitcoinj gets a small but
steady stream of bug reports of the form "my transaction did not
propagate". It's flaky because the library picks one peer to send the
transaction to, and then watches it propagate across the network. But if
that selected peer refuses the tx for whatever reason, that propagation
never comes, and there's currently no timeout to make it retry with a
different node. The transactions as created usually look fine, so it's not
clear to me why some nodes would accept it others wouldn't given the
absence of double spends, and there's no way to debug and find out :(




On Sat, Oct 26, 2013 at 6:32 AM, kjj <bitcoin-devel@jerviss.org> wrote:

> The HTTP status code system seems to work well enough, and seems to give
> the best of both worlds.  A 3 digit numeric code that is
> machine-readable, and a freeform text note for humans.
>
> The clever part about that system was in realizing that the numeric
> codes didn't need to account for every possible error. They just need to
> give the other node the most useful information, like "try that again
> later, I'm having a temporary problem" vs. "That is just plain wrong and
> it will still be wrong next time too, so don't bother to retry".
>
> We can leave it to the humans to puzzle out the meaning of "403: values
> of txid gives rise to dom!"
>
> Gavin wrote:
> >
> > On Oct 26, 2013, at 11:01 AM, Jean-Paul Kogelman <
> jeanpaulkogelman@me.com> wrote:
> >
> >> Would it make sense to use either fixed length strings or maybe even
> enums?
> > No. Enums or fixed length strings just make it harder to extend, for no
> benefit (bandwidth of 'reject' messages doesn't matter, they will be rare
> and are not relayed).
> >
> >
> >
> ------------------------------------------------------------------------------
> > October Webinars: Code for Performance
> > Free Intel webinars can help you accelerate application performance.
> > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> > the latest Intel processors and coprocessors. See abstracts and register
> >
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Bitcoin-development mailing list
> > Bitcoin-development@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 4344 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Luke-Jr]

On Sunday, October 27, 2013 2:32:57 PM Mike Hearn wrote:
> Currently bitcoinj gets a small but steady stream of bug reports of the form
> "my transaction did not propagate". It's flaky because the library picks one
> peer to send the transaction to, and then watches it propagate across the
> network. But if that selected peer refuses the tx for whatever reason, that
> propagation never comes, and there's currently no timeout to make it retry
> with a different node.

Sounds like the real bug is "BitcoinJ relies on good/servant behaviour from 
other nodes". Don't assume your random node isn't hostile. Handling a peer 
that doesn't relay your transaction for any reason (including if they lie to 
you about having done so) should be expected behaviour.

Luke

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1402 bytes --]

These nodes are much more likely to just be broken than malicious, but
without any way to diagnose why they are dropping a transaction it's hard
to find out what's really going on.

Anyway, yes, I need to spend time adding timeouts and all kinds of other
things, although of course if the transactions are being rejected due to a
change in network rules that won't help either - if the nodes you're
connected to are silently eating your transaction, there's no sane UI that
can result from that without more explicit error handling.


On Sun, Oct 27, 2013 at 3:39 PM, Luke-Jr <luke@dashjr.org> wrote:

> On Sunday, October 27, 2013 2:32:57 PM Mike Hearn wrote:
> > Currently bitcoinj gets a small but steady stream of bug reports of the
> form
> > "my transaction did not propagate". It's flaky because the library picks
> one
> > peer to send the transaction to, and then watches it propagate across the
> > network. But if that selected peer refuses the tx for whatever reason,
> that
> > propagation never comes, and there's currently no timeout to make it
> retry
> > with a different node.
>
> Sounds like the real bug is "BitcoinJ relies on good/servant behaviour from
> other nodes". Don't assume your random node isn't hostile. Handling a peer
> that doesn't relay your transaction for any reason (including if they lie
> to
> you about having done so) should be expected behaviour.
>
> Luke
>

[-- Attachment #2: Type: text/html, Size: 1914 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Gavin Andresen]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

RE: use HTTP-like status codes:

Okey dokey, I'll add a one-byte machine-readable HTTP-like status code.
Unless y'all want a 32-bit status code.  Or maybe a varint. Or a
three-character numeric string. I really and truly don't care, but I am
writing this code right now so whatever you want, decide quickly.

If anybody has strong feelings about what the reject categories should be,
then please take the time to write a specific list, I can't read your
mind....


-- 
--
Gavin Andresen

[-- Attachment #2: Type: text/html, Size: 625 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[kjj]

[-- Attachment #1: Type: text/plain, Size: 1670 bytes --]

Any reason not to use actual HTTP codes?  I'm not aware of any major 
deficiency in them.  Most of them won't apply to us, which is fine, they 
don't seem to apply to HTTP either.  We can extend the scheme on our own 
if we find a good reason to.

That implies 16 bits, or a varint.  I would avoid a string or varstring 
here; we already have a text field.  Varint vs. 16 bits is a minor 
issue, and arguments can be made in both directions.  I flipped a coin 
and got heads, so I'll say varint.

Gavin Andresen wrote:
> RE: use HTTP-like status codes:
>
> Okey dokey, I'll add a one-byte machine-readable HTTP-like status 
> code. Unless y'all want a 32-bit status code.  Or maybe a varint. Or a 
> three-character numeric string. I really and truly don't care, but I 
> am writing this code right now so whatever you want, decide quickly.
>
> If anybody has strong feelings about what the reject categories should 
> be, then please take the time to write a specific list, I can't read 
> your mind....
>
>
> -- 
> --
> Gavin Andresen
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #2: Type: text/html, Size: 2928 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Luke-Jr]

On Sunday, October 27, 2013 10:52:25 PM Gavin Andresen wrote:
> If anybody has strong feelings about what the reject categories should be,
> then please take the time to write a specific list, I can't read your
> mind....

It might make sense to use the rejection reasons from BIP 22 where applicable.

https://en.bitcoin.it/wiki/BIP_0022#Appendix:_Example_Rejection_Reasons

Luke

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Pieter Wuille]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

Categories that make sense to me:
1) protocol related problems
1.a) failed to deserialize transaction
2) core principle violations
2.a) script evaluation fail (only owner is allowed to spend)
2.b) outputs larger than inputs (no creation of new money)
2.c) outputs not found/already spent (no double spending)
3) policy rules
3.a) not standard
3.b) ...

-- 
Pieter
 On Oct 27, 2013 11:54 PM, "Gavin Andresen" <gavinandresen@gmail.com> wrote:

> RE: use HTTP-like status codes:
>
> Okey dokey, I'll add a one-byte machine-readable HTTP-like status code.
> Unless y'all want a 32-bit status code.  Or maybe a varint. Or a
> three-character numeric string. I really and truly don't care, but I am
> writing this code right now so whatever you want, decide quickly.
>
> If anybody has strong feelings about what the reject categories should be,
> then please take the time to write a specific list, I can't read your
> mind....
>
>
> --
> --
> Gavin Andresen
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 2330 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Andreas Schildbach]

HTTP also defines success codes (2xx). Are we also talking about ACK
messages now, rather than just REJECT messages?


On 10/28/2013 03:52 AM, kjj wrote:
> Any reason not to use actual HTTP codes?  I'm not aware of any major
> deficiency in them.  Most of them won't apply to us, which is fine, they
> don't seem to apply to HTTP either.  We can extend the scheme on our own
> if we find a good reason to.
> 
> That implies 16 bits, or a varint.  I would avoid a string or varstring
> here; we already have a text field.  Varint vs. 16 bits is a minor
> issue, and arguments can be made in both directions.  I flipped a coin
> and got heads, so I'll say varint.
> 
> Gavin Andresen wrote:
>> RE: use HTTP-like status codes:
>>
>> Okey dokey, I'll add a one-byte machine-readable HTTP-like status
>> code. Unless y'all want a 32-bit status code.  Or maybe a varint. Or a
>> three-character numeric string. I really and truly don't care, but I
>> am writing this code right now so whatever you want, decide quickly.
>>
>> If anybody has strong feelings about what the reject categories should
>> be, then please take the time to write a specific list, I can't read
>> your mind....
>>
>>
>> -- 
>> --
>> Gavin Andresen
>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
>> the latest Intel processors and coprocessors. See abstracts and register >
>> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
>>
>>
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 
> 
> 
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> 
> 
> 
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Gregory Maxwell]

On Mon, Oct 28, 2013 at 2:26 AM, Andreas Schildbach
<andreas@schildbach.de> wrote:
> HTTP also defines success codes (2xx). Are we also talking about ACK
> messages now, rather than just REJECT messages?

I do not believe we should do that:  It would be a non-trivial
increase the protocol bandwidth requirements.

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Gavin Andresen]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

Thanks for the feedback, everybody, gist updated:
  https://gist.github.com/gavinandresen/7079034

Categories are:

0x01-0x0fProtocol syntax errors0x10-0x1fProtocol semantic errors0x40-0x4fServer
policy rule
<https://gist.github.com/gavinandresen/7079034#rejection-codes-common-to-all-message-types>

RE: why not a varint:  because we're never ever going to run out of reject
codes.  Eight are defined right now, if we ever defined eight more I'd be
surprised.

RE: why not use HTTP codes directly: because we'd be fitting round pegs
into square holes.

-- 
--
Gavin Andresen

[-- Attachment #2: Type: text/html, Size: 2240 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Warren Togami Jr.]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

How about rejection codes to notify you that you have been rate limited?

Warren


On Mon, Oct 28, 2013 at 7:37 PM, Gavin Andresen <gavinandresen@gmail.com>wrote:

>
> Thanks for the feedback, everybody, gist updated:
>   https://gist.github.com/gavinandresen/7079034
>
> Categories are:
>
> 0x01-0x0f Protocol syntax errors0x10-0x1f Protocol semantic errors0x40-0x4fServer
> policy rule
> <https://gist.github.com/gavinandresen/7079034#rejection-codes-common-to-all-message-types>
>
> RE: why not a varint:  because we're never ever going to run out of reject
> codes.  Eight are defined right now, if we ever defined eight more I'd be
> surprised.
>
> RE: why not use HTTP codes directly: because we'd be fitting round pegs
> into square holes.
>
> --
> --
> Gavin Andresen
>
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 3729 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

On Mon, Oct 28, 2013 at 10:55:59PM -1000, Warren Togami Jr. wrote:
> How about rejection codes to notify you that you have been rate limited?

ACK

However note that for the rejection messages defined these are actually
covered by the "too-low-fees" rejection codes. What would would want a
rate limiting rejection code is things like getblock and other requests.

-- 
'peter'[:-1]@petertodd.org
00000000aefda5391d2a12987ee8dc048c046c8f3e1ad1f1a3a1dbbe4954bfaf

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 685 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 2097 bytes --]

For tx reject, should there be a code for "unknown version"? That is,
tx.nVersion > bestKnownVersion == reject? In that case 0x40 would become
"non-standard transaction type". I think "unknown transaction type" is a
bit vague. Or do we want new tx messages to always be backwards compatible?

0x42 and 0x43 seems a bit similar to me. The sender knows what fee was paid
(presumably). If free transactions and fee-paying transactions end up
having a unified ranking applied, then distinguishing between them in the
reject message won't make much sense.

For block 0x11 again shall there be a separate code for "block is from the
future"? We don't want to lose the nVersion field to people just using it
for nonsense, so does it make sense to reject blocks that claim to be v2 or
v3?




On Tue, Oct 29, 2013 at 6:37 AM, Gavin Andresen <gavinandresen@gmail.com>wrote:

>
> Thanks for the feedback, everybody, gist updated:
>   https://gist.github.com/gavinandresen/7079034
>
> Categories are:
>
> 0x01-0x0f Protocol syntax errors0x10-0x1f Protocol semantic errors0x40-0x4fServer
> policy rule
> <https://gist.github.com/gavinandresen/7079034#rejection-codes-common-to-all-message-types>
>
> RE: why not a varint:  because we're never ever going to run out of reject
> codes.  Eight are defined right now, if we ever defined eight more I'd be
> surprised.
>
> RE: why not use HTTP codes directly: because we'd be fitting round pegs
> into square holes.
>
> --
> --
> Gavin Andresen
>
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 4540 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

On Tue, Oct 29, 2013 at 10:52:31AM +0100, Mike Hearn wrote:
> For block 0x11 again shall there be a separate code for "block is from the
> future"? We don't want to lose the nVersion field to people just using it
> for nonsense, so does it make sense to reject blocks that claim to be v2 or
> v3?

That would prevent us from using nVersion as a soft-forking mechanism.

-- 
'peter'[:-1]@petertodd.org
000000000000000908fddb47210344de50e6d3bd842e649c68853eeee0390dcd

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 685 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



Peter Todd <pete@petertodd.org> wrote:
>On Tue, Oct 29, 2013 at 10:52:31AM +0100, Mike Hearn wrote:
>> For block 0x11 again shall there be a separate code for "block is
>from the
>> future"? We don't want to lose the nVersion field to people just
>using it
>> for nonsense, so does it make sense to reject blocks that claim to be
>v2 or
>> v3?
>
>That would prevent us from using nVersion as a soft-forking mechanism.

Actually, that statement didn't go far enough: rejecting blocks with nVersions that you don't expect is a hard fork.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.0.9

iQFQBAEBCAA6BQJSb544MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhfuGCADHB+5WZ3oSRCCYgId+
5c4rxZHjjmXXIVOlXySjoRQ20JUnGbkUqN057VlutYbWaGV7OqR0oQyzh0LGpMdL
BU9hg8XoHbyIvA0WhCfEJvFzkwseN8Ac77UxtV3leBpBkSzjqlMS9QBGU6L5rw2U
uo8Sd7bQaqkadOPode3MMWDtmmqAZaj2dN02w/8C1rRna3SrbYRVYbaVAuN9yREO
99DOGEM2V7ni+eo4sQoxP2jf8vmNzy1EuQH8v1OloPgcpxl/GkLVXzQh4ZfO1ApE
UVKBo93oT34Tce9LwZy+k8XpeCvBRJ/+QwsbAAgdVYKr8KmRcAW4oR2KN7Y0jjq4
44xU
=OaON
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Yes, exactly. That's the point. As you well know I think the whole
soft-fork mechanism is wrong and should not be used. If the rules change,
your node is *supposed* to end up on a chain fork and trigger an alert to
you, that's pretty much the whole purpose of Bitcoin's design. Undermining
that security model is problematic.


On Tue, Oct 29, 2013 at 12:38 PM, Peter Todd <pete@petertodd.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
>
>
> Peter Todd <pete@petertodd.org> wrote:
> >On Tue, Oct 29, 2013 at 10:52:31AM +0100, Mike Hearn wrote:
> >> For block 0x11 again shall there be a separate code for "block is
> >from the
> >> future"? We don't want to lose the nVersion field to people just
> >using it
> >> for nonsense, so does it make sense to reject blocks that claim to be
> >v2 or
> >> v3?
> >
> >That would prevent us from using nVersion as a soft-forking mechanism.
>
> Actually, that statement didn't go far enough: rejecting blocks with
> nVersions that you don't expect is a hard fork.
> -----BEGIN PGP SIGNATURE-----
> Version: APG v1.0.9
>
> iQFQBAEBCAA6BQJSb544MxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
> cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhfuGCADHB+5WZ3oSRCCYgId+
> 5c4rxZHjjmXXIVOlXySjoRQ20JUnGbkUqN057VlutYbWaGV7OqR0oQyzh0LGpMdL
> BU9hg8XoHbyIvA0WhCfEJvFzkwseN8Ac77UxtV3leBpBkSzjqlMS9QBGU6L5rw2U
> uo8Sd7bQaqkadOPode3MMWDtmmqAZaj2dN02w/8C1rRna3SrbYRVYbaVAuN9yREO
> 99DOGEM2V7ni+eo4sQoxP2jf8vmNzy1EuQH8v1OloPgcpxl/GkLVXzQh4ZfO1ApE
> UVKBo93oT34Tce9LwZy+k8XpeCvBRJ/+QwsbAAgdVYKr8KmRcAW4oR2KN7Y0jjq4
> 44xU
> =OaON
> -----END PGP SIGNATURE-----
>
>

[-- Attachment #2: Type: text/html, Size: 2139 bytes --]

Re: [Bitcoin-development] On soft-forks and hard-forks

[Peter Todd]

[-- Attachment #1: Type: text/plain, Size: 4311 bytes --]

> On Tue, Oct 29, 2013 at 12:38 PM, Peter Todd <pete@petertodd.org> wrote:
> > Peter Todd <pete@petertodd.org> wrote:
> > >On Tue, Oct 29, 2013 at 10:52:31AM +0100, Mike Hearn wrote:
> > >> For block 0x11 again shall there be a separate code for "block is
> > >from the
> > >> future"? We don't want to lose the nVersion field to people just
> > >using it
> > >> for nonsense, so does it make sense to reject blocks that claim to be
> > >v2 or
> > >> v3?
> > >
> > >That would prevent us from using nVersion as a soft-forking mechanism.
> >
> > Actually, that statement didn't go far enough: rejecting blocks with
> > nVersions that you don't expect is a hard fork.
>
> Yes, exactly. That's the point. As you well know I think the whole
> soft-fork mechanism is wrong and should not be used. If the rules change,
> your node is *supposed* to end up on a chain fork and trigger an alert to
> you, that's pretty much the whole purpose of Bitcoin's design. Undermining
> that security model is problematic.

That's a nice sentiment, but there's a lot more nuance to it than
"soft-forks are bad"

We're talking about rejection here: you don't want to end up on an
isolated chain fork wondering if maybe miners have been unlucky. You
want to know that a longer chain exists so as to have solid evidence
that you're local configuration isn't what miners are mining.  Thus not
only should you "accept" blocks with versions you don't know about, you
should relay those blocks as well so that other out-of-date nodes have
the highest possible chance of finding out about them. Creating a block
is expensive, so with some minor safeguards - a high minimum difficulty,
and maximum size - relaying blocks you consider invalid is perfectly
safe and doesn't enable DoS attacks. Relaying block headers has similar
logic, and even less DoS attack worry. (don't apply bloom filters to
invalid blocks though!)

I had this discussion with Warren the other day actually: Litecoin is
considering banning old node versions and rejecting their attempts to
connect. I pointed out how you wanted to be damn sure there was no-one
mining with them, least you wind up creating a slowly growing fork mined
by nodes unaware of the main chain.

Soft-forks and SPV nodes is another topic. SPV nodes don't do any
meaningful validation - they usually don't even have the transaction
inputs spent by a transaction to determine if a scriptSig is valid.
Their security is already dependent on miners, so allowing those miners
to upgrade does no harm. In addition there are even cases where what
would be a hard-fork for a full node, is a soft-fork for a SPV node. On
the other hand if your "SPV" node is more sophisticated, then by all
means use a nVersion change to trigger an alert to the user. If you're
implementation relays blockchain data, continue doing so to ensure other
nodes find out about the new version as soon as possible. (all SPV nodes
should relay block headers if possible)


Note how the nVersion field is useful for voting: the "chain height in
coinbase" soft-fork was accomplished this way, changing nVersion from 1
to 2 with full enforcement of the rule triggered by a 95% supermajority.
Bitcoin is a decentralized system, so any changes need to be done by
voting to show that a clear consensus of hashing power will enforce and
validate the new rules. (time and height deadlines can be disasters if
the upgrade is ever ignored or delayed)

Interestingly this suggests that what we actually want is two nVersions
per upgrade: the first to signal that nodes wish to upgrade, and are
showing their intent to use the new rules. The second to signal that the
upgrade has actually happened and the old rules are now ignored. Client
software can use this two stage approach to know when rules may have
changed, and the user probably should consider upgrading. As applied to
the chain height upgrade we would have gone from version 2 during the
voting, to version 3 for any block produced while the rules were in
effect. Put another way, the last in nVersion is simply to signify that
the new blockchain rules are now active, as opposed to being proposed.

-- 
'peter'[:-1]@petertodd.org
000000000000000180dabf823b09a30b4f2032b5cab7ba1d0351cab350bee91f

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Gavin Andresen]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

On Tue, Oct 29, 2013 at 10:32 PM, Mike Hearn <mike@plan99.net> wrote:

> Yes, exactly. That's the point. As you well know I think the whole
> soft-fork mechanism is wrong and should not be used. If the rules change,
> your node is *supposed* to end up on a chain fork and trigger an alert to
> you, that's pretty much the whole purpose of Bitcoin's design. Undermining
> that security model is problematic.
>

But if you are getting soft-forked recent versions of the reference
implementation WILL alert you; see this code in main.cpp:

        if (nUpgraded > 100/2)
            strMiscWarning = _("Warning: This version is obsolete, upgrade
required!");

That is, if more than half of the last 100 blocks are up-version, warn.
 block.version is part of the block header, so SPV clients can (and
probably should) do the same.

There are also warnings if you are forked, and, most recently, warnings if
there is a high-work alternative fork.

-- 
--
Gavin Andresen

[-- Attachment #2: Type: text/html, Size: 1493 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

> But if you are getting soft-forked recent versions of the reference
> implementation WILL alert you; see this code in main.cpp:
>

Perhaps I'm confused about how we're using the term soft fork. My
understanding is that this is where a new upgrade is designed to look valid
to old nodes, and if you don't upgrade you rely on the miner majority to
get you "back on track". For instance, P2SH was done this way - old nodes
that didn't upgrade during that transition believed all spends of P2SH
outputs were valid, even those spending someone elses coins.

In this case, the code you cite won't do anything because your client will
never reject a block during a soft-forking upgrade, even if it does
something that's supposed to be invalid or nonsensical.

If a new block version changes the serialization format or script language
or SIGHASH rules such that old clients reject the block, then they will end
up on a hard fork and the alerting code will trigger, which is correct and
as it should be.

[-- Attachment #2: Type: text/html, Size: 1403 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mark Friedenbach]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

If I understand the code correctly, it's not about rejecting blocks.
It's about noticing that >50% of recent blocks are declaring a version
number that is meaningless to you. Chances are, there's been a soft
fork and you should upgrade.

On 10/30/13 1:24 AM, Mike Hearn wrote:
> 
> But if you are getting soft-forked recent versions of the
> reference implementation WILL alert you; see this code in
> main.cpp:
> 
> 
> Perhaps I'm confused about how we're using the term soft fork. My 
> understanding is that this is where a new upgrade is designed to
> look valid to old nodes, and if you don't upgrade you rely on the
> miner majority to get you "back on track". For instance, P2SH was
> done this way - old nodes that didn't upgrade during that
> transition believed all spends of P2SH outputs were valid, even
> those spending someone elses coins.
> 
> In this case, the code you cite won't do anything because your
> client will never reject a block during a soft-forking upgrade,
> even if it does something that's supposed to be invalid or
> nonsensical.
> 
> If a new block version changes the serialization format or script 
> language or SIGHASH rules such that old clients reject the block,
> then they will end up on a hard fork and the alerting code will
> trigger, which is correct and as it should be.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJScMvBAAoJEAdzVfsmodw4I5sQAL0Wq4e7b6/KX2zl9RmtFl5S
qM9ZJkJV8qzirj1hMgVwvnuOj147Vb3EkJujxeMa8ujepXKZ215mMCLnCHPzWPpJ
jTtOBn1FCkCUbyt9uSbZ+56NR+ElmSOsDpAJ8IR9VywHriCxm4OIAMCLTR6CKRfr
6NWySvjEMlsSHyr7DFaJTlMqo+KIUWGmP7tdPu1L2AvNE+613dI5q76IjUHYoxhu
2dDtanYUvFCsdLZEnjTr1N45BBf1mTSlPfmA1ZWHgM779h6VIyb0TeO+iCaxpvWp
2RpSDj3+diFdMUK2uu69ZcwkREH0/RoQLOys6U5DfaGkpPtjY0YXB5DwN9quKgzX
padWzbQ0flpwWLYOPYrWATz4sWflxZJu6wHAcUkRS5k9crOLVjritXs1205x7YET
0H9jtbqXmBRXidCP2BOZPdq0PGDF8g2VeEHR69JRe3F3dBfSvbgHfKoiF1jpLLqb
rttoP+nD4ZRX8FesV2E/DEZgDZJMd8eqDKNDjq7Db4BTDg24Nq2ATNE2fBtenXwI
nXVNdmnvjDxjF0weJGlYgaQTfgVwHRxs+j4qgY4VLM0qEYplhHgg+KmOMFUtxAF/
sZv6w56XtCZS3LdNONAJSZzXIcqgmcodiWKVxkTL29dsWKikcBL5cG9ipdfmjQKT
eccFOHArsbW3eSfKP/Mb
=FSQI
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

On Wed, Oct 30, 2013 at 10:05 AM, Mark Friedenbach <mark@monetize.io> wrote:

> If I understand the code correctly, it's not about rejecting blocks.
>

I was referring to the fork alerts that Matt did. They also alert you if
there's a missed upgrade.

[-- Attachment #2: Type: text/html, Size: 571 bytes --]

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Gregory Maxwell]

On Sun, Oct 27, 2013 at 7:32 AM, Mike Hearn <mike@plan99.net> wrote:
> I'm really looking forward to this. Currently bitcoinj gets a small but
> steady stream of bug reports of the form "my transaction did not propagate".
> It's flaky because the library picks one peer to send the transaction to,
> and then watches it propagate across the network. But if that selected peer
> refuses the tx for whatever reason, that propagation never comes, and

Actually, we'll probably need to explicitly document that a failure to
reject is by no means a promise to forward.

If a node is using priority queued rate limiting for its relaying then
it might "accept" a transaction from you, but have it fall out of its
memory pool (due to higher priority txn arriving, or getting
restarted, etc.) before it ever gets a chance to send it on to any
other peers.

Finding out that it rejected is still useful information, but even
assuming all nodes are honest and well behaved I don't think you could
count on its absence to be sure of forwarding.

Re: [Bitcoin-development] Feedback requested: "reject" p2p message

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

On Wed, Oct 30, 2013 at 6:13 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:

> If a node is using priority queued rate limiting for its relaying then
> it might "accept" a transaction from you, but have it fall out of its
> memory pool (due to higher priority txn arriving, or getting
> restarted, etc.) before it ever gets a chance to send it on to any
> other peers.
>

That's a good point, however, I would hope that this fairly trivial race
condition can be resolved. There's no requirement that a transaction be
placed into a buffer from which it can be removed before relaying. After
relaying - sure. But the gap of a few seconds between that shouldn't cause
any issues to eliminate.

I believe Gavin's smartfees branch adds mempool persistence to disk, so
restarting nodes won't clear the mempool in future. Or at least that's a
part of the longer term plan once mempool limiting is done.


> Finding out that it rejected is still useful information, but even
> assuming all nodes are honest and well behaved I don't think you could
> count on its absence to be sure of forwarding.
>

I think measuring propagation will be a part of bitcoin wallets for the
forseeable future, although if all nodes reject that allows for a more
responsive and more helpful UI than just waiting for some arbitrary timeout
to elapse.

[-- Attachment #2: Type: text/html, Size: 1946 bytes --]