* [bitcoin-dev] Blockchain verification flag (BIP draft) @ 2015-12-04 8:26 Gregory Maxwell 2015-12-04 12:44 ` Jannes Faber 2015-12-04 17:34 ` Gavin Andresen 0 siblings, 2 replies; 8+ messages in thread From: Gregory Maxwell @ 2015-12-04 8:26 UTC (permalink / raw) To: Bitcoin Dev For discussion, A significant fraction of hashrate currently mines blocks without verifying them for a span of time after a new block shows up on the network for economically rational reasons. This otherwise harmful behavior can be made a beneficial to the whole network; but only if it is communicated. This BIP proposal suggests a communication channel and describes its use and the motivations for it. I wrote it in response to suggestions that Bitcoin Core add explicit support for this kind of mining, which could also implement best in class risk mitigations. I believe signaling the behavior is a necessary component for risk mitigation here. ----------------------------------------------------------------- <pre> BIP: draft-maxwell-flagverify Title: Blockchain verification flag Author: Greg Maxwell <greg@xiph.org> Status: Draft Type: Standards Track Created: 2015-12-02 </pre> ==Abstract== This BIP describes a flag that the authors of blocks can use to voluntarily signal that they have completely validated the content of their block and the blocks before it. Correct use of this signaling is not enforced internally to the network but if used it can act as a hint allowing more intelligent risk analysis. If deployed and adhered to, this mechanism turns otherwise harmful validation skipping by miners into a behavior which benefits the public. ==Summary== The version field in a Bitcoin block header is a 32-bit signed integer. The most significant bit (30) of the block version is defined to signal that the author of the block has validated the whole chain up to and including the content of the block. Conforming miners MUST NOT set this flag when they have not completely validated the prior block(s) or the content of their own block. Miners should continue to try to minimize the amount of time spent mining on a non-validated chain. Blocks which extend an invalid chain will continue to be rejected and ultimately orphaned as validation catches up. It is recommended, but not required, that miners also not set the flag on blocks created by the same device which created the block immediately prior. This will reduce the incorrect implication of independent validation when the two most recent blocks are both the product of the same, single, faulty system. The set state for the bit is defined as verified so that that un(der)maintained systems do not falsely signal validation. Non-verifying clients of the network may check this bit (e.g. checking that the version is >= 1073741824) and use it as an input to their risk modeling. It is recommended that once this BIP is widely accepted by the network that non-full-node wallets refrain from counting confirmations on blocks where the bit is not set. The authors of non-verifying clients should keep in mind that this flag is only correct with the cooperation of the block author, and even then a validating miner may still accidentally accept or produce an invalid block due to faulty hardware or software. Additionally, any miner which correctly uses this flag could stop doing so at any time, and might do so intentionally in order to increase the effectiveness of an attack. As a result of misunderstanding, misconfiguration, laziness, or other human factors some miners may falsely set the flag. Because invalid blocks are rare it may take a long time to detect misuse of the flag. As such, the accuracy of this field MUST NOT be strongly relied upon. Especially due to the non-enforceability of the flag, the user community should keep in mind that both setting the flag correctly and mining without verification (for brief periods of time) are healthy for the network. If participants are punished for following this specification they will simply lie, and its utility will be diminished. ==Motivation== Some applications of the Bitcoin system such as thin-client wallets make a strong assumption that all the authors of the blocks have faithfully verified the blockchain. Because many of these applications also take irreversible actions based on only one or two confirmations and the time between blocks is often very short, these clients are vulnerable to even small and short-duration violations of this assumption. Processing and propagation delays resulting from increased transaction load contribute to block orphaning when multiple blocks are found at close to the same time. This has caused some miners to work on extending the chain with the most proof-of-work prior to validating the latest block(s). Although this validation skipping undermines the security assumptions of thin clients, it also has a beneficial effect: these delays also make the mining process unfair and cause increased rewards for the largest miners relative to other miners, resulting in a centralization pressure. Deferring validation can reduce this pressure and improve the security of the Bitcoin system long term. This BIP seeks to mitigate the harm of breaking the thin client assumption by allowing miners to efficiently provide additional information on their level of validation. By doing so the network can take advantage of the benefits of bypassed validation with minimal collateral damage. ==Deployment== Because there is no consensus enforced behavior there is no special deployment strategy required. [BIP 9 will need to be updated.] ==Credits== Thanks goes to Jeremy Rubin for his two-phase mining suggestion which inspired this simplified proposal. ==Copyright== This document is placed in the public domain. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [bitcoin-dev] Blockchain verification flag (BIP draft) 2015-12-04 8:26 [bitcoin-dev] Blockchain verification flag (BIP draft) Gregory Maxwell @ 2015-12-04 12:44 ` Jannes Faber 2015-12-04 16:46 ` Thomas Kerin 2015-12-04 17:34 ` Gavin Andresen 1 sibling, 1 reply; 8+ messages in thread From: Jannes Faber @ 2015-12-04 12:44 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 7465 bytes --] 1) (I would assume this is already current default behaviour, but just in case.) Would it not make sense to *never* send a blockheader to an SPV client unless the node itself fully validated that block? Regardless of who mined the block and whether this verification flag has been set or not. 2) Besides having your verification flag in the block, would it not also make sense to have such a flag in the P2P protocol when blocks (or headers) are communicated? That way a node could simply do some quick sanity checks (difficulty as anti-DOS) on an incoming block and then immediately propagate it to the next (non-SPV) node, but with a flag "Looks good, but I haven't fully validated it myself, so please don't blame me". And if the block does turn out to be invalid, the node does not get banned if it was honest about it. 3) With the above implemented, I can imagine miners running 2 (or more) nodes side by side, one of them doesn't validate in order to reduce latency and orphan rates, but the other one does validate and quickly signals the first one if there's a problem. Both nodes don't necessarily need to be in the same network or even on the same side of the Great Firewall. Of course they would be whitelisting each other for trust, or the signal would need to include some sort of proof. This probably has been suggested many times already, sorry if this is a dumb idea. -- Jannes On 4 December 2015 at 09:26, Gregory Maxwell via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > For discussion, > > A significant fraction of hashrate currently mines blocks without > verifying them for a span of time after a new block shows up on the > network for economically rational reasons. This otherwise harmful > behavior can be made a beneficial to the whole network; but only if it > is communicated. > > This BIP proposal suggests a communication channel and describes its > use and the motivations for it. I wrote it in response to suggestions > that Bitcoin Core add explicit support for this kind of mining, which > could also implement best in class risk mitigations. I believe > signaling the behavior is a necessary component for risk mitigation > here. > > ----------------------------------------------------------------- > > <pre> > BIP: draft-maxwell-flagverify > Title: Blockchain verification flag > Author: Greg Maxwell <greg@xiph.org> > Status: Draft > Type: Standards Track > Created: 2015-12-02 > </pre> > > ==Abstract== > > This BIP describes a flag that the authors of blocks can use to voluntarily > signal that they have completely validated the content of their > block and the blocks before it. > > Correct use of this signaling is not enforced internally to the network > but if used it can act as a hint allowing more intelligent risk analysis. > > If deployed and adhered to, this mechanism turns otherwise harmful > validation skipping by miners into a behavior which benefits the public. > > ==Summary== > > The version field in a Bitcoin block header is a 32-bit signed integer. > > The most significant bit (30) of the block version is defined to signal > that > the author of the block has validated the whole chain up to and including > the > content of the block. > > Conforming miners MUST NOT set this flag when they have not completely > validated the prior block(s) or the content of their own block. Miners > should continue to try to minimize the amount of time spent mining > on a non-validated chain. Blocks which extend an invalid chain will > continue to be rejected and ultimately orphaned as validation catches up. > > It is recommended, but not required, that miners also not set the flag on > blocks > created by the same device which created the block immediately prior. This > will reduce the incorrect implication of independent validation when the > two > most recent blocks are both the product of the same, single, faulty system. > > The set state for the bit is defined as verified so that that > un(der)maintained systems do not falsely signal validation. > > Non-verifying clients of the network may check this bit (e.g. checking > that the version is >= 1073741824) and use it as an input to their risk > modeling. It is recommended that once this BIP is widely accepted by the > network that non-full-node wallets refrain from counting confirmations on > blocks where the bit is not set. > > The authors of non-verifying clients should keep in mind that this flag > is only correct with the cooperation of the block author, and even then > a validating miner may still accidentally accept or produce an invalid > block due to faulty hardware or software. Additionally, any miner which > correctly uses this flag could stop doing so at any time, and might > do so intentionally in order to increase the effectiveness of an attack. > As a result of misunderstanding, misconfiguration, laziness, or other > human factors some miners may falsely set the flag. Because invalid > blocks are rare it may take a long time to detect misuse of the flag. > > As such, the accuracy of this field MUST NOT be strongly relied upon. > > Especially due to the non-enforceability of the flag, the user community > should keep in mind that both setting the flag correctly and mining > without verification (for brief periods of time) are healthy for the > network. If participants are punished for following this specification > they will simply lie, and its utility will be diminished. > > ==Motivation== > > Some applications of the Bitcoin system such as thin-client wallets make > a strong assumption that all the authors of the blocks have faithfully > verified the blockchain. Because many of these applications also take > irreversible actions based on only one or two confirmations and the time > between blocks is often very short, these clients are vulnerable to > even small and short-duration violations of this assumption. > > Processing and propagation delays resulting from increased transaction > load contribute to block orphaning when multiple blocks are found at > close to the same time. This has caused some miners to work on extending > the chain with the most proof-of-work prior to validating the latest > block(s). > > Although this validation skipping undermines the security assumptions > of thin clients, it also has a beneficial effect: these delays also > make the mining process unfair and cause increased rewards for the > largest miners relative to other miners, resulting in a centralization > pressure. Deferring validation can reduce this pressure and improve > the security of the Bitcoin system long term. > > This BIP seeks to mitigate the harm of breaking the thin client > assumption by allowing miners to efficiently provide additional > information on their level of validation. By doing so the > network can take advantage of the benefits of bypassed > validation with minimal collateral damage. > > ==Deployment== > > Because there is no consensus enforced behavior there is no special > deployment strategy required. [BIP 9 will need to be updated.] > > ==Credits== > > Thanks goes to Jeremy Rubin for his two-phase mining suggestion > which inspired this simplified proposal. > > ==Copyright== > > This document is placed in the public domain. > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > [-- Attachment #2: Type: text/html, Size: 8608 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [bitcoin-dev] Blockchain verification flag (BIP draft) 2015-12-04 12:44 ` Jannes Faber @ 2015-12-04 16:46 ` Thomas Kerin 0 siblings, 0 replies; 8+ messages in thread From: Thomas Kerin @ 2015-12-04 16:46 UTC (permalink / raw) To: bitcoin-dev 1. Not relaying can cause problems. Gossip networks operate by propagating new information (like a single new header), and refuse to relay information if it's obviously invalid. From the POV of a full node, which will normally hear about the header first, there's no point to not telling peers about this information. It's likely in the interest of SPV wallets to hear about EVERY contending chain, so they can go about their business deciding which is correct. 2. The only difference between a block and it's header is the list of transactions. There isn't anywhere else to put the flag but the header's version. Which is good, because clients usually receive headers first. 3. "Signal would need to include some sort of proof" That's not the point of this BIP. You can't prove the miner has or hasn't verified the chain. What purpose would it even serve? If clients accepted this 'proof', they might ignore blocks they should pay attention to. The BIP doesn't involve proof at all, it's just an indicator you can chose to use or ignore. On 04/12/15 12:44, Jannes Faber via bitcoin-dev wrote: > nodes side by side, one of them doesn't validate in order to reduce latency ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [bitcoin-dev] Blockchain verification flag (BIP draft) 2015-12-04 8:26 [bitcoin-dev] Blockchain verification flag (BIP draft) Gregory Maxwell 2015-12-04 12:44 ` Jannes Faber @ 2015-12-04 17:34 ` Gavin Andresen 2015-12-04 22:43 ` Rusty Russell 1 sibling, 1 reply; 8+ messages in thread From: Gavin Andresen @ 2015-12-04 17:34 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 833 bytes --] Overall, good idea. Is there a write-up somewhere describing in detail the 'accidental selfish mining' problem that this mitigates? I think a link in the BIP to a fuller description of the problem and how validation-skipping makes it go away would be helpful. RE: which bit to use: the draft versionbits BIP and BIP101 use bit 30; to avoid confusion, I think it would be better to use bit 0. I agree with Jannes Faber, behavior with respect to SPV clients should be to only tell them about fully validated headers. And I also agree that immediately relaying full-proof-of-work blocks before validation (with an indication that they haven't been fully validated) is a good idea, but that discussion didn't reach consensus when I brought it up two years ago ( https://github.com/bitcoin/bitcoin/pull/3580). -- -- Gavin Andresen [-- Attachment #2: Type: text/html, Size: 1107 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [bitcoin-dev] Blockchain verification flag (BIP draft) 2015-12-04 17:34 ` Gavin Andresen @ 2015-12-04 22:43 ` Rusty Russell 2015-12-06 2:47 ` James Hilliard 2015-12-06 5:13 ` Gregory Maxwell 0 siblings, 2 replies; 8+ messages in thread From: Rusty Russell @ 2015-12-04 22:43 UTC (permalink / raw) To: Gavin Andresen, Gregory Maxwell; +Cc: Bitcoin Dev Gavin Andresen via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> writes: > Overall, good idea. > > Is there a write-up somewhere describing in detail the 'accidental selfish > mining' problem that this mitigates? I think a link in the BIP to a fuller > description of the problem and how validation-skipping makes it go away > would be helpful. > > RE: which bit to use: the draft versionbits BIP and BIP101 use bit 30; to > avoid confusion, I think it would be better to use bit 0. Yes, BIP9 need to be adjusted (setting bit 30 means BIP9 counts it as a vote against all softforks). BIP101 uses bits 0,1,2 AFAICT, so perhaps start from the other end and use bit 29? We can bikeshed that later though... > I agree with Jannes Faber, behavior with respect to SPV clients should be > to only tell them about fully validated headers. A delicate balance. If we penalize these blocks too much, it's disincentive to set the bit. Fortunately it's easy for SPV clients to decide for themselves, I think? Cheers, Rusty. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [bitcoin-dev] Blockchain verification flag (BIP draft) 2015-12-04 22:43 ` Rusty Russell @ 2015-12-06 2:47 ` James Hilliard 2015-12-06 6:26 ` Gregory Maxwell 2015-12-06 5:13 ` Gregory Maxwell 1 sibling, 1 reply; 8+ messages in thread From: James Hilliard @ 2015-12-06 2:47 UTC (permalink / raw) To: bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 1702 bytes --] I think something that anyone who isn't validating should be aware of is that cgminer(which powers the vast majority of the current mining network) doesn't allow for a pool to revert to mining on the previous block due to the way chain tracking is implemented. https://github.com/ckolivas/cgminer/blob/master/cgminer.c#L4727 On Fri, Dec 4, 2015 at 4:43 PM, Rusty Russell via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Gavin Andresen via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> > writes: > > Overall, good idea. > > > > Is there a write-up somewhere describing in detail the 'accidental > selfish > > mining' problem that this mitigates? I think a link in the BIP to a > fuller > > description of the problem and how validation-skipping makes it go away > > would be helpful. > > > > RE: which bit to use: the draft versionbits BIP and BIP101 use bit 30; > to > > avoid confusion, I think it would be better to use bit 0. > > Yes, BIP9 need to be adjusted (setting bit 30 means BIP9 counts it as a > vote against all softforks). BIP101 uses bits 0,1,2 AFAICT, so perhaps > start from the other end and use bit 29? We can bikeshed that later > though... > > > I agree with Jannes Faber, behavior with respect to SPV clients should be > > to only tell them about fully validated headers. > > A delicate balance. If we penalize these blocks too much, it's > disincentive to set the bit. Fortunately it's easy for SPV clients to > decide for themselves, I think? > > Cheers, > Rusty. > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > [-- Attachment #2: Type: text/html, Size: 2575 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [bitcoin-dev] Blockchain verification flag (BIP draft) 2015-12-06 2:47 ` James Hilliard @ 2015-12-06 6:26 ` Gregory Maxwell 0 siblings, 0 replies; 8+ messages in thread From: Gregory Maxwell @ 2015-12-06 6:26 UTC (permalink / raw) To: James Hilliard; +Cc: Bitcoin Dev On Sun, Dec 6, 2015 at 2:47 AM, James Hilliard via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote: > I think something that anyone who isn't validating should be aware of is > that cgminer(which powers the vast majority of the current mining network) > doesn't allow for a pool to revert to mining on the previous block due to > the way chain tracking is implemented. An interesting potential use for the flag suggested in this draft would be allowing non-monotone mining for non-verified blocks. I could add a recommendation for that as well. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [bitcoin-dev] Blockchain verification flag (BIP draft) 2015-12-04 22:43 ` Rusty Russell 2015-12-06 2:47 ` James Hilliard @ 2015-12-06 5:13 ` Gregory Maxwell 1 sibling, 0 replies; 8+ messages in thread From: Gregory Maxwell @ 2015-12-06 5:13 UTC (permalink / raw) To: Rusty Russell; +Cc: Bitcoin Dev On Fri, Dec 4, 2015 at 10:43 PM, Rusty Russell <rusty@rustcorp.com.au> wrote: >> I agree with Jannes Faber, behavior with respect to SPV clients should be >> to only tell them about fully validated headers. > > A delicate balance. If we penalize these blocks too much, it's > disincentive to set the bit. Fortunately it's easy for SPV clients to > decide for themselves, I think? I think this is orthogonal: You should only tell SPV clients* about blocks you've fully validated yourself. The bit in the header doesn't matter with respect to that. (Effectively, the wallet risk model gets as input the fact that they got given the block in the first place as well as the flag where the miner said they were validating things or not.) Whatever the ideal behavior is for network nodes towards lite clients; it's insanely cheap to just spin up a lot of 'nodes' that have arbitrary behavior; so it shouldn't be relied on too much; but absolutely they should be filtering to things they've verified themselves... unlike the mining case, there is no reason not to. [Specific attacks to consider: You get a broken miner to include both your payment, and some invalid transaction. Other miners extend it without verifying. To avoid the fact that nodes sensibly filter invalid blocks from their lite clients, you simply just run a lot of 'nodes' so that virtually every lite client has a connection to you] (More specifically, peers should be able to specify that they want to know about pre-validated blocks and you should be able to fetch blocks from them which haven't been validated... but no one should get fed unverified blocks by surprise.) ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-12-06 6:26 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-12-04 8:26 [bitcoin-dev] Blockchain verification flag (BIP draft) Gregory Maxwell 2015-12-04 12:44 ` Jannes Faber 2015-12-04 16:46 ` Thomas Kerin 2015-12-04 17:34 ` Gavin Andresen 2015-12-04 22:43 ` Rusty Russell 2015-12-06 2:47 ` James Hilliard 2015-12-06 6:26 ` Gregory Maxwell 2015-12-06 5:13 ` Gregory Maxwell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox