From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Sun, 27 Apr 2025 11:32:53 -0700 Received: from mail-oo1-f55.google.com ([209.85.161.55]) by mail.fairlystable.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1u96o8-00054K-6k for bitcoindev@gnusha.org; Sun, 27 Apr 2025 11:32:53 -0700 Received: by mail-oo1-f55.google.com with SMTP id 006d021491bc7-60214b7cdbcsf3314431eaf.0 for ; Sun, 27 Apr 2025 11:32:52 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1745778766; cv=pass; d=google.com; s=arc-20240605; b=kGXhcT4bBTKbRy2yLSH3B9SObSFLxYgb/nn6uQHHiIcm23zeOV8eMkypVAIz/hQCBl xgO5FhIb4Wm0XXRTFWLNOt5DT+PhrLK4Nb9Th1t44hZqBtegfbuSAbUHJctx+REgBkNP kHDVbUR2yLLJlCX2t2LxGZiuDw2DNx53N/ki0CA+TosTqSHUV6BZy3v7Z+F4+Y0AXdyF cRe8QLe02xuOJayEB5Lz35cMQYDaAO8pLqIcHSZy0LkijCp9m/w8a8TZm0MXb+b7dh99 Qe8oDi9Dt/cwwxnXFzB6/+Z6WLGHgsRrDGN3trPBN3iHXc10F9dioLcGy9mNPrHjttPp X/fg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-language:thread-index :content-transfer-encoding:mime-version:message-id:date:subject :in-reply-to:references:to:from:sender:dkim-signature; bh=E92byQMUDbvmWYU0rTXGBEU97gq+4HRc0sI1pQcSPtw=; fh=6G2DFUyjp0S7N3g3asfVnDva3ZLLr3lChMdpBUxnfy8=; b=NsyjD3ViI6qB+LS7sQXaMYcd2bkxrwhZNLzWoXLCwkUooWwDaUamPjn4voOsH+NIaT T+itM0MD9n+/mcCF5SukrPinAA6ONlzawWRfjwGxs5P3Duy0A8s82hgBpLOEdF+VN/H0 C4qg/G0Eq+iBiDSS8A5H+CBPDgvSyX50IZV+RlewuluTODxYQvgqvQY9U87oIINKLrCX B6fQ4wkfkqgsg/jKG7FOpp+e3PEITYDliEHoaaQSFCZlvBnp9rzK7qh/1+QLETO5jFz3 Ij9R/KLNGrnV0lPYpvd1w3JzvFpeeECNfqknBz00sFEmYw1YLob8TFHRj51ege3yEbVX tNBQ==; darn=gnusha.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@voskuil-org.20230601.gappssmtp.com header.s=20230601 header.b=GgN0Fjjh; spf=none (google.com: eric@voskuil.org does not designate permitted sender hosts) smtp.mailfrom=eric@voskuil.org; dara=pass header.i=@googlegroups.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1745778766; x=1746383566; darn=gnusha.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:content-language:thread-index :content-transfer-encoding:mime-version:message-id:date:subject :in-reply-to:references:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=E92byQMUDbvmWYU0rTXGBEU97gq+4HRc0sI1pQcSPtw=; b=s92zUWmbuqu2iVIfcQ6EruPO2Xg2/JuqQqNEmPrnwUfE1+n8AsgfiJ28JIYGl7IWDg Sp4eMCzLo8bubf2qUb+2Mpwzg5jlawTLbWMMRewgANrnVodtoLdRCYSU+0aS7YDVV85e sU2EwhR6r6J1ugmD4Ij+X0WbEmgFxIAvf8AEfYxXipO1vpsYXQem6sP4wOEexCXpvgXh Jsep0lf3Jj2pZ45LWcFJAX+hoEUOy+nZkIme3RhLrCTo3sBoa/O+ydMhXcGUugOwlwMm bZkdnvobBA97Bvy4RZIg8ZDRpzOIB2iZofKfaIj84eWhWPoDump+cfaB0s9uKMHQrdZP qFgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745778766; x=1746383566; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:content-language:thread-index :content-transfer-encoding:mime-version:message-id:date:subject :in-reply-to:references:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=E92byQMUDbvmWYU0rTXGBEU97gq+4HRc0sI1pQcSPtw=; b=mu35YwtCKuqXkn68H/Me8pjXp3g/z0UY+X0MWJKchmu6FC4FttuhhT5bILh8VCv8jV K8xYdR2MN/i2EIg3F4V+NUVucXX5XfZKvgxIq3veMEnfq1kJNYXWgY8bMqtCWLp+FI7P cCs9CoHwK8P0Y4VgxeO5bwPpjorjH7gYr8aVPQBqgJ+uiO3w7FhBvPZzLU1PK2cPXofu VdTgQ+rX7nQ+CFf+06npLzP6S2m9VTC0MMLeEeyHZEEvtDmi09Ubuucd9JzylQrvPXHz HNFf2EPjCx7r39Alo/EXoEhSWsquRDDxMZjMAeuFdEot5j2201JR69Znqh4pj5Obiq4L 5zOw== Sender: bitcoindev@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVnAB4RUf3vpoFjT/hl1tPNaqXS1NPu5TgFWJ3ujoGXmcvfGc6vdxxa8L69lLfTnppFgaIVb85gsrdL@gnusha.org X-Gm-Message-State: AOJu0YxConG1BZ83UFlEYhpNGQ0qKBKeF6DP1F5PK9vCcBMS1963zYBb e47Keptas9w3FKZhPBp47h4OObCWxYRmgbVO9MjoxpdtNuBZQ2XB X-Google-Smtp-Source: AGHT+IGZG3nqiupqP8cnCNvV1kWjAKlBeE0z1FOnMHkVo/7QXzdBPZoXpHO10+lyY2vyigBrGaS4Cg== X-Received: by 2002:a4a:da42:0:b0:604:ac85:abe2 with SMTP id 006d021491bc7-60646b73dc7mr7039264eaf.3.1745778766059; Sun, 27 Apr 2025 11:32:46 -0700 (PDT) X-BeenThere: bitcoindev@googlegroups.com; h=AVT/gBH0yJfxYuC2k7LS+3eKZIt2EtPxg+jZo4KNwRAIOWA4LQ== Received: by 2002:a4a:8381:0:b0:602:a14b:beba with SMTP id 006d021491bc7-606432a6881ls936289eaf.0.-pod-prod-00-us; Sun, 27 Apr 2025 11:32:42 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWTLg0+vjRbmkITmbalCcXzZ8m77CRL2vc6KnVv+/Vxgml7IBcSKNOkbiVeheyL+VNOU3p65mk53w1W@googlegroups.com X-Received: by 2002:a05:6808:6a82:b0:3f9:176a:3958 with SMTP id 5614622812f47-401ec4a1ddcmr7499969b6e.11.1745778762805; Sun, 27 Apr 2025 11:32:42 -0700 (PDT) Received: by 2002:a05:6808:2002:b0:3fa:da36:efcd with SMTP id 5614622812f47-401f2fc0e20msb6e; Sun, 27 Apr 2025 11:30:33 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXuveBvjEeDldQIwfY4OcJFXLzYaCYJPNxRcyLKbQz/Own0BIVNPzcS0W6qBqN0pVfP+CV1Hpii5kvp@googlegroups.com X-Received: by 2002:a17:90b:2e42:b0:2ff:5267:e7da with SMTP id 98e67ed59e1d1-309f895d406mr11352378a91.3.1745778632445; Sun, 27 Apr 2025 11:30:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1745778632; cv=none; d=google.com; s=arc-20240605; b=TuNg55AOoBwPBg1YgNgSBreeCyghd5+HzHjs76rGteabBU3XN8I8QVhd4cd3K4M5Qp fNyQWNaCfmQI/djfxwNOXGIRNo9Z60jxAwdLmYjgXzVD2EWAs6lXPMu4kaSEeqIfFegD fhc7vrFo6IYKkraFcSQZT5fmmU36DmP5dY9BMiWOTvrcUkxL2SnXRzkBetNcXSW+zBbr kmCUFh9GCVZ4n3CxFv1R3yJc3/U+RauTqtEPMO/YzfyokU66hlP6mrt862ZDiBGmqHsJ 4M515+++axzxq7s5ErZGBbEpjsuI+cYIPkl+pgpqnjRUSH2BCjl87Q4QZkxPJ8hmOXOg Zo1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:to:from :dkim-signature; bh=aBakv+ag9XD73nZQUGXbhhutWhY0qM5E3pHDzcopr48=; fh=ZRvT9JXefonuUPKJRNJFGFFFTAPtD0+LJ5Rw++rU9QM=; b=bqAa+BjrNZZwUkEAJiRZQKhYbX3fJNeq501SYrSducR1sp9hnTFjfVPA/VraQJGoRv 0YzJzUOAonb6B0/huuWv4zevGEreuNUg+EpPeCkVs70Wz+AL13UQr75QY1D2DJIWbQl+ noy78lMjgX17tTxIDyZMeYhnhdJXpYg1aJC9rkMA0LHSO6Mky2ZOGezMlnSNlx8d0Y82 aIEDvcQd5okvKMCQ77nvJVEecXaUFluBCnS550SXRXoTTPwKYblN1DmE4MkaywKh29D7 fXVZhCkULbrYaDw2LZp12WmY+KiZhlCtYqN9/qrv4UHCJ5VTRXnshg0vpR16mfTyy4h3 mg8g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@voskuil-org.20230601.gappssmtp.com header.s=20230601 header.b=GgN0Fjjh; spf=none (google.com: eric@voskuil.org does not designate permitted sender hosts) smtp.mailfrom=eric@voskuil.org; dara=pass header.i=@googlegroups.com Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com. [2607:f8b0:4864:20::835]) by gmr-mx.google.com with ESMTPS id 98e67ed59e1d1-309d34b4642si717757a91.0.2025.04.27.11.30.32 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 27 Apr 2025 11:30:32 -0700 (PDT) Received-SPF: none (google.com: eric@voskuil.org does not designate permitted sender hosts) client-ip=2607:f8b0:4864:20::835; Received: by mail-qt1-x835.google.com with SMTP id d75a77b69052e-4772f48f516so58762281cf.1 for ; Sun, 27 Apr 2025 11:30:32 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCUmD62SSjKxs2sD6PZtKzcWY4uNe9DLqQLV6QCGHjqDa6na4HyYcTO/NIfVIdGYyBmKIXw7yaZX0/VA@googlegroups.com X-Gm-Gg: ASbGncugDf+qrTdQIDdo1l8ZA7zijiioEr5HKNfCsTYS0SZAhvYf2G7rg8w8IRo98ld IvkccZ2G2q2G15btioB4QZFUhV+cUkMxoNfSsw1+PApXHFbP9hAVubatlvmAaneT1VXsASLQIka 35dTRGO971kft/QlzcfIDHD0ZJt7bYmqufimmfKu9YqCwxo+7XEhWk0FEjUtHNcVyV9yGcg8n+D sT1xl+JAOJPTGtn9M48QFeIfQX61SC87j06WRGookgN3tN7tuBW0hmpP20atnS7DAxiOCqrshdy Iz5O9uf/W8Ej6nZLTOHI3QgeAu7mxCRr2TidXQwQKmyGIfnXpeDFc8fYOP4TgzzYGdforr7oX/9 XpSxVXQ== X-Received: by 2002:ac8:5a44:0:b0:471:fef5:ee84 with SMTP id d75a77b69052e-4802db97fd8mr147366781cf.7.1745778631178; Sun, 27 Apr 2025 11:30:31 -0700 (PDT) Received: from ERICDESKTOP (c-73-227-67-43.hsd1.nh.comcast.net. [73.227.67.43]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-47e9f5b4f0asm52925141cf.33.2025.04.27.11.30.30 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 27 Apr 2025 11:30:30 -0700 (PDT) From: To: "'Ruben Somsen'" , References: In-Reply-To: Subject: RE: [bitcoindev] The Tragic Tale of BIP30 Date: Sun, 27 Apr 2025 14:30:29 -0400 Message-ID: <002201dbb7a2$74676640$5d3632c0$@voskuil.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQHjQEGI0hVWVnHItjONPNIY3TwlQrOouk9w Content-Language: en-us X-Original-Sender: eric@voskuil.org X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@voskuil-org.20230601.gappssmtp.com header.s=20230601 header.b=GgN0Fjjh; spf=none (google.com: eric@voskuil.org does not designate permitted sender hosts) smtp.mailfrom=eric@voskuil.org; dara=pass header.i=@googlegroups.com Precedence: list Mailing-list: list bitcoindev@googlegroups.com; contact bitcoindev+owners@googlegroups.com List-ID: X-Google-Group-Id: 786775582512 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Score: -0.8 (/) > The problem occurs when we reorg back to a point between block 91880 and > 91722. When we rewind the blockchain, previously created outputs get > removed from the UTXO set again.=20 Consider that a UTXO set (accumulator) is an implementation detail. One of = its underlying assumptions is that a given txid cannot repeat in confirmed = blocks. This assumption did not hold, and the bip30 workaround for the two = exceptions failed to consider the effect of the above reorg given a UTXO se= t design. Instead of removing the second instance, it removes all instances= . > we could fix the bug by no longer removing the coinbase transaction in ca= se of a reorg of block 91880 and 91842. IMO this would be the most reasonable resolution, as it would produce the o= utcome that in fact exists independent of the UTXO set. The previous blocks= still actually have the first instance of each duplicated coinbase. This is also inconsequential from a performance standpoint. It only affects= reorganization (infrequent), and only proceeds for these two specific bloc= ks (rare). Furthermore these two specific blocks are already exceptions and= the implementation is trivial. > However, it seems this never occurred. Correct, only the exception coinbases have been duplicated in the current s= trong chain. > Aside from checking for coinbase uniqueness, we could also check that the > coinbase will not conflict with any future coinbases (i.e. not conflict w= ith > BIP34 as well as the Consensus Cleanup BIP). The relationship between BIP34 and BIP30 is also a bit sordid, but the pres= umption is that the Consensus Cleanup would resolve the existing flaw in BI= P34, and that the combination would effectively obsolete BIP30. Under the a= ssumption that this does in fact produce the intended outcome - that BIP34 = (presently) Consensus Cleanup (forever) makes further coinbase duplication = impossible, BIP30 can remain deactivated to the extent that these are activ= e. Nothing additional is required to avoid the presumably inefficient BIP30= checks. > Once we fully [remove the checkpoints][3], the bug becomes theoretically = (not practically) exploitable. I would never advocate for adding more, but I'm not aware of any compelling= argument to hard fork out the existing checkpoints. The top checkpoint is = consensus for over 11 years and materially reduces the validation cost of 2= 95,000 blocks. > Doing this until block 227931 results in a modest ~7MB cache. However, > BIP30 might not deactivate, in which case we'd have an ever-growing cache= . > This is solvable as follows.... This is a consequence of the presumed removal of checkpoints above BIP34 ac= tivation height. IOW, removing the checkpoints makes it necessary to valida= te BIP30 until BIP34 activates (block 227,931). The obvious solution to thi= s problem is to not create the problem in the first place. e > -----Original Message----- > From: bitcoindev@googlegroups.com On > Behalf Of Ruben Somsen > Sent: Sunday, April 27, 2025 12:45 PM > To: bitcoindev@googlegroups.com > Subject: [bitcoindev] The Tragic Tale of BIP30 >=20 > Markdown version: > https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b > 7c >=20 > ### Introduction >=20 > In my recent exploration of [SwiftSync][1], I came to the realization tha= t > [BIP30][2] has an unresolved consensus bug. It seems this bug can't be > triggered without a reorg back to 2010, so its seriousness is debatable. = We > currently have checkpoints up to 2013, preventing such a reorg. Once we f= ully > [remove the checkpoints][3], the bug becomes theoretically (not practical= ly) > exploitable. >=20 > BIP30 is also a bit of an odd duck in terms of consensus checks in that i= t > involves the entire UTXO set without being related to the spending of inp= uts. > This is inefficient, and complicates the implementation of alternative va= lidation > methods such as utreexo, SwiftSync and quite possibly ZKP systems such as > ZeroSync. It would be nice if we could sunset BIP30 altogether. >=20 > Without necessarily advocating for action (the status quo seems quite > tenable), I'd like to lay out possible solutions for both and open up the > discussion. >=20 > ### 1. Consensus bug >=20 > There are two duplicate transactions that BIP30 treats like exceptions. T= he last > duplicate was the coinbase transaction in block 91880. When this transact= ion > gets processed, the coinbase transaction in block 91722 is overwritten. T= he > other instance occurs between these two blocks (91812, 91842). >=20 > The problem occurs when we reorg back to a point between block 91880 and > 91722. When we rewind the blockchain, previously created outputs get > removed from the UTXO set again. As a result, the overwritten output > disappears from the UTXO set completely. A node that never witnessed the > reorg, however, will still have the UTXO in its set. If subsequently the = UTXO is > ever spent, this would result in a fork. >=20 > #### Solution A >=20 > We could enforce that no reorg can take place between block 91722 and > 91880 - you'd either have to reorg all of them, or none. This ensures bot= h > reorged and fresh nodes will not have the problematic outputs in their UT= XO > set. Considering this is only ~160 blocks at the low mining difficulty of= 2010, > this wouldn't be a big constraint. >=20 > #### Solution B >=20 > When discussing my findings with Sjors Provoost, he pointed out that the > removal of the checkpoints (which can be seen as a hard fork) [that is be= ing > considered][3] also presents a window of opportunity to change the pre- > checkpoint consensus rules - we could fix the bug by no longer removing t= he > coinbase transaction in case of a reorg of block 91880 and 91842. Aside f= rom > that, Sjors' observation also opens up the question whether there are oth= er > pre-2013 consensus changes we'd want to consider making. >=20 > ### 2. Sunsetting BIP30's UTXO set check >=20 > BIP30 is currently active from genesis until [BIP34][4] activates at bloc= k height > 227931 (March 2013). If this block is reorged out, BIP30 remains active > indefinitely. BIP34 has issues of its own that are being addressed in the > [Consensus Cleanup BIP][5] - you could go and read that, I won't go into = too > much detail here. >=20 > Technically, BIP30 only prevents duplicate _unspent_ outputs. It does thi= s by > checking for each output whether it's already in the UTXO set (this is th= e > inefficient part), and rejecting the block if it is. The two 2010 duplica= tes are > hard-coded in as exceptions. Under these rules, spending an output and > recreating it is allowed. However, it seems this never occurred. >=20 > One last point to address is why BIP34 gets deactivated if block 227931 i= s > reorged out. The reason for this is because otherwise it'd open the door = to > possibly creating outputs prior to BIP34's activation that will conflict = with > BIP34's rules for ensuring coinbase transaction uniqueness (the exact iss= ue the > Consensus Cleanup is seeking to resolve). >=20 > Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check compl= etely, > ensuring it's no longer required, even in case of a reorg. >=20 > #### Solution >=20 > Given that we have no duplicates, barring the two exceptions, we could > replace the inefficient BIP30 UTXO set check with a coinbase uniqueness > check. We simply cache the coinbase TXIDs and ensure there are no duplica= tes. > Doing this until block 227931 results in a modest ~7MB cache. However, > BIP30 might not deactivate, in which case we'd have an ever-growing cache= . > This is solvable as follows. >=20 > Aside from checking for coinbase uniqueness, we could also check that the > coinbase will not conflict with any future coinbases (i.e. not conflict w= ith > BIP34 as well as the Consensus Cleanup BIP). This ensures BIP34 can simpl= y > activate at block height 227931, regardless of whether there's a reorg. >=20 > ### In closing >=20 > These were some of the issues with BIP30 and possible solutions. Regardle= ss > of whether we choose to take action, this write-up will serve as a refere= nce. > Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for the > discussions prior to publishing. >=20 > -- Ruben Somsen >=20 >=20 > [1]: > https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c7813 > 3cd > [2]: https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki > [3]: https://github.com/bitcoin/bitcoin/pull/31649 > [4]: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki > [5]: https://github.com/bitcoin/bips/pull/1800 >=20 > -- > You received this message because you are subscribed to the Google Groups > "Bitcoin Development Mailing List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to bitcoindev+unsubscribe@googlegroups.com > . > To view this discussion visit > https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Yo > U3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com > oU3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com?utm_medium=3Dem > ail&utm_source=3Dfooter> . --=20 You received this message because you are subscribed to the Google Groups "= Bitcoin Development Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to bitcoindev+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/= 002201dbb7a2%2474676640%245d3632c0%24%40voskuil.org.