From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Sun, 27 Apr 2025 14:04:17 -0700 Received: from mail-oo1-f58.google.com ([209.85.161.58]) by mail.fairlystable.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1u99Ae-0001S4-8o for bitcoindev@gnusha.org; Sun, 27 Apr 2025 14:04:17 -0700 Received: by mail-oo1-f58.google.com with SMTP id 006d021491bc7-60221ba69d1sf923468eaf.3 for ; Sun, 27 Apr 2025 14:04:16 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1745787850; cv=pass; d=google.com; s=arc-20240605; b=kTvBX9QGEFsV+0rbTDp+3dt51TGyj82DeRdSXZ5eGHmXkSd9psdS+1cPxwn3rR5bCM tnALTEkjuShTDfjMsAfTQF7MetaZNnLy8ztK0dr2B8kX1jK3y6Otl0h1tiY3PxOLq5T5 f/aaR9y9rsQY35VveAm1paSUtzwATsfTByLe499ydROaiz+KSTrBgipIqsq53My38Ss3 ImYArD6hgkaNvHXHCjLNr7MzNynqlrqjYqf/wJmMK+3q6IHuOnF8a0drQMXW1K+t/aX7 Y+2XzBYBYvsSI66HAlMW3hlkrhKFTb+S3sqirWZQYC18hHJZic08xMCGF2kfMtYh+Xv4 reaw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-language:thread-index :mime-version:message-id:date:subject:in-reply-to:references:to:from :sender:dkim-signature; bh=TiuY8Fo81TGlPF9m4fQ0EpuIgaI6/mwS+jGEs2f8RLM=; fh=uzZcFgkWoFaEn2FQuAwLw0IIIZbO811PB0+OE3lSgKk=; b=H019e2T3WvlX5W0F5e1ORsNDBCUJszhlP4xiUbq9humbrIeDv1MRF7nUITpFxBOq2+ W51Sln6zNsuGOjbzpWgd+IqUH5xdg9H0X57TaAf9Uo4RDGeC0Yg1oqGbHzLICuAhvEwB RNl2kc9eNaFjPijjFBLXv2RTHQpXyZKoaVhyhlbJtPH6ZLP/pyOrtvxbIWSSC1R5UytS /81VoIY3fECEw56MfY10EJkLgvH4FH8BL6+Rw2xX0PbItifOWeH8hnj2cCGYWrG2x5gg eblZprmJ9Cvbj3H8/nhyEOUAn1dyo8BVOaxKM+LOe18EWvzoMHarEnzlDyP1r/xvJNPF aEvw==; darn=gnusha.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@voskuil-org.20230601.gappssmtp.com header.s=20230601 header.b=qpFPAoJn; spf=none (google.com: eric@voskuil.org does not designate permitted sender hosts) smtp.mailfrom=eric@voskuil.org; dara=pass header.i=@googlegroups.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1745787850; x=1746392650; darn=gnusha.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:content-language:thread-index:mime-version :message-id:date:subject:in-reply-to:references:to:from:sender:from :to:cc:subject:date:message-id:reply-to; bh=TiuY8Fo81TGlPF9m4fQ0EpuIgaI6/mwS+jGEs2f8RLM=; b=eomATqSKKTspcIOuBOxjnlzHRYaJOhHZ29U+8ICjUN26VCFMPbVNhbD8J49qbsRaU2 lmEkXZuoIzSaAX913AFviRZArY3v5Y/PqIZmN5AP7fLHN+YNgHj6BBHBEQfiC6lgLQ4m TQOqtDip4WSRZ6QaFBzG8ez4M/6evxhVTNyOSYhMWAc1WA/PGNu61eg6CUTI4cOaTfaQ WxbVbAHHcAFw7bZahEiDpBgZAmALZgP7lLNqEz0P0q6636zpGt16zwy3HAwO6Q8wjA2z bHpgDvd/DteimDoYqAUjkdefs5gFZ+aB+D7lDW0WkC3saoavrJ97vT2ukhpuTKCBKPii KB9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745787850; x=1746392650; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:content-language:thread-index:mime-version :message-id:date:subject:in-reply-to:references:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=TiuY8Fo81TGlPF9m4fQ0EpuIgaI6/mwS+jGEs2f8RLM=; b=s3nQAakL/7TqD4BmXCn+clzOOWK3Q7YC4hTQZaCA83FdhZwZLPxFJbKkR/3n1ssy+o 0a9phMsODSxMkhv5llazmoi5OyjPrCBah+yb2y3lnzJNbQPTbs1X+KD797eSx1SUaXll OV0hPTWciWn0Rm5bJMoVVdGgewaHgC2dCmPF2pcBz6+zpMWrQ/r5FT9WK7NEdNU2ECzs QYyb6/uQrY3wang8TPevBBH11YHvlEBHnNDMqOf+Ogduq4Eoz7vBOlBYKdk8GcOFkuqt XQgedfMISj9WnlSh7yfyZzXZDHuhUpz23Wv8Sp78LSbhPnTMj5CTdvqHjJtw8a0kTafv I3YQ== Sender: bitcoindev@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUP0ub1F9Ghy6lhT/PPrzt3SfQS7V7HcGlq83m32Qx0TDVa2OolCmjtOp3mJPIkH80X+0bf0kZREPTO@gnusha.org X-Gm-Message-State: AOJu0YyyPnCqsvB8M34ks5jGEW7CuAp4YXzp+H15Lap1UWcSO59AEQti w/WwEotSpbPG1xa5u5mg8LV09ThV4Pf5lWx4IZrdV2YdTcAoYzUn X-Google-Smtp-Source: AGHT+IH6ROewK/0ZIhQ3zfi5YOKagU/P0ytkMuMPwh510ugAXrsTC/fA9PMLMJpywkuubTdrvATwow== X-Received: by 2002:a05:6820:818a:b0:604:4846:78a with SMTP id 006d021491bc7-60652a42acbmr4343188eaf.2.1745787850165; Sun, 27 Apr 2025 14:04:10 -0700 (PDT) X-BeenThere: bitcoindev@googlegroups.com; h=AVT/gBEF2kLScR84/cjv0lJZEjtKQi3bqdxrB+odKYFIYGktFg== Received: by 2002:a4a:e7d8:0:b0:5fc:fc5a:c55b with SMTP id 006d021491bc7-606432b814dls1006873eaf.0.-pod-prod-02-us; Sun, 27 Apr 2025 14:04:06 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVmrfr3FcLAV1aNSe1fy/WczWemNmCQLMapvTNT42p1TZ+Mv8v1pLlhmxIhhVFmaV6JHVeXTnp6WbcP@googlegroups.com X-Received: by 2002:a05:6808:34f:b0:3fa:daa:dd8e with SMTP id 5614622812f47-401f28e8047mr5197826b6e.35.1745787845925; Sun, 27 Apr 2025 14:04:05 -0700 (PDT) Received: by 2002:a05:6808:2002:b0:3fa:da36:efcd with SMTP id 5614622812f47-401f2fc0e20msb6e; Sun, 27 Apr 2025 14:01:03 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXVyMJldhSAwcsCz9hhrlRNxNhqjqUorO5Un5XRqtwg0jJm6WGPj7p4yR00eB+/aQ+z57MaV6s9Ot4C@googlegroups.com X-Received: by 2002:a05:6a20:3d96:b0:1f5:83bd:6cc1 with SMTP id adf61e73a8af0-2045b448498mr14689216637.0.1745787662867; Sun, 27 Apr 2025 14:01:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1745787662; cv=none; d=google.com; s=arc-20240605; b=DUaoOfZaJiADmvx3J/t3cBtvAlG9pMmYf5m+RU2gcx/qzLYSsZ+PbCfNzTpPffFoGH ll4b8YaR5un5eGfc7uYv30kuyAuQHb43N062Xqv6wEJDFKWsssQIOdZoLvMj3XZL0rqd ko4rEFFu3fRDanheFhISQY5VlVaNTDnGWz6b4ayYPSjVI6cHCZqhGZ7b4Y6XxGHvfWTu IfJ4Djy6hyXp7s+wUW8QrRKYH/2NwBwHFDz9gugFwwIcF6/ogtkUcHq2OjzmhGKi+KIG TcSdUh7lBfL1jRlJHX4dgYxdQWLShqNwLDNdiyKw0D9GZRwIB//yBIieI7r24WiHHZDG Eb0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-language:thread-index:content-transfer-encoding :mime-version:message-id:date:subject:in-reply-to:references:to:from :dkim-signature; bh=QSEBLuG0VT9ub0viHzGLB7XXqX+Rbb47EjZunNVTtnY=; fh=kpeheXYBoZ8+TLcINRb5fzDTm7O7f81UGCRZw5Pl31I=; b=Z1ywL27YB5/d1EIGvmofxrCQoU5zHRBcua7+X+a1JEuRur+AzpPGLBj0KVl+UMPZnN CJZDVOiHCB0PS+tcYJ8lKtKlh1gfFCz4NP1qKB4r5rWc9CJnC1pHnPcsQvIPsgrIA0hz 15xpcZPXhhpSG6j68aIZVU+ivdClFktWxR4K/LJ3VJ+DXajkWz0BnmdohMunqW5CvFQy flReGFAeqwv8iFyZSGnKc4jEL8Dso74R95UWOBVdIOvlV5gvD4LqvG88aV6X1Eqh7xoZ fQLmwpzMr6GfWidkDs7Qk7BLcMKs1e+/OCyr6dzwLGzhV2/Q8QA+e5lZFguDhVz2Xk9A uEVg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@voskuil-org.20230601.gappssmtp.com header.s=20230601 header.b=qpFPAoJn; spf=none (google.com: eric@voskuil.org does not designate permitted sender hosts) smtp.mailfrom=eric@voskuil.org; dara=pass header.i=@googlegroups.com Received: from mail-qv1-xf36.google.com (mail-qv1-xf36.google.com. [2607:f8b0:4864:20::f36]) by gmr-mx.google.com with ESMTPS id 41be03b00d2f7-b15fb4c0884si374319a12.5.2025.04.27.14.01.02 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 27 Apr 2025 14:01:02 -0700 (PDT) Received-SPF: none (google.com: eric@voskuil.org does not designate permitted sender hosts) client-ip=2607:f8b0:4864:20::f36; Received: by mail-qv1-xf36.google.com with SMTP id 6a1803df08f44-6e8f254b875so47954366d6.1 for ; Sun, 27 Apr 2025 14:01:02 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCWDJHQy/DpXtT1eMK/YaTC1CJJpADGZdQSMSYdoDSt5iYbQOvwkM1Aw3vtP3CHcpkkUS8JFBBclp7pb@googlegroups.com X-Gm-Gg: ASbGnctGEY9C9lukwIrsaOsraToV7GTtNYZZRh67Z8OK+DRNuMSTEu6blZg8+LtUJqY oTVJQka203uvvir4SEcTn/lLR6T6aYORP8cQSWnKzj3t8jLuUU1dwbRJ2Y/pk/5tausEdZr/nvu mry1NqE7SLb1X87xMmvqz2Xo/Z/2t90eH0028Zzr2eZbcDsaUvohtklnkzALmkIVOOZ5zsqU1Yh RUGTs8SSU05w6khnCsMSzCxzWkL4RJRM0xr8nXxi98MOp8RSqvDb6X2Z35rTGtfiqAogO9e1ZdU 6rDCZGBYSDyKiptJDMQwCgaeoH8mdVNF2tJ8BObx/5BEAfAQUry/8i6T7kBjos0SqOUzV8gnQMq kGFeD/Q== X-Received: by 2002:ad4:5e8b:0:b0:6e8:f940:50af with SMTP id 6a1803df08f44-6f4cbb58df5mr168293066d6.44.1745787661547; Sun, 27 Apr 2025 14:01:01 -0700 (PDT) Received: from ERICDESKTOP (c-73-227-67-43.hsd1.nh.comcast.net. [73.227.67.43]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6f4c096eb13sm49474806d6.68.2025.04.27.14.01.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 27 Apr 2025 14:01:01 -0700 (PDT) From: To: "'Ruben Somsen'" , References: <002201dbb7a2$74676640$5d3632c0$@voskuil.org> In-Reply-To: <002201dbb7a2$74676640$5d3632c0$@voskuil.org> Subject: RE: [bitcoindev] The Tragic Tale of BIP30 Date: Sun, 27 Apr 2025 17:01:00 -0400 Message-ID: <000201dbb7b7$7af02be0$70d083a0$@voskuil.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQHjQEGI0hVWVnHItjONPNIY3TwlQgE5+x3es58eFnA= Content-Language: en-us X-Original-Sender: eric@voskuil.org X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@voskuil-org.20230601.gappssmtp.com header.s=20230601 header.b=qpFPAoJn; spf=none (google.com: eric@voskuil.org does not designate permitted sender hosts) smtp.mailfrom=eric@voskuil.org; dara=pass header.i=@googlegroups.com Precedence: list Mailing-list: list bitcoindev@googlegroups.com; contact bitcoindev+owners@googlegroups.com List-ID: X-Google-Group-Id: 786775582512 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Score: -0.8 (/) > The top checkpoint is consensus for over 11 years Correction: the block of the top checkpoint has been confirmed for over 11 years. e > -----Original Message----- > From: eric@voskuil.org > Sent: Sunday, April 27, 2025 2:30 PM > To: 'Ruben Somsen' ; bitcoindev@googlegroups.com > Subject: RE: [bitcoindev] The Tragic Tale of BIP30 > > > The problem occurs when we reorg back to a point between block 91880 > > and 91722. When we rewind the blockchain, previously created outputs > > get removed from the UTXO set again. > > Consider that a UTXO set (accumulator) is an implementation detail. One of its > underlying assumptions is that a given txid cannot repeat in confirmed blocks. > This assumption did not hold, and the bip30 workaround for the two > exceptions failed to consider the effect of the above reorg given a UTXO set > design. Instead of removing the second instance, it removes all instances. > > > we could fix the bug by no longer removing the coinbase transaction in case > of a reorg of block 91880 and 91842. > > IMO this would be the most reasonable resolution, as it would produce the > outcome that in fact exists independent of the UTXO set. The previous blocks > still actually have the first instance of each duplicated coinbase. > > This is also inconsequential from a performance standpoint. It only affects > reorganization (infrequent), and only proceeds for these two specific blocks > (rare). Furthermore these two specific blocks are already exceptions and the > implementation is trivial. > > > However, it seems this never occurred. > > Correct, only the exception coinbases have been duplicated in the current > strong chain. > > > Aside from checking for coinbase uniqueness, we could also check that > > the coinbase will not conflict with any future coinbases (i.e. not > > conflict with > > BIP34 as well as the Consensus Cleanup BIP). > > The relationship between BIP34 and BIP30 is also a bit sordid, but the > presumption is that the Consensus Cleanup would resolve the existing flaw in > BIP34, and that the combination would effectively obsolete BIP30. Under the > assumption that this does in fact produce the intended outcome - that BIP34 > (presently) Consensus Cleanup (forever) makes further coinbase duplication > impossible, BIP30 can remain deactivated to the extent that these are active. > Nothing additional is required to avoid the presumably inefficient BIP30 > checks. > > > Once we fully [remove the checkpoints][3], the bug becomes theoretically > (not practically) exploitable. > > I would never advocate for adding more, but I'm not aware of any compelling > argument to hard fork out the existing checkpoints. The top checkpoint is > consensus for over 11 years and materially reduces the validation cost of > 295,000 blocks. > > > Doing this until block 227931 results in a modest ~7MB cache. However, > > BIP30 might not deactivate, in which case we'd have an ever-growing cache. > > This is solvable as follows.... > > This is a consequence of the presumed removal of checkpoints above BIP34 > activation height. IOW, removing the checkpoints makes it necessary to > validate BIP30 until BIP34 activates (block 227,931). The obvious solution to > this problem is to not create the problem in the first place. > > e > > > -----Original Message----- > > From: bitcoindev@googlegroups.com On > > Behalf Of Ruben Somsen > > Sent: Sunday, April 27, 2025 12:45 PM > > To: bitcoindev@googlegroups.com > > Subject: [bitcoindev] The Tragic Tale of BIP30 > > > > Markdown version: > > > https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b > > 7c > > > > ### Introduction > > > > In my recent exploration of [SwiftSync][1], I came to the realization > > that [BIP30][2] has an unresolved consensus bug. It seems this bug > > can't be triggered without a reorg back to 2010, so its seriousness is > > debatable. We currently have checkpoints up to 2013, preventing such a > > reorg. Once we fully [remove the checkpoints][3], the bug becomes > > theoretically (not practically) exploitable. > > > > BIP30 is also a bit of an odd duck in terms of consensus checks in > > that it involves the entire UTXO set without being related to the spending of > inputs. > > This is inefficient, and complicates the implementation of alternative > > validation methods such as utreexo, SwiftSync and quite possibly ZKP > > systems such as ZeroSync. It would be nice if we could sunset BIP30 > altogether. > > > > Without necessarily advocating for action (the status quo seems quite > > tenable), I'd like to lay out possible solutions for both and open up > > the discussion. > > > > ### 1. Consensus bug > > > > There are two duplicate transactions that BIP30 treats like > > exceptions. The last duplicate was the coinbase transaction in block > > 91880. When this transaction gets processed, the coinbase transaction > > in block 91722 is overwritten. The other instance occurs between these two > blocks (91812, 91842). > > > > The problem occurs when we reorg back to a point between block 91880 > > and 91722. When we rewind the blockchain, previously created outputs > > get removed from the UTXO set again. As a result, the overwritten > > output disappears from the UTXO set completely. A node that never > > witnessed the reorg, however, will still have the UTXO in its set. If > > subsequently the UTXO is ever spent, this would result in a fork. > > > > #### Solution A > > > > We could enforce that no reorg can take place between block 91722 and > > 91880 - you'd either have to reorg all of them, or none. This ensures > > both reorged and fresh nodes will not have the problematic outputs in > > their UTXO set. Considering this is only ~160 blocks at the low mining > > difficulty of 2010, this wouldn't be a big constraint. > > > > #### Solution B > > > > When discussing my findings with Sjors Provoost, he pointed out that > > the removal of the checkpoints (which can be seen as a hard fork) > > [that is being considered][3] also presents a window of opportunity to > > change the pre- checkpoint consensus rules - we could fix the bug by > > no longer removing the coinbase transaction in case of a reorg of > > block 91880 and 91842. Aside from that, Sjors' observation also opens > > up the question whether there are other > > pre-2013 consensus changes we'd want to consider making. > > > > ### 2. Sunsetting BIP30's UTXO set check > > > > BIP30 is currently active from genesis until [BIP34][4] activates at > > block height > > 227931 (March 2013). If this block is reorged out, BIP30 remains > > active indefinitely. BIP34 has issues of its own that are being > > addressed in the [Consensus Cleanup BIP][5] - you could go and read > > that, I won't go into too much detail here. > > > > Technically, BIP30 only prevents duplicate _unspent_ outputs. It does > > this by checking for each output whether it's already in the UTXO set > > (this is the inefficient part), and rejecting the block if it is. The > > two 2010 duplicates are hard-coded in as exceptions. Under these > > rules, spending an output and recreating it is allowed. However, it seems this > never occurred. > > > > One last point to address is why BIP34 gets deactivated if block > > 227931 is reorged out. The reason for this is because otherwise it'd > > open the door to possibly creating outputs prior to BIP34's activation > > that will conflict with BIP34's rules for ensuring coinbase > > transaction uniqueness (the exact issue the Consensus Cleanup is seeking to > resolve). > > > > Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check > > completely, ensuring it's no longer required, even in case of a reorg. > > > > #### Solution > > > > Given that we have no duplicates, barring the two exceptions, we could > > replace the inefficient BIP30 UTXO set check with a coinbase > > uniqueness check. We simply cache the coinbase TXIDs and ensure there are > no duplicates. > > Doing this until block 227931 results in a modest ~7MB cache. However, > > BIP30 might not deactivate, in which case we'd have an ever-growing cache. > > This is solvable as follows. > > > > Aside from checking for coinbase uniqueness, we could also check that > > the coinbase will not conflict with any future coinbases (i.e. not > > conflict with > > BIP34 as well as the Consensus Cleanup BIP). This ensures BIP34 can > > simply activate at block height 227931, regardless of whether there's a > reorg. > > > > ### In closing > > > > These were some of the issues with BIP30 and possible solutions. > > Regardless of whether we choose to take action, this write-up will serve as a > reference. > > Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for the > > discussions prior to publishing. > > > > -- Ruben Somsen > > > > > > [1]: > > > https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c7813 > > 3cd > > [2]: https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki > > [3]: https://github.com/bitcoin/bitcoin/pull/31649 > > [4]: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki > > [5]: https://github.com/bitcoin/bips/pull/1800 > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Bitcoin Development Mailing List" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to bitcoindev+unsubscribe@googlegroups.com > > . > > To view this discussion visit > > > https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Yo > > U3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com > > > > > oU3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com?utm_medium=em > > ail&utm_source=footer> . > -- You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/000201dbb7b7%247af02be0%2470d083a0%24%40voskuil.org.