From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2B2BAC0032 for ; Fri, 20 Oct 2023 22:02:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 026D34344B for ; Fri, 20 Oct 2023 22:02:07 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 026D34344B Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=protonmail.com header.i=@protonmail.com header.a=rsa-sha256 header.s=protonmail3 header.b=Ws2VTzOH X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -2.799 X-Spam-Level: X-Spam-Status: No, score=-2.799 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OF3XBNSnxqvV for ; Fri, 20 Oct 2023 22:02:02 +0000 (UTC) Received: from mail-0301.mail-europe.com (mail-0301.mail-europe.com [188.165.51.139]) by smtp2.osuosl.org (Postfix) with ESMTPS id 367F1404DA for ; Fri, 20 Oct 2023 22:02:02 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 367F1404DA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1697839314; x=1698098514; bh=yGpQoqq+qzDfv7S3hlVi41niXPvgNL9i+5e32xlxctQ=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=Ws2VTzOHAyyTONimWYGSAG/j4Hpdv4vRGeAw39t6w5w9TJ3ccePxBn0snbFD7R6TY VKBgRnhWpFcR0qObs0hcTr9koJ9gfcfP6tslh9Xeynfh4na2Vj2DX1pY6m2SgRrGkS lYUlx5DgpmHVX27vQoDX1fBXJQ2eY93K5n6QAyXZovSgWDtMzarv9HwjkuAam7sl0Y aRlC6g1nH5i0W7G8GhHUjz9Zb/NQdueGKBSKBDNh9q0Xagii0MfVHrqaMcmP+4EuxS q7OLWuSzDBDJJhhPOSoYRCDrVIsDeIX3nQOxlXFTujV/WeVrsvJ7ecUlF5nER4SFn2 eCZPLJGLdRKyw== Date: Fri, 20 Oct 2023 22:01:40 +0000 To: Peter Todd From: Fabian Message-ID: In-Reply-To: References: Feedback-ID: 5067558:user:proton MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailman-Approved-At: Sat, 21 Oct 2023 12:04:52 +0000 Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] Breaking change in calculation of hash_serialized_2 X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2023 22:02:07 -0000 Hi Peter, to my knowledge, this was never considered as an option previously (James c= orrect me if I am wrong). At least I couldn't find any reference to that in= the original proposal [1] and I can not remember it being discussed since = I have followed the project more closely (ca. 2020). Here are the reasons that I can think of why that might be the case: - If the serialization and hashing of the UTXO set works as intended, that = hash should be working just as well as the flat file hash and hash_serializ= ed_2 certainly was assumed to be robust since it has been around for a very= long time. So it may simply have been viewed as additional overhead. - We may want to optimize the serialization of data to file further, adding= compression, etc. to have smaller files that result in the same UTXO set w= ithout having to change the chainparams committing to that UTXO set or pote= ntially having multiple file hashes for the same block. - We may want to introduce other file hashing strategies instead that are m= ore optimized for P2P sharing of the UTXO snapshots. P2P sharing the UTXO s= et has always been part of the idea of assumeutxo but so far it hasn't been= explored very deeply. For more on this see the conversation on IRC that st= arted in the meeting yesterday between sipa, aj et al [2][3]. Cheers, Fabian [1] https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/propos= al [2] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-19#976439; [3] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-20#976636; ------- Original Message ------- On Friday, October 20th, 2023 at 7:34 PM, Peter Todd w= rote: > On Fri, Oct 20, 2023 at 05:19:19PM +0000, Fabian via bitcoin-dev wrote: >=20 > > Hello list, > >=20 > > on Wednesday I found a potential malleability issue in the UTXO set dum= p files > > generated for and used by assumeutxo [1]. On Thursday morning theStack = had > > found the cause of the issue [2]: A bug in the serialization of UTXOs f= or the > > calculation of hash_serialized_2. This is the value used by Bitcoin Cor= e to > > check if the UTXO set loaded from a dump file matches what is expected.= The > > value of hash_serialized_2 expected for a particular block is hardcoded= into > > the chainparams of each chain. >=20 >=20 > >=20 > > [1] https://github.com/bitcoin/bitcoin/issues/28675 > > [2] https://github.com/bitcoin/bitcoin/issues/28675#issuecomment-177038= 9468[3] https://github.com/bitcoin/bitcoin/pull/28685 >=20 >=20 > James made the following comment on the above issue: >=20 > > Wow, good find @fjahr et al. I wonder if there's any value in committin= g to a > > sha256sum of the snapshot file itself in the source code as a > > belt-and-suspenders remediation for issues like this. >=20 >=20 > Why isn't the sha256 hash of the snapshot file itself the canonical hash? > That would obviously eliminate any malleability issues. gettxoutsetinfo a= lready > has to walk the entire UTXO set to calculate the hash. Making it simply > generate the actual contents of the dump file and calculate the hash of i= t is > the obvious way to implement this. >=20 > -- > https://petertodd.org 'peter'[:-1]@petertodd.org