Re: [bitcoin-dev] Breaking change in calculation of hash_serialized_2

[Fabian <fjahr@protonmail.com>]

Hi Peter,

to my knowledge, this was never considered as an option previously (James correct me if I am wrong). At least I couldn't find any reference to that in the original proposal [1] and I can not remember it being discussed since I have followed the project more closely (ca. 2020).

Here are the reasons that I can think of why that might be the case:

- If the serialization and hashing of the UTXO set works as intended, that hash should be working just as well as the flat file hash and hash_serialized_2 certainly was assumed to be robust since it has been around for a very long time. So it may simply have been viewed as additional overhead.
- We may want to optimize the serialization of data to file further, adding compression, etc. to have smaller files that result in the same UTXO set without having to change the chainparams committing to that UTXO set or potentially having multiple file hashes for the same block.
- We may want to introduce other file hashing strategies instead that are more optimized for P2P sharing of the UTXO snapshots. P2P sharing the UTXO set has always been part of the idea of assumeutxo but so far it hasn't been explored very deeply. For more on this see the conversation on IRC that started in the meeting yesterday between sipa, aj et al [2][3].

Cheers,
Fabian

[1] https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal
[2] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-19#976439;
[3] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-20#976636;

------- Original Message -------
On Friday, October 20th, 2023 at 7:34 PM, Peter Todd <pete@petertodd.org> wrote:


> On Fri, Oct 20, 2023 at 05:19:19PM +0000, Fabian via bitcoin-dev wrote:
> 
> > Hello list,
> > 
> > on Wednesday I found a potential malleability issue in the UTXO set dump files
> > generated for and used by assumeutxo [1]. On Thursday morning theStack had
> > found the cause of the issue [2]: A bug in the serialization of UTXOs for the
> > calculation of hash_serialized_2. This is the value used by Bitcoin Core to
> > check if the UTXO set loaded from a dump file matches what is expected. The
> > value of hash_serialized_2 expected for a particular block is hardcoded into
> > the chainparams of each chain.
> 
> 
> <snip>
> 
> > [1] https://github.com/bitcoin/bitcoin/issues/28675
> > [2] https://github.com/bitcoin/bitcoin/issues/28675#issuecomment-1770389468[3] https://github.com/bitcoin/bitcoin/pull/28685
> 
> 
> James made the following comment on the above issue:
> 
> > Wow, good find @fjahr et al. I wonder if there's any value in committing to a
> > sha256sum of the snapshot file itself in the source code as a
> > belt-and-suspenders remediation for issues like this.
> 
> 
> Why isn't the sha256 hash of the snapshot file itself the canonical hash?
> That would obviously eliminate any malleability issues. gettxoutsetinfo already
> has to walk the entire UTXO set to calculate the hash. Making it simply
> generate the actual contents of the dump file and calculate the hash of it is
> the obvious way to implement this.
> 
> --
> https://petertodd.org 'peter'[:-1]@petertodd.org