[bitcoin-dev] Breaking change in calculation of hash_serialized_2

[Fabian <fjahr@protonmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2470 bytes --]

Hello list,

on Wednesday I found a potential malleability issue in the UTXO set dump files
generated for and used by assumeutxo [1]. On Thursday morning theStack had
found the cause of the issue [2]: A bug in the serialization of UTXOs for the
calculation of hash_serialized_2. This is the value used by Bitcoin Core to
check if the UTXO set loaded from a dump file matches what is expected. The
value of hash_serialized_2 expected for a particular block is hardcoded into
the chainparams of each chain.

Implications:
We have been working on a fix [3] for the serialization and aim to include it
in v26.0 (aimed to be released in November). The serialization must change
which means that all historical UTXO set hash results will change after you
upgrade your node to v26.0. To further highlight this, we will also increment
the version, i.e., the value returned in gettxoutset will be renamed to
hash_serialized_3.
It should also be noted that there were additional potentially problematic
issues found from fuzz testing by dergoegge which is why we decided to switch
the serialization completely rather than implementing a minimal fix. The
serialization format is now the same as used by MuHash.

How this may concern you:
1. If you are using hash_serialized_2 for any security critical purposes, you
should check if the bugs in the serialization code could cause issues for you.
You may switch to using hash_serialized_3 as soon as possible (or maybe
consider using MuHash).
2. If you are utilizing hash_serialized_2 for anything critical in your project
in general and require time to upgrade and adapt to the change described above,
please let us know. While we usually try to avoid breaking changes in our APIs without deprecation warning, we currently tend to think it is not necessary to
keep the buggy hash_serialized_2 around since we don’t know of any substantial
use cases and using it may even pose security risks. Furthermore, keeping the
old code around comes at some additional review and maintenance burden and may
lead to some delay in the release of v26.0. But we are happy to reconsider if
keeping hash_serialized_2 around holds serious value for downstream projects.

Feel free to reach out to me directly or comment in the PR [3] or here on the
list.

Cheers,
Fabian

[1] https://github.com/bitcoin/bitcoin/issues/28675
[2] https://github.com/bitcoin/bitcoin/issues/28675#issuecomment-1770389468[3] https://github.com/bitcoin/bitcoin/pull/28685

[-- Attachment #2: Type: text/html, Size: 4039 bytes --]