Re: [bitcoin-dev] Huge wallets make Bitcoin Core unusable (Daemon+CLI & Qt)

Re: [bitcoin-dev] Huge wallets make Bitcoin Core unusable (Daemon+CLI & Qt)

[Andrew Chow]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

This is a known issue that I've been working on. The wallet is a large module in Bitcoin Core and changing it takes quite a bit of time.

On 08/20/2022 10:16 AM, Felipe Micaroni Lalli via bitcoin-dev wrote:

> 8.1) Can we "optimize" a huge wallet without moving the funds to a new one? Like a "fsck" or eqv?

You can remove old transactions using the removeprunedfunds RPC. That should greatly speed up balance calculations and transaction creation.

> 8.2) Can we improve the cache usage somehow? Putting the entire wallet in memory, for example?

It's already entirely in memory.

[-- Attachment #2: Type: text/html, Size: 1410 bytes --]

Re: [bitcoin-dev] Huge wallets make Bitcoin Core unusable (Daemon+CLI & Qt)

[rhavar]

[-- Attachment #1: Type: text/plain, Size: 7056 bytes --]

Of all wallets I've tried, by a huge margin, bitcoin core performs the best with large wallets. I know several massive casinos like bustabit (which has a lot more transactions than your wallet) have been using bitcoin core for large wallets, so it can work.

The reason I don't recommend normal people use core, is because of the initial-block-download and not defaulting to a pruned mode.

The thing that makes your wallet slow is the large amount of wallet transactions. So the secret to running a large bitcoin core wallet is to just call `RemovePrunedFunds` on old transactions. Its actually a little tricky though, because you need to make sure the transaction is "safe to remove". The first, most obvious, reason a transaction would not be safe to remove is if it has a wallet utxo that you have not spent or have recently spent. The second, less obvious reason is if transaction B spends from transaction A, and you remove transaction B your wallet will think transaction A is unspent (but it's not!). So you have to prune "Depth first".

-Ryan

------- Original Message -------
On Saturday, August 20th, 2022 at 7:16 AM, Felipe Micaroni Lalli via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi dear devs,
>
> 1. THE ISSUE - DAEMON+CLI
> ========================
> I had a wallet in a server production since 2017 (5 years old) and when it reached about 273 MB, 2.079.337 transactions and 446.503 generated addresses, the performance started to degrade exponentially.
>
> Most of the commands, e.g. "getbalance", "walletpassphrase" and "getreceivedbyaddress" started to timeout (more than 15 minutes delay - default timeout). The CPU was 100% used (all 32 cores - with 150 load avg) and the machine became almost unusable breaking everything else, with the default config of 16 RPC threads and 15 min timeout and some attempt calls per mi
>
> Increasing the timeout and/or the RPC threads in the config file turns things even worse.
>
> Putting the wallet.dat in a very fast SSD disk and increasing the size of the cache (I tried with 8GB) have improved but I'm not sure if it is enough.
>
> 2. TEST ON BITCOIN QT
> ====================
> If you try to load the wallet in the "bitcoin-qt" everything gets stuck, even the system (OS/UI) doesn't respond anymore. You click on a button and receive the message "window doesn't respond, wait or terminate?" - if you wait it releases after a while but it is slow and hard to use the wallet anyway.
>
> 3. WHY IS THIS SO BAD?
> =====================
> This is bad because the standard client becomes almost useless for the wallet feature:
>
> 3.1) the wallet Qt already is not so popular among end users. It doesn't look modern, slow to first sync and hard to use. That's why people prefer to use Electrum or Wasabi - I personally don't care but it's the sad truth;
>
> 3.2) it becomes useless now also for servers in production, forcing them to use third party solutions for huge wallets. Even if you split in 10 wallets it will just delay 10 times more each to degrade, postponing the problem but not eliminating it. Not to mention the slow and daily degradation.
>
> 4. SHOULD WE GIVE UP THE WALLET FEATURE?
> ========================================
> Then, Bitcoin Core becomes just a reference implementation and blocks relayers, but as an application wallet itself turns into a really bad choice. --- It leads me to the following question: if we won't invest time on improving this, shouldn't we remove the wallet feature at all? Why keep a wallet feature that is not useful for the end user nor the production server? Is it useful for what then?
>
> 5. THE CURRENT "SOLUTION" IS BAD
> ===============================
> Currently, the only "solution" for huge wallets is shameful: create a new one and send the funds there from time to time. But when is the right time exactly? The performance degrades suddenly or gets worse slowly for each new address and/or tx?. And besides not being an elegant solution and "not in the handbook", it also can break a lot of things like monitoring old addresses and also can lead to privacy concerns unifying lots of inputs in a big and expensive tx.
>
> 6. OTHER USER CASES?
> ====================
> I think this could also become an issue if we have LN nodes that use the Bitcoin Core wallet infrastructure behind to open / close many channels for a long time.
>
> 7. FINAL THOUGHTS
> =================
> If moving the wallet from a HDD to a SSD improved a lot, maybe just caching the entire wallet in memory could improve even more, but I'm afraid some code optimization is also necessary.
>
> 8. SOME QUESTIONS
> ==================
> 8.1) Can we "optimize" a huge wallet without moving the funds to a new one? Like a "fsck" or eqv?
>
> 8.2) Can we improve the cache usage somehow? Putting the entire wallet in memory, for example?
>
> 8.3) Is it possible to reduce the wallet size (273 MB is too much for a HD wallet)?
>
> 8.4) Can we tell the CLI to ignore old addresses? What if I need to watch only the last 30 days?
>
> 8.5) How to improve the I/O treatment and/or CPU usage in the main thread on Bitcoin-Qt to avoid window freezing on big and huge wallets?
>
> 8.6) In the last case (if it was not possible to optimize the wallet or the CLI & Qt), can the CLI just warn the user like: "the wallet is becoming too big and slow, execute the command 'archive'". And then, the command "archive" could rename the current wallet to something like "wallet.dat.archive_until_20220818", create a new "wallet.dat" and move the funds automatically? Also, would it be nice to have an "autoarchivehugewallets=1" in the file config?
>
> 9. POSSIBLE RELATED AND TESTS
> =============================
>
> [1] https://github.com/bitcoin/bitcoin/issues/15015
> [2] https://github.com/bitcoin/bitcoin/issues/15148
> [3] https://github.com/bitcoin/bitcoin/issues/16874
> [4] https://github.com/bitcoin/bitcoin/pull/17135
> [5] https://github.com/bitcoin/bitcoin/pull/18160
> [6] https://github.com/bitpay/bitcore-node/issues/463
> [7] https://github.com/RavenProject/Ravencoin/issues/499
> [8] https://github.com/sugarchain-project/sugarchain/issues/106
> [9] https://bitcoin.stackexchange.com/questions/111844/loadwallet-takes-too-much-and-times-out
> [10] https://bitcoin.stackexchange.com/a/45713/1761
>
> ANOTHER POSSIBLE BUG
> ======================
>
> Even if my node is 100% sync:
> 2022-08-20T13:11:43Z UpdateTip: new best=00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b height=750300 version=0x2fffe000 log2_work=93.687081 tx=758181489 date='2022-08-20T13:11:16Z' progress=1.000000 cache=4.6MiB(34964txo)
>
> After a "loadwallet" command I am getting an old / wrong balance. The wallet is already empty because I had moved the funds to a new one 3 or 4 days ago but it is still showing the old balance. I didn't receive any warning message saying the need for a rescan or something like that.
>
> I am trying the "rescanblockchain" command but it is running and it taking a looooooooooooooong time.
>
> Best regards,
>
> Felipe.

[-- Attachment #2: Type: text/html, Size: 18651 bytes --]

[bitcoin-dev] Huge wallets make Bitcoin Core unusable (Daemon+CLI & Qt)

[micaroni]

[-- Attachment #1: Type: text/plain, Size: 5720 bytes --]

Hi dear devs,


1. THE ISSUE - DAEMON+CLI
========================
I had a wallet in a server production since 2017 (5 years old) and when it
reached about 273 MB, 2.079.337 transactions and 446.503 generated
addresses, the performance started to degrade exponentially.

Most of the commands, e.g. "getbalance", "walletpassphrase" and
"getreceivedbyaddress" started to timeout (more than 15 minutes delay -
default timeout). The CPU was 100% used (all 32 cores - with 150 load avg)
and the machine became almost unusable breaking everything else, with the
default config of 16 RPC threads and 15 min timeout and some attempt calls
per mi

Increasing the timeout and/or the RPC threads in the config file turns
things even worse.

Putting the wallet.dat in a very fast SSD disk and increasing the size of
the cache (I tried with 8GB) have improved but I'm not sure if it is enough.


2. TEST ON BITCOIN QT
====================
If you try to load the wallet in the "bitcoin-qt" everything gets stuck,
even the system (OS/UI) doesn't respond anymore. You click on a button and
receive the message "window doesn't respond, wait or terminate?" - if you
wait it releases after a while but it is slow and hard to use the wallet
anyway.


3. WHY IS THIS SO BAD?
=====================
This is bad because the standard client becomes almost useless for the
wallet feature:

3.1) the wallet Qt already is not so popular among end users. It doesn't
look modern, slow to first sync and hard to use. That's why people prefer
to use Electrum or Wasabi - I personally don't care but it's the sad truth;

3.2) it becomes useless now also for servers in production, forcing them to
use third party solutions for huge wallets. Even if you split in 10 wallets
it will just delay 10 times more each to degrade, postponing the problem
but not eliminating it. Not to mention the slow and daily degradation.


4. SHOULD WE GIVE UP THE WALLET FEATURE?
========================================
Then, Bitcoin Core becomes just a reference implementation and blocks
relayers, but as an application wallet itself turns into a really bad
choice. --- It leads me to the following question: if we won't invest time
on improving this, shouldn't we remove the wallet feature at all? Why keep
a wallet feature that is not useful for the end user nor the production
server? Is it useful for what then?


5. THE CURRENT "SOLUTION" IS BAD
===============================
Currently, the only "solution" for huge wallets is shameful: create a new
one and send the funds there from time to time. But when is the right time
exactly? The performance degrades suddenly or gets worse slowly for each
new address and/or tx?. And besides not being an elegant solution and "not
in the handbook", it also can break a lot of things like monitoring old
addresses and also can lead to privacy concerns unifying lots of inputs in
a big and expensive tx.


6. OTHER USER CASES?
====================
I think this could also become an issue if we have LN nodes that use the
Bitcoin Core wallet infrastructure behind to open / close many channels for
a long time.


7. FINAL THOUGHTS
=================
If moving the wallet from a HDD to a SSD improved a lot, maybe just caching
the entire wallet in memory could improve even more, but I'm afraid some
code optimization is also necessary.


8. SOME QUESTIONS
==================
8.1) Can we "optimize" a huge wallet without moving the funds to a new one?
Like a "fsck" or eqv?

8.2) Can we improve the cache usage somehow? Putting the entire wallet in
memory, for example?

8.3) Is it possible to reduce the wallet size (273 MB is too much for a HD
wallet)?

8.4) Can we tell the CLI to ignore old addresses? What if I need to watch
only the last 30 days?

8.5) How to improve the I/O treatment and/or CPU usage in the main thread
on Bitcoin-Qt to avoid window freezing on big and huge wallets?

8.6) In the last case (if it was not possible to optimize the wallet or the
CLI & Qt), can the CLI just warn the user like: "the wallet is becoming too
big and slow, execute the command 'archive'". And then, the command
"archive" could rename the current wallet to something like
"wallet.dat.archive_until_20220818", create a new "wallet.dat" and move the
funds automatically? Also, would it be nice to have an
"autoarchivehugewallets=1" in the file config?


9. POSSIBLE RELATED AND TESTS
=============================

[1] https://github.com/bitcoin/bitcoin/issues/15015
[2] https://github.com/bitcoin/bitcoin/issues/15148
[3] https://github.com/bitcoin/bitcoin/issues/16874
[4] https://github.com/bitcoin/bitcoin/pull/17135
[5] https://github.com/bitcoin/bitcoin/pull/18160
[6] https://github.com/bitpay/bitcore-node/issues/463
[7] https://github.com/RavenProject/Ravencoin/issues/499
[8] https://github.com/sugarchain-project/sugarchain/issues/106
[9]
https://bitcoin.stackexchange.com/questions/111844/loadwallet-takes-too-much-and-times-out
[10] https://bitcoin.stackexchange.com/a/45713/1761


ANOTHER POSSIBLE BUG
======================

Even if my node is 100% sync:
2022-08-20T13:11:43Z UpdateTip: new
best=00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b
height=750300 version=0x2fffe000 log2_work=93.687081 tx=758181489
date='2022-08-20T13:11:16Z' progress=1.000000 cache=4.6MiB(34964txo)

After a "loadwallet" command I am getting an old / wrong balance. The
wallet is already empty because I had moved the funds to a new one 3 or 4
days ago but it is still showing the old  balance. I didn't receive any
warning message saying the need for a rescan or something like that.

I am trying the "rescanblockchain" command but it is running and it taking
a looooooooooooooong time.




Best regards,

Felipe.

[-- Attachment #2: Type: text/html, Size: 16379 bytes --]