Hi dear devs,
1. THE ISSUE - DAEMON+CLI
========================
I had a wallet in a server production since 2017 (5 years old) and when it reached about 273 MB, 2.079.337 transactions and 446.503 generated addresses, the performance started to degrade exponentially.
Most of the commands, e.g. "getbalance", "walletpassphrase" and "getreceivedbyaddress" started to timeout (more than 15 minutes delay - default timeout). The CPU was 100% used (all 32 cores - with 150 load avg) and the machine became almost unusable breaking everything else, with the default config of 16 RPC threads and 15 min timeout and some attempt calls per mi
Increasing the timeout and/or the RPC threads in the config file turns things even worse.
Putting the wallet.dat in a very fast SSD disk and increasing the size of the cache (I tried with 8GB) have improved but I'm not sure if it is enough.
2. TEST ON BITCOIN QT
====================
If you try to load the wallet in the "bitcoin-qt" everything gets stuck, even the system (OS/UI) doesn't respond anymore. You click on a button and receive the message "window doesn't respond, wait or terminate?" - if you wait it releases after a while but it is slow and hard to use the wallet anyway.
3. WHY IS THIS SO BAD?
=====================
This is bad because the standard client becomes almost useless for the wallet feature:
3.1) the wallet Qt already is not so popular among end users. It doesn't look modern, slow to first sync and hard to use. That's why people prefer to use Electrum or Wasabi - I personally don't care but it's the sad truth;
3.2) it becomes useless now also for servers in production, forcing them to use third party solutions for huge wallets. Even if you split in 10 wallets it will just delay 10 times more each to degrade, postponing the problem but not eliminating it. Not to mention the slow and daily degradation.
4. SHOULD WE GIVE UP THE WALLET FEATURE?
========================================
Then, Bitcoin Core becomes just a reference implementation and blocks relayers, but as an application wallet itself turns into a really bad choice. --- It leads me to the following question: if we won't invest time on improving this, shouldn't we remove the wallet feature at all? Why keep a wallet feature that is not useful for the end user nor the production server? Is it useful for what then?
5. THE CURRENT "SOLUTION" IS BAD
===============================
Currently, the only "solution" for huge wallets is shameful: create a new one and send the funds there from time to time. But when is the right time exactly? The performance degrades suddenly or gets worse slowly for each new address and/or tx?. And besides not being an elegant solution and "not in the handbook", it also can break a lot of things like monitoring old addresses and also can lead to privacy concerns unifying lots of inputs in a big and expensive tx.
6. OTHER USER CASES?
====================
I think this could also become an issue if we have LN nodes that use the Bitcoin Core wallet infrastructure behind to open / close many channels for a long time.
7. FINAL THOUGHTS
=================
If moving the wallet from a HDD to a SSD improved a lot, maybe just caching the entire wallet in memory could improve even more, but I'm afraid some code optimization is also necessary.
8. SOME QUESTIONS
==================
8.1) Can we "optimize" a huge wallet without moving the funds to a new one? Like a "fsck" or eqv?
8.2) Can we improve the cache usage somehow? Putting the entire wallet in memory, for example?
8.3) Is it possible to reduce the wallet size (273 MB is too much for a HD wallet)?
8.4) Can we tell the CLI to ignore old addresses? What if I need to watch only the last 30 days?
8.5) How to improve the I/O treatment and/or CPU usage in the main thread on Bitcoin-Qt to avoid window freezing on big and huge wallets?
8.6) In the last case (if it was not possible to optimize the wallet or the CLI & Qt), can the CLI just warn the user like: "the wallet is becoming too big and slow, execute the command 'archive'". And then, the command "archive" could rename the current wallet to something like "wallet.dat.archive_until_20220818", create a new "wallet.dat" and move the funds automatically? Also, would it be nice to have an "autoarchivehugewallets=1" in the file config?
9. POSSIBLE RELATED AND TESTS
=============================
ANOTHER POSSIBLE BUG
======================
Even if my node is 100% sync:
2022-08-20T13:11:43Z UpdateTip: new best=00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b height=750300 version=0x2fffe000 log2_work=93.687081 tx=758181489 date='2022-08-20T13:11:16Z' progress=1.000000 cache=4.6MiB(34964txo)
After a "loadwallet" command I am getting an old / wrong balance. The wallet is already empty because I had moved the funds to a new one 3 or 4 days ago but it is still showing the old balance. I didn't receive any warning message saying the need for a rescan or something like that.
I am trying the "rescanblockchain" command but it is running and it taking a looooooooooooooong time.
Best regards,
Felipe.