public inbox for bitcoindev@googlegroups.com
 help / color / mirror / Atom feed
From: Mike Hearn <hearn@vinumeris.com>
To: bitcoin-xt <bitcoin-xt@googlegroups.com>
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Memory leaks?
Date: Tue, 20 Oct 2015 12:12:12 +0200	[thread overview]
Message-ID: <CA+w+GKTU6C7KKFx9dDd_--s1DQCO15n=034Lku2-kTYKf96XYw@mail.gmail.com> (raw)
In-Reply-To: <CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 4675 bytes --]

OK, then running under Valgrind whilst sending gbt RPCs would be the next
step.

On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> wrote:

> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880==    definitely lost: 0 bytes in 0 blocks
>> ==6880==    indirectly lost: 0 bytes in 0 blocks
>> ==6880==      possibly lost: 288 bytes in 1 blocks
>> ==6880==    still reachable: 10,552 bytes in 39 blocks
>> ==6880==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==    definitely lost: 0 bytes in 0 blocks
>> ==6778==    indirectly lost: 0 bytes in 0 blocks
>> ==6778==      possibly lost: 320 bytes in 1 blocks
>> ==6778==    still reachable: 10,080 bytes in 32 blocks
>> ==6778==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[-- Attachment #2: Type: text/html, Size: 7777 bytes --]

  parent reply	other threads:[~2015-10-20 10:12 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-10-13 21:56 [bitcoin-dev] Memory leaks? Jonathan Toomim (Toomim Bros)
2015-10-13 22:49 ` odinn
2015-10-13 23:14   ` Jonathan Toomim (Toomim Bros)
2015-10-13 23:52     ` Dave Scotese
2015-10-14  0:08       ` Jonathan Toomim (Toomim Bros)
2015-10-14  0:25         ` Dave Scotese
2015-10-13 23:59     ` odinn
2015-10-14  7:22 ` Pavel Janík
2015-10-14  7:58 ` Tom Zander
2015-10-14  9:09   ` Mike Hearn
2015-10-14 15:19     ` odinn
2015-10-14  9:39 ` Wladimir J. van der Laan
2015-10-18 15:59   ` Jonathan Toomim
2015-10-19 19:17     ` Multipool Admin
2015-10-19 22:24       ` Multipool Admin
2015-10-20 10:12       ` Mike Hearn [this message]
2015-10-20 12:39         ` Jonathan Toomim
2015-10-21  3:01           ` Jonathan Toomim
2015-10-21  7:29             ` Tom Zander
2015-10-21 17:58               ` Jonathan Toomim
2015-10-22 16:06               ` Multipool Admin
2015-10-22 16:27                 ` Jonathan Toomim
2015-10-22 17:01                   ` Btc Drak
2015-10-23  6:41                     ` Rusty Russell
2015-10-21  7:59             ` Ross Bennetts
2015-10-17  4:37 ` admin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CA+w+GKTU6C7KKFx9dDd_--s1DQCO15n=034Lku2-kTYKf96XYw@mail.gmail.com' \
    --to=hearn@vinumeris.com \
    --cc=bitcoin-dev@lists.linuxfoundation.org \
    --cc=bitcoin-xt@googlegroups.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox