From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id E803C3EE for ; Tue, 20 Oct 2015 10:12:13 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id CD9CBA9 for ; Tue, 20 Oct 2015 10:12:12 +0000 (UTC) Received: by igbni9 with SMTP id ni9so68320648igb.1 for ; Tue, 20 Oct 2015 03:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vinumeris.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+eJf9QL3JcrGdODQ0lESHLUb1tOScVfzsP5rJK9J9F0=; b=M/3tuJZcGpo9synulAf1wv1MiqfjFxkIXTXq2hEhC33mMtJ7q9khXLb5wbFBzkc0wg umRmGnq6ucKZQXCfRVo7IEI16yBRfJ5IW7B2MKMIowujUzKPDgu7Ne4wTazlJUQ8FGZj /G8MRKvOJIcVleNxIF/jyhHda2uJbhd9+m1so= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+eJf9QL3JcrGdODQ0lESHLUb1tOScVfzsP5rJK9J9F0=; b=jL4mbhXTeyJ2UYUpTXksMdkRj8S7MJQ7hGjLaMjZj1x3Lw+J8p6hY293M/V12xk7eZ yE6/aIUguhMQ87qptDzs3P+KjXqvlba+yN6P/HDpNtYxZm6Xx9BdYgF8N1//uz9l7eTo I6i1Gv3eYW7RCV7Ufn+b7nU6QOpxoDCGlJmD5rQY1vRN8BfzEYmflOWjsQZmdU6XTsfg nSkkbMRej9Tcytlt8mV6v5YZnoiNRr8G0e0RhEuLk1fHiZ/pb+wUt7NDMcBCSIApCsHS ckk1FjQlEMnoXqVzk6YiR35c7SxmVCJAJ0o6nsXs9DOC6NCEQaiJABvOgq7xN7aNGb9f biQA== X-Gm-Message-State: ALoCoQnXEHVtmiKyryK2zW4E0uJLrm7wRybc0opsPSTSE0pAa+weETJfbX7lEl8SEh9rkoTWlRIu MIME-Version: 1.0 X-Received: by 10.50.103.99 with SMTP id fv3mr23349471igb.69.1445335932201; Tue, 20 Oct 2015 03:12:12 -0700 (PDT) Received: by 10.50.123.166 with HTTP; Tue, 20 Oct 2015 03:12:12 -0700 (PDT) In-Reply-To: References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im> <20151014093913.GB19607@amethyst.visucore.com> Date: Tue, 20 Oct 2015 12:12:12 +0200 Message-ID: From: Mike Hearn To: bitcoin-xt Content-Type: multipart/alternative; boundary=047d7b2e08abd2095e0522867e14 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev Subject: Re: [bitcoin-dev] Memory leaks? X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 10:12:14 -0000 --047d7b2e08abd2095e0522867e14 Content-Type: text/plain; charset=UTF-8 OK, then running under Valgrind whilst sending gbt RPCs would be the next step. On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin wrote: > My nodes are continuously running getblocktemplate and getinfo, and I also > suspected the issue is in either gbt or the rpc server. > > The instance only takes a few hours to get up to that memory usage. > On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan >> wrote: >> >> This is *most likely* the mempool, but is just not reported correctly. >> >> >> I did some testing with PR #6410's better mempool reporting. The improved >> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is >> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). >> The excess memory usage that I'm seeing is on the order of 100x higher than >> the mempool "bytes": value. As such, I think it's unlikely that this is the >> mempool, or at least not normal/correct mempool behavior. >> >> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm >> guessing his bitcoind has been running longer than any of mine. His server >> definitely has more RAM. I don't know which email list he is subscribed to >> (probably XT), so I'm sharing it with both lists to make sure you're all >> aware of how big an issue this can be. >> >> In the meantime you can mitigate the mempool growth by setting >> `-mintxfee`, see >> >> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding >> >> >> I have mintxfee and minrelaytxfee set to about 0.00003, which is high >> enough to exclude essentially all of the of the 14700-14800 byte flood >> transactions. My nodes' mempools only contain about one or two blocks' >> worth of transactions. So I don't think this is correct either. >> >> >> >> Some additional notes on this issue: >> >> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a >> Core bitcoind process (commit d78a880) overnight with no mining connected >> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around >> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I >> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around >> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, >> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, >> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] >> getmempoolinfo is still showing "usage" around 25MB or less. >> >> I'll do some more testing with this and see if I can make it repeatable, >> and record the results more carefully. Expect a follow-up from me in a day >> or two. >> >> 2. valgrind did not show anything super promising. It did report this: >> >> ==6880== LEAK SUMMARY: >> ==6880== definitely lost: 0 bytes in 0 blocks >> ==6880== indirectly lost: 0 bytes in 0 blocks >> ==6880== possibly lost: 288 bytes in 1 blocks >> ==6880== still reachable: 10,552 bytes in 39 blocks >> ==6880== suppressed: 0 bytes in 0 blocks >> (Bitcoin Core commit d78a880) >> >> and this: >> ==6778== LEAK SUMMARY: >> ==6778== definitely lost: 0 bytes in 0 blocks >> ==6778== indirectly lost: 0 bytes in 0 blocks >> ==6778== possibly lost: 320 bytes in 1 blocks >> ==6778== still reachable: 10,080 bytes in 32 blocks >> ==6778== suppressed: 0 bytes in 0 blocks >> (Bitcoin XT commit fe446d) >> >> I haven't found anything in there yet that I think would produce the >> multi-GB memory usage after running for a few days, but I could be missing >> it. Email me if you want the full log. >> >> I did not try running getblocktemplate while valgrind was running. I'll >> have to try that. I also have not let valgrind run for more than an hour. >> >> >> >> P.S.: Sorry for all the cross-post confusion and consequent flamewar >> fallout. While it's probably too late for this thread, I'll make sure to >> post in a manner that keeps the threads clearly separate in the future >> (e.g. different subject lines). >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >> -- > You received this message because you are subscribed to the Google Groups > "bitcoin-xt" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to bitcoin-xt+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > --047d7b2e08abd2095e0522867e14 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
OK, then running under Valgrind whilst sending gbt RPCs wo= uld be the next step.

On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipo= ol.us> wrote:

My nodes are continuously running getblocktemplate and getinfo, and I als= o suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory= usage.

On Oct 18, 2015 8:59 AM, = "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists.linuxfoun= dation.org> wrote:
On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> w= rote:
This is *most likely* the mempool, but= is just not reported correctly.

I did = some testing with PR #6410's better mempool reporting. The improved rep= orting suggests that actual in-memory usage ("usage":) by CTxMemP= ool is about 2.5x to 3x higher than the serialized transaction sizes ("= ;bytes":). The excess memory usage that I'm seeing is on the order= of 100x higher than the mempool "bytes": value. As such, I think= it's unlikely that this is the mempool, or at least not normal/correct= mempool behavior.=C2=A0

Another user (admin@multipool.us) report= ed 35 GB of RSS usage. I'm guessing his bitcoind has been running longe= r than any of mine. His server definitely has more RAM. I don't know wh= ich email list he is subscribed to (probably XT), so I'm sharing it wit= h both lists to make sure you're all aware of how big an issue this can= be.

In the meantime you can mitigate th= e mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/relea= se-notes.md#transaction-flooding

I have = mintxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough = to exclude essentially all of the of the 14700-14800 byte flood transaction= s. My nodes' mempools only contain about one or two blocks' worth o= f transactions. So I don't think this is correct either.

=


Some additional notes on this issu= e:

1. I think it's related to CreateNewBlock()= and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overn= ight with no mining connected to it, and (IIRC -- my memory is fuzzy) when = I woke up it was using around 400 MB of RSS and the mempool was at around &= quot;bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getbloc= ktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getbl= ocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 M= B. An hour after that with more getblocktemplates, and now RSS is at 1350 M= B. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing = "usage" around 25MB or less.

I'll do= some more testing with this and see if I can make it repeatable, and recor= d the results more carefully. Expect a follow-up from me in a day or two.

2. valgrind did not show anything super promising. = It did report this:

=3D=3D6880=3D=3D LEAK SUMMARY:
= =3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los= t: 0 bytes in 0 blocks
=3D=3D6880=3D= =3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,= 552 bytes in 39 blocks
=3D=3D6880=3D= =3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks
=
(Bitcoin Core commit d78a880)

and this:
=
=3D=3D6778=3D=3D LEAK SUMMARY:
=
=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely = lost: 0 bytes in 0 blocks
=3D=3D6778= =3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32= 0 bytes in 1 blocks
=3D=3D6778=3D=3D= =C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks
=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse= d: 0 bytes in 0 blocks
(Bitcoin XT commit=C2=A0fe446d)

I haven't found anything in there yet that I think= would produce the multi-GB memory usage after running for a few days, but = I could be missing it. Email me if you want the full log.

I did not try running getblocktemplate while valgrind was running. = I'll have to try that. I also have not let valgrind run for more than a= n hour.=C2=A0



P.S.: = Sorry for all the cross-post confusion and consequent flamewar fallout. Whi= le it's probably too late for this thread, I'll make sure to post i= n a manner that keeps the threads clearly separate in the future (e.g. diff= erent subject lines).

_________= ______________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev

--
You received this message because you are subscribed to the Google Groups &= quot;bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to bitcoin-xt+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--047d7b2e08abd2095e0522867e14--