From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <hearn@vinumeris.com> Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id E803C3EE for <bitcoin-dev@lists.linuxfoundation.org>; Tue, 20 Oct 2015 10:12:13 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id CD9CBA9 for <bitcoin-dev@lists.linuxfoundation.org>; Tue, 20 Oct 2015 10:12:12 +0000 (UTC) Received: by igbni9 with SMTP id ni9so68320648igb.1 for <bitcoin-dev@lists.linuxfoundation.org>; Tue, 20 Oct 2015 03:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vinumeris.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+eJf9QL3JcrGdODQ0lESHLUb1tOScVfzsP5rJK9J9F0=; b=M/3tuJZcGpo9synulAf1wv1MiqfjFxkIXTXq2hEhC33mMtJ7q9khXLb5wbFBzkc0wg umRmGnq6ucKZQXCfRVo7IEI16yBRfJ5IW7B2MKMIowujUzKPDgu7Ne4wTazlJUQ8FGZj /G8MRKvOJIcVleNxIF/jyhHda2uJbhd9+m1so= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+eJf9QL3JcrGdODQ0lESHLUb1tOScVfzsP5rJK9J9F0=; b=jL4mbhXTeyJ2UYUpTXksMdkRj8S7MJQ7hGjLaMjZj1x3Lw+J8p6hY293M/V12xk7eZ yE6/aIUguhMQ87qptDzs3P+KjXqvlba+yN6P/HDpNtYxZm6Xx9BdYgF8N1//uz9l7eTo I6i1Gv3eYW7RCV7Ufn+b7nU6QOpxoDCGlJmD5rQY1vRN8BfzEYmflOWjsQZmdU6XTsfg nSkkbMRej9Tcytlt8mV6v5YZnoiNRr8G0e0RhEuLk1fHiZ/pb+wUt7NDMcBCSIApCsHS ckk1FjQlEMnoXqVzk6YiR35c7SxmVCJAJ0o6nsXs9DOC6NCEQaiJABvOgq7xN7aNGb9f biQA== X-Gm-Message-State: ALoCoQnXEHVtmiKyryK2zW4E0uJLrm7wRybc0opsPSTSE0pAa+weETJfbX7lEl8SEh9rkoTWlRIu MIME-Version: 1.0 X-Received: by 10.50.103.99 with SMTP id fv3mr23349471igb.69.1445335932201; Tue, 20 Oct 2015 03:12:12 -0700 (PDT) Received: by 10.50.123.166 with HTTP; Tue, 20 Oct 2015 03:12:12 -0700 (PDT) In-Reply-To: <CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com> References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im> <20151014093913.GB19607@amethyst.visucore.com> <F938BD02-3D80-4E99-BD1C-490543187895@toom.im> <CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com> Date: Tue, 20 Oct 2015 12:12:12 +0200 Message-ID: <CA+w+GKTU6C7KKFx9dDd_--s1DQCO15n=034Lku2-kTYKf96XYw@mail.gmail.com> From: Mike Hearn <hearn@vinumeris.com> To: bitcoin-xt <bitcoin-xt@googlegroups.com> Content-Type: multipart/alternative; boundary=047d7b2e08abd2095e0522867e14 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org> Subject: Re: [bitcoin-dev] Memory leaks? X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org> List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe> List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/> List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org> List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help> List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe> X-List-Received-Date: Tue, 20 Oct 2015 10:12:14 -0000 --047d7b2e08abd2095e0522867e14 Content-Type: text/plain; charset=UTF-8 OK, then running under Valgrind whilst sending gbt RPCs would be the next step. On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> wrote: > My nodes are continuously running getblocktemplate and getinfo, and I also > suspected the issue is in either gbt or the rpc server. > > The instance only takes a few hours to get up to that memory usage. > On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> >> wrote: >> >> This is *most likely* the mempool, but is just not reported correctly. >> >> >> I did some testing with PR #6410's better mempool reporting. The improved >> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is >> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). >> The excess memory usage that I'm seeing is on the order of 100x higher than >> the mempool "bytes": value. As such, I think it's unlikely that this is the >> mempool, or at least not normal/correct mempool behavior. >> >> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm >> guessing his bitcoind has been running longer than any of mine. His server >> definitely has more RAM. I don't know which email list he is subscribed to >> (probably XT), so I'm sharing it with both lists to make sure you're all >> aware of how big an issue this can be. >> >> In the meantime you can mitigate the mempool growth by setting >> `-mintxfee`, see >> >> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding >> >> >> I have mintxfee and minrelaytxfee set to about 0.00003, which is high >> enough to exclude essentially all of the of the 14700-14800 byte flood >> transactions. My nodes' mempools only contain about one or two blocks' >> worth of transactions. So I don't think this is correct either. >> >> >> >> Some additional notes on this issue: >> >> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a >> Core bitcoind process (commit d78a880) overnight with no mining connected >> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around >> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I >> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around >> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, >> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, >> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] >> getmempoolinfo is still showing "usage" around 25MB or less. >> >> I'll do some more testing with this and see if I can make it repeatable, >> and record the results more carefully. Expect a follow-up from me in a day >> or two. >> >> 2. valgrind did not show anything super promising. It did report this: >> >> ==6880== LEAK SUMMARY: >> ==6880== definitely lost: 0 bytes in 0 blocks >> ==6880== indirectly lost: 0 bytes in 0 blocks >> ==6880== possibly lost: 288 bytes in 1 blocks >> ==6880== still reachable: 10,552 bytes in 39 blocks >> ==6880== suppressed: 0 bytes in 0 blocks >> (Bitcoin Core commit d78a880) >> >> and this: >> ==6778== LEAK SUMMARY: >> ==6778== definitely lost: 0 bytes in 0 blocks >> ==6778== indirectly lost: 0 bytes in 0 blocks >> ==6778== possibly lost: 320 bytes in 1 blocks >> ==6778== still reachable: 10,080 bytes in 32 blocks >> ==6778== suppressed: 0 bytes in 0 blocks >> (Bitcoin XT commit fe446d) >> >> I haven't found anything in there yet that I think would produce the >> multi-GB memory usage after running for a few days, but I could be missing >> it. Email me if you want the full log. >> >> I did not try running getblocktemplate while valgrind was running. I'll >> have to try that. I also have not let valgrind run for more than an hour. >> >> >> >> P.S.: Sorry for all the cross-post confusion and consequent flamewar >> fallout. While it's probably too late for this thread, I'll make sure to >> post in a manner that keeps the threads clearly separate in the future >> (e.g. different subject lines). >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >> -- > You received this message because you are subscribed to the Google Groups > "bitcoin-xt" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to bitcoin-xt+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > --047d7b2e08abd2095e0522867e14 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">OK, then running under Valgrind whilst sending gbt RPCs wo= uld be the next step.</div><div class=3D"gmail_extra"><br><div class=3D"gma= il_quote">On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <span dir=3D"ltr= "><<a href=3D"mailto:admin@multipool.us" target=3D"_blank">admin@multipo= ol.us</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"m= argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir=3D"ltr= ">My nodes are continuously running getblocktemplate and getinfo, and I als= o suspected the issue is in either gbt or the rpc server.</p> <p dir=3D"ltr">The instance only takes a few hours to get up to that memory= usage.</p> <div class=3D"gmail_quote"><div><div class=3D"h5">On Oct 18, 2015 8:59 AM, = "Jonathan Toomim via bitcoin-dev" <<a href=3D"mailto:bitcoin-d= ev@lists.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoun= dation.org</a>> wrote:<br type=3D"attribution"></div></div><blockquote c= lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;= padding-left:1ex"><div><div class=3D"h5"><div style=3D"word-wrap:break-word= "><div><div>On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <<a hr= ef=3D"mailto:laanwj@gmail.com" target=3D"_blank">laanwj@gmail.com</a>> w= rote:</div><blockquote type=3D"cite">This is *most likely* the mempool, but= is just not reported correctly.<br></blockquote><div><br></div><div>I did = some testing with PR #6410's better mempool reporting. The improved rep= orting suggests that actual in-memory usage ("usage":) by CTxMemP= ool is about 2.5x to 3x higher than the serialized transaction sizes ("= ;bytes":). The excess memory usage that I'm seeing is on the order= of 100x higher than the mempool "bytes": value. As such, I think= it's unlikely that this is the mempool, or at least not normal/correct= mempool behavior.=C2=A0</div><div><br></div><div>Another user (<a href=3D"= mailto:admin@multipool.us" target=3D"_blank">admin@multipool.us</a>) report= ed 35 GB of RSS usage. I'm guessing his bitcoind has been running longe= r than any of mine. His server definitely has more RAM. I don't know wh= ich email list he is subscribed to (probably XT), so I'm sharing it wit= h both lists to make sure you're all aware of how big an issue this can= be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate th= e mempool growth by setting `-mintxfee`, see<br><a href=3D"https://github.c= om/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding" = target=3D"_blank">https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/relea= se-notes.md#transaction-flooding</a><br></blockquote><div><br></div>I have = mintxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough = to exclude essentially all of the of the 14700-14800 byte flood transaction= s. My nodes' mempools only contain about one or two blocks' worth o= f transactions. So I don't think this is correct either.</div><div><br>= </div><div><br></div><div><br></div><div>Some additional notes on this issu= e:</div><div><br></div><div>1. I think it's related to CreateNewBlock()= and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overn= ight with no mining connected to it, and (IIRC -- my memory is fuzzy) when = I woke up it was using around 400 MB of RSS and the mempool was at around &= quot;bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getbloc= ktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getbl= ocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 M= B. An hour after that with more getblocktemplates, and now RSS is at 1350 M= B. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing = "usage" around 25MB or less.</div><div><br></div><div>I'll do= some more testing with this and see if I can make it repeatable, and recor= d the results more carefully. Expect a follow-up from me in a day or two.</= div><div><br></div><div>2. valgrind did not show anything super promising. = It did report this:</div><div><br></div><div><div style=3D"margin:0px;font-= family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,= 0)">=3D=3D6880=3D=3D LEAK SUMMARY:</div><div style=3D"margin:0px;font-famil= y:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">= =3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks</div><di= v style=3D"margin:0px;font-family:'Andale Mono';color:rgb(41,249,20= );background-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los= t: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:'Andal= e Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D= =3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks</div><div styl= e=3D"margin:0px;font-family:'Andale Mono';color:rgb(41,249,20);back= ground-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,= 552 bytes in 39 blocks</div><div style=3D"margin:0px;font-family:'Andal= e Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D= =3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks</div></div>= <div>(Bitcoin Core commit d78a880)</div><div><br></div><div>and this:</div>= <div><div style=3D"margin:0px;font-family:'Andale Mono';color:rgb(4= 1,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D LEAK SUMMARY:</div>= <div style=3D"margin:0px;font-family:'Andale Mono';color:rgb(41,249= ,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely = lost: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:'An= dale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778= =3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks</div><div style=3D= "margin:0px;font-family:'Andale Mono';color:rgb(41,249,20);backgrou= nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32= 0 bytes in 1 blocks</div><div style=3D"margin:0px;font-family:'Andale M= ono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D= =C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks</div><div style=3D= "margin:0px;font-family:'Andale Mono';color:rgb(41,249,20);backgrou= nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse= d: 0 bytes in 0 blocks</div></div><div>(Bitcoin XT commit=C2=A0fe446d)</div= ><div><br></div><div>I haven't found anything in there yet that I think= would produce the multi-GB memory usage after running for a few days, but = I could be missing it. Email me if you want the full log.</div><div><br></d= iv><div>I did not try running getblocktemplate while valgrind was running. = I'll have to try that. I also have not let valgrind run for more than a= n hour.=C2=A0</div><div><br></div><div><br></div><div><br></div><div>P.S.: = Sorry for all the cross-post confusion and consequent flamewar fallout. Whi= le it's probably too late for this thread, I'll make sure to post i= n a manner that keeps the threads clearly separate in the future (e.g. diff= erent subject lines).</div></div><br></div></div><span class=3D"">_________= ______________________________________<br> bitcoin-dev mailing list<br> <a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">= bitcoin-dev@lists.linuxfoundation.org</a><br> <a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" = rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev</a><br> <br></span></blockquote></div><div class=3D"HOEnZb"><div class=3D"h5"> <p></p> -- <br> You received this message because you are subscribed to the Google Groups &= quot;bitcoin-xt" group.<br> To unsubscribe from this group and stop receiving emails from it, send an e= mail to <a href=3D"mailto:bitcoin-xt+unsubscribe@googlegroups.com" target= =3D"_blank">bitcoin-xt+unsubscribe@googlegroups.com</a>.<br> For more options, visit <a href=3D"https://groups.google.com/d/optout" targ= et=3D"_blank">https://groups.google.com/d/optout</a>.<br> </div></div></blockquote></div><br></div> --047d7b2e08abd2095e0522867e14--