From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 9D7251BB for ; Mon, 19 Oct 2015 19:17:16 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-oi0-f47.google.com (mail-oi0-f47.google.com [209.85.218.47]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id B9147128 for ; Mon, 19 Oct 2015 19:17:15 +0000 (UTC) Received: by oiev17 with SMTP id v17so57219144oie.2 for ; Mon, 19 Oct 2015 12:17:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=A1AoA1WoG4ZQNpbVIgH+aqD4g+SJ7lt88jLEvw7V9pw=; b=ERLDw+VDeXH5ZPZ8KbXv3YrJF/fzPsA+BRzJ8IuiuR9BidVDVKkEGMeB4bWqms7qbl i80esks5nzegMaum7CxgrzSABe15DCekcupL65LZfDave7WMZIC8tK5q+DU7bXyChPbP FlEekHm1MaguJdhbywJqNqMvj3lP6MEVA+AXCbmay2NvNFmuJtBQfX7y6uIUSZmB9/mH X18o4qfy+u4EgIno7Rc4ErLEYjdvw7Ne0NnoFImZNuX/TQWWFH90S86EapYTcYYgVi6n 6Bq0npTh9KAPZDjgvOMmu/eTRadacSWrsBW5iKBPo/dAP3A5BvRWxMRTdY1u0wsmzo4j vZHA== X-Gm-Message-State: ALoCoQkZx1fv8+eE56GeQPh/1LbHNs5qhOYp7hNI6B/OAlsUSXhjlyh3//V+oalodBo0J8vYTF/B MIME-Version: 1.0 X-Received: by 10.202.86.86 with SMTP id k83mr18476971oib.68.1445282235063; Mon, 19 Oct 2015 12:17:15 -0700 (PDT) Received: by 10.182.220.193 with HTTP; Mon, 19 Oct 2015 12:17:14 -0700 (PDT) Received: by 10.182.220.193 with HTTP; Mon, 19 Oct 2015 12:17:14 -0700 (PDT) In-Reply-To: References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im> <20151014093913.GB19607@amethyst.visucore.com> Date: Mon, 19 Oct 2015 12:17:14 -0700 Message-ID: From: Multipool Admin To: Jonathan Toomim Content-Type: multipart/alternative; boundary=001a113dd26838b8c7052279fec9 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev , bitcoin-xt Subject: Re: [bitcoin-dev] Memory leaks? X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 19:17:16 -0000 --001a113dd26838b8c7052279fec9 Content-Type: text/plain; charset=UTF-8 My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server. The instance only takes a few hours to get up to that memory usage. On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" < bitcoin-dev@lists.linuxfoundation.org> wrote: > On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan > wrote: > > This is *most likely* the mempool, but is just not reported correctly. > > > I did some testing with PR #6410's better mempool reporting. The improved > reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is > about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). > The excess memory usage that I'm seeing is on the order of 100x higher than > the mempool "bytes": value. As such, I think it's unlikely that this is the > mempool, or at least not normal/correct mempool behavior. > > Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm > guessing his bitcoind has been running longer than any of mine. His server > definitely has more RAM. I don't know which email list he is subscribed to > (probably XT), so I'm sharing it with both lists to make sure you're all > aware of how big an issue this can be. > > In the meantime you can mitigate the mempool growth by setting > `-mintxfee`, see > > https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding > > > I have mintxfee and minrelaytxfee set to about 0.00003, which is high > enough to exclude essentially all of the of the 14700-14800 byte flood > transactions. My nodes' mempools only contain about one or two blocks' > worth of transactions. So I don't think this is correct either. > > > > Some additional notes on this issue: > > 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a > Core bitcoind process (commit d78a880) overnight with no mining connected > to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around > 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I > ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around > 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, > and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, > and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] > getmempoolinfo is still showing "usage" around 25MB or less. > > I'll do some more testing with this and see if I can make it repeatable, > and record the results more carefully. Expect a follow-up from me in a day > or two. > > 2. valgrind did not show anything super promising. It did report this: > > ==6880== LEAK SUMMARY: > ==6880== definitely lost: 0 bytes in 0 blocks > ==6880== indirectly lost: 0 bytes in 0 blocks > ==6880== possibly lost: 288 bytes in 1 blocks > ==6880== still reachable: 10,552 bytes in 39 blocks > ==6880== suppressed: 0 bytes in 0 blocks > (Bitcoin Core commit d78a880) > > and this: > ==6778== LEAK SUMMARY: > ==6778== definitely lost: 0 bytes in 0 blocks > ==6778== indirectly lost: 0 bytes in 0 blocks > ==6778== possibly lost: 320 bytes in 1 blocks > ==6778== still reachable: 10,080 bytes in 32 blocks > ==6778== suppressed: 0 bytes in 0 blocks > (Bitcoin XT commit fe446d) > > I haven't found anything in there yet that I think would produce the > multi-GB memory usage after running for a few days, but I could be missing > it. Email me if you want the full log. > > I did not try running getblocktemplate while valgrind was running. I'll > have to try that. I also have not let valgrind run for more than an hour. > > > > P.S.: Sorry for all the cross-post confusion and consequent flamewar > fallout. While it's probably too late for this thread, I'll make sure to > post in a manner that keeps the threads clearly separate in the future > (e.g. different subject lines). > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > --001a113dd26838b8c7052279fec9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

My nodes are continuously running getblocktemplate and getin= fo, and I also suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory= usage.

On Oct 18, 2015 8:59 AM, "Jonathan Toomim v= ia bitcoin-dev" <bitcoin-dev@lists.linuxfoundation.org> wrote:
On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> wr= ote:
This is *most likely* the mempool, but = is just not reported correctly.

I did s= ome testing with PR #6410's better mempool reporting. The improved repo= rting suggests that actual in-memory usage ("usage":) by CTxMemPo= ol is about 2.5x to 3x higher than the serialized transaction sizes ("= bytes":). The excess memory usage that I'm seeing is on the order = of 100x higher than the mempool "bytes": value. As such, I think = it's unlikely that this is the mempool, or at least not normal/correct = mempool behavior.=C2=A0

Another user (admin@multipool.us) reporte= d 35 GB of RSS usage. I'm guessing his bitcoind has been running longer= than any of mine. His server definitely has more RAM. I don't know whi= ch email list he is subscribed to (probably XT), so I'm sharing it with= both lists to make sure you're all aware of how big an issue this can = be.

In the meantime you can mitigate the= mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/releas= e-notes.md#transaction-flooding

I have m= intxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough t= o exclude essentially all of the of the 14700-14800 byte flood transactions= . My nodes' mempools only contain about one or two blocks' worth of= transactions. So I don't think this is correct either.

<= /div>


Some additional notes on this issue= :

1. I think it's related to CreateNewBlock() = and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overni= ght with no mining connected to it, and (IIRC -- my memory is fuzzy) when I= woke up it was using around 400 MB of RSS and the mempool was at around &q= uot;bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblock= template once, and IIRC the RSS shot up to around 800 MB. I then ran getblo= cktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB= . An hour after that with more getblocktemplates, and now RSS is at 1350 MB= . [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing &= quot;usage" around 25MB or less.

I'll do = some more testing with this and see if I can make it repeatable, and record= the results more carefully. Expect a follow-up from me in a day or two.

2. valgrind did not show anything super promising. I= t did report this:

=3D=3D6880=3D=3D LEAK SUMMARY:
= =3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los= t: 0 bytes in 0 blocks
=3D=3D6880=3D= =3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,= 552 bytes in 39 blocks
=3D=3D6880=3D= =3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks
=
(Bitcoin Core commit d78a880)

and this:
=
=3D=3D6778=3D=3D LEAK SUMMARY:
=
=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely = lost: 0 bytes in 0 blocks
=3D=3D6778= =3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32= 0 bytes in 1 blocks
=3D=3D6778=3D=3D= =C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks
=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse= d: 0 bytes in 0 blocks
(Bitcoin XT commit=C2=A0fe446d)

I haven't found anything in there yet that I think= would produce the multi-GB memory usage after running for a few days, but = I could be missing it. Email me if you want the full log.

I did not try running getblocktemplate while valgrind was running. = I'll have to try that. I also have not let valgrind run for more than a= n hour.=C2=A0



P.S.: = Sorry for all the cross-post confusion and consequent flamewar fallout. Whi= le it's probably too late for this thread, I'll make sure to post i= n a manner that keeps the threads clearly separate in the future (e.g. diff= erent subject lines).

______________________________________= _________
bitcoin-dev mailing list
bitcoin-dev@lists.= linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev

--001a113dd26838b8c7052279fec9--