From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id B99AF410 for ; Mon, 19 Oct 2015 22:24:12 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-ob0-f176.google.com (mail-ob0-f176.google.com [209.85.214.176]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DE18E122 for ; Mon, 19 Oct 2015 22:24:11 +0000 (UTC) Received: by obbda8 with SMTP id da8so151538162obb.1 for ; Mon, 19 Oct 2015 15:24:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=dH56yv/sQF2QQQ3QKj83S2DTGVO5+ILjnNggaiY4KBE=; b=fFDWVrntA3wEYj6om3oYQF4Yz1y+wroSglQN24pSFeQnA0sm9TA8hrYe0uRfOkxfMf L8FbS5DQQLEeMKOwfoxcMp95YCY+7frjzQ+98NrsO2MkIPd59cc7zKAiotyy5cnCYgAF szDGwZoqS6N1o4cJd6bdFgfDkjFM3EdWW5RfI5tDdBYio+o9QDsFHEZrjzOllHm0BvR1 Nycogfbfcj1pyryzTaldl81k73x4P4d1YfnNLZlyZhtm7Z0z2dqo796AcZ7EY4T3HrL5 fgbbswgsg7Y0xIhqjc2/+MNc1VzZ0JpvSBXWoQQ81UybUVZqOr1A3v/nBxeVQJXDwaIa tA1A== X-Gm-Message-State: ALoCoQkwCvRwQ6BA/WTZsRGbgoFHo7YKHfSbW1I3HQXwZq/P/Wx4yUdaev9FAV+31E8wiaeHOmVY MIME-Version: 1.0 X-Received: by 10.182.133.9 with SMTP id oy9mr20196412obb.41.1445293451318; Mon, 19 Oct 2015 15:24:11 -0700 (PDT) Received: by 10.182.220.193 with HTTP; Mon, 19 Oct 2015 15:24:11 -0700 (PDT) In-Reply-To: References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im> <20151014093913.GB19607@amethyst.visucore.com> Date: Mon, 19 Oct 2015 15:24:11 -0700 Message-ID: From: Multipool Admin To: Jonathan Toomim Content-Type: multipart/alternative; boundary=e89a8ff1ccc2c32ff605227c9a7d X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev , bitcoin-xt Subject: Re: [bitcoin-dev] Memory leaks? X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 22:24:12 -0000 --e89a8ff1ccc2c32ff605227c9a7d Content-Type: text/plain; charset=UTF-8 I should also mention that this is definitely not an attack coming from connected nodes. My node experiencing the issue is only connected to 3 other nodes, all of which I control (via connect=). --Adam On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin wrote: > My nodes are continuously running getblocktemplate and getinfo, and I also > suspected the issue is in either gbt or the rpc server. > > The instance only takes a few hours to get up to that memory usage. > On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan >> wrote: >> >> This is *most likely* the mempool, but is just not reported correctly. >> >> >> I did some testing with PR #6410's better mempool reporting. The improved >> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is >> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). >> The excess memory usage that I'm seeing is on the order of 100x higher than >> the mempool "bytes": value. As such, I think it's unlikely that this is the >> mempool, or at least not normal/correct mempool behavior. >> >> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm >> guessing his bitcoind has been running longer than any of mine. His server >> definitely has more RAM. I don't know which email list he is subscribed to >> (probably XT), so I'm sharing it with both lists to make sure you're all >> aware of how big an issue this can be. >> >> In the meantime you can mitigate the mempool growth by setting >> `-mintxfee`, see >> >> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding >> >> >> I have mintxfee and minrelaytxfee set to about 0.00003, which is high >> enough to exclude essentially all of the of the 14700-14800 byte flood >> transactions. My nodes' mempools only contain about one or two blocks' >> worth of transactions. So I don't think this is correct either. >> >> >> >> Some additional notes on this issue: >> >> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a >> Core bitcoind process (commit d78a880) overnight with no mining connected >> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around >> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I >> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around >> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, >> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, >> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] >> getmempoolinfo is still showing "usage" around 25MB or less. >> >> I'll do some more testing with this and see if I can make it repeatable, >> and record the results more carefully. Expect a follow-up from me in a day >> or two. >> >> 2. valgrind did not show anything super promising. It did report this: >> >> ==6880== LEAK SUMMARY: >> ==6880== definitely lost: 0 bytes in 0 blocks >> ==6880== indirectly lost: 0 bytes in 0 blocks >> ==6880== possibly lost: 288 bytes in 1 blocks >> ==6880== still reachable: 10,552 bytes in 39 blocks >> ==6880== suppressed: 0 bytes in 0 blocks >> (Bitcoin Core commit d78a880) >> >> and this: >> ==6778== LEAK SUMMARY: >> ==6778== definitely lost: 0 bytes in 0 blocks >> ==6778== indirectly lost: 0 bytes in 0 blocks >> ==6778== possibly lost: 320 bytes in 1 blocks >> ==6778== still reachable: 10,080 bytes in 32 blocks >> ==6778== suppressed: 0 bytes in 0 blocks >> (Bitcoin XT commit fe446d) >> >> I haven't found anything in there yet that I think would produce the >> multi-GB memory usage after running for a few days, but I could be missing >> it. Email me if you want the full log. >> >> I did not try running getblocktemplate while valgrind was running. I'll >> have to try that. I also have not let valgrind run for more than an hour. >> >> >> >> P.S.: Sorry for all the cross-post confusion and consequent flamewar >> fallout. While it's probably too late for this thread, I'll make sure to >> post in a manner that keeps the threads clearly separate in the future >> (e.g. different subject lines). >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >> --e89a8ff1ccc2c32ff605227c9a7d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I should also mention that this is definitely not an attac= k coming from connected nodes.=C2=A0 My node experiencing the issue is only= connected to 3 other nodes, all of which I control (via connect=3D).
<= br>
--Adam

On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <admin= @multipool.us> wrote:

My nodes are continuously running getblocktemplate and getinfo, a= nd I also suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory= usage.

On Oct 18, 2015 8:59 AM, = "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists.linuxfoun= dation.org> wrote:
On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> w= rote:
This is *most likely* the mempool, but= is just not reported correctly.

I did = some testing with PR #6410's better mempool reporting. The improved rep= orting suggests that actual in-memory usage ("usage":) by CTxMemP= ool is about 2.5x to 3x higher than the serialized transaction sizes ("= ;bytes":). The excess memory usage that I'm seeing is on the order= of 100x higher than the mempool "bytes": value. As such, I think= it's unlikely that this is the mempool, or at least not normal/correct= mempool behavior.=C2=A0

Another user (admin@multipool.us) report= ed 35 GB of RSS usage. I'm guessing his bitcoind has been running longe= r than any of mine. His server definitely has more RAM. I don't know wh= ich email list he is subscribed to (probably XT), so I'm sharing it wit= h both lists to make sure you're all aware of how big an issue this can= be.

In the meantime you can mitigate th= e mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/relea= se-notes.md#transaction-flooding

I have = mintxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough = to exclude essentially all of the of the 14700-14800 byte flood transaction= s. My nodes' mempools only contain about one or two blocks' worth o= f transactions. So I don't think this is correct either.

=


Some additional notes on this issu= e:

1. I think it's related to CreateNewBlock()= and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overn= ight with no mining connected to it, and (IIRC -- my memory is fuzzy) when = I woke up it was using around 400 MB of RSS and the mempool was at around &= quot;bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getbloc= ktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getbl= ocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 M= B. An hour after that with more getblocktemplates, and now RSS is at 1350 M= B. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing = "usage" around 25MB or less.

I'll do= some more testing with this and see if I can make it repeatable, and recor= d the results more carefully. Expect a follow-up from me in a day or two.

2. valgrind did not show anything super promising. = It did report this:

=3D=3D6880=3D=3D LEAK SUMMARY:
= =3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los= t: 0 bytes in 0 blocks
=3D=3D6880=3D= =3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,= 552 bytes in 39 blocks
=3D=3D6880=3D= =3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks
=
(Bitcoin Core commit d78a880)

and this:
=
=3D=3D6778=3D=3D LEAK SUMMARY:
=
=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely = lost: 0 bytes in 0 blocks
=3D=3D6778= =3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32= 0 bytes in 1 blocks
=3D=3D6778=3D=3D= =C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks
=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse= d: 0 bytes in 0 blocks
(Bitcoin XT commit=C2=A0fe446d)

I haven't found anything in there yet that I think= would produce the multi-GB memory usage after running for a few days, but = I could be missing it. Email me if you want the full log.

I did not try running getblocktemplate while valgrind was running. = I'll have to try that. I also have not let valgrind run for more than a= n hour.=C2=A0



P.S.: = Sorry for all the cross-post confusion and consequent flamewar fallout. Whi= le it's probably too late for this thread, I'll make sure to post i= n a manner that keeps the threads clearly separate in the future (e.g. diff= erent subject lines).

_________= ______________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev


--e89a8ff1ccc2c32ff605227c9a7d--