[bitcoin-dev] Memory leaks?

[bitcoin-dev] Memory leaks?

[Jonathan Toomim (Toomim Bros)]

[-- Attachment #1.1: Type: text/plain, Size: 2112 bytes --]

I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.

XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
{
    "size" : 1896,
    "bytes" : 37341328
}

[total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]

37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot of unconfirmed txns floating around the network, memory usage is around 600 MB, so this is quite unusual.

After restarting the process and letting it run for a few minutes, I get:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
[###] [XXXX]     20   0 1402M  317M 49836 S  1.0  8.2  0:41.71 ./bitcoind -daemon

XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
{
    "size" : 1072,
    "bytes" : 670000
}

0.67 MB mempool, 317 MB RAM usage. Much more reasonable.


Here's another node I'm running that has been online longer, before restarting:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
[###] [XXXX]     20   0 4961M 3540M 11080 S  2.8 45.3  8h20:11 bin/bitcoind -daemon

XXXX@feather:~$ bin/bitcoin-cli getmempoolinfo
{
    "size" : 3045,
    "bytes" : 39656126
}

39 MB mempool, 3540 MB total memory usage. After restarting bitcoind, I see:

[XXXX]@feather:~$ bin/bitcoin-cli stop
Bitcoin server stopping
[XXXX]@feather:~$ bin/bitcoind -daemon
Bitcoin server starting
[XXXX]@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo
{
    "size" : 39,
    "bytes" : 47037
}


  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
[###] [XXXX]     20   0 1640M  247M 67960 S  0.0  3.2  0:05.17 bin/bitcoind -daemon




Does anybody have any guesses where we might be leaking memory, or what is using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a leak in the minrelaytxfee code path? Has anyone else seen something similar?

This issue appears to happen both with Bitcoin Core 0.10.1 and with Bitcoin XT 0.11B.

[-- Attachment #1.2: Type: text/html, Size: 6926 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[odinn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

You should compare this to having set up a node on a completely clean
computer.  It would also help to know what operating system(s) you are
using for both the oldie and the freshie.

Also, dump your XT, is poo.  Then try again, look at Core nodes on
your oldie and freshie.  Watch them for a bit.

Cheers,

O

Jonathan Toomim (Toomim Bros) via bitcoin-dev:
> I just noticed that several of my running bitcoind processes were
> using around 3+ GB of RAM, even though the mempool itself seemed to
> be under control.
> 
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo { "size" : 1896, 
> "bytes" : 37341328 }
> 
> [total memory usage not shown -- I restarted bitcoind as soon as I
> noticed, and didn't copy it down from top]
> 
> 37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot
> of unconfirmed txns floating around the network, memory usage is
> around 600 MB, so this is quite unusual.
> 
> After restarting the process and letting it run for a few minutes,
> I get:
> 
> PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+
> Command [###] [XXXX]     20   0 1402M  317M 49836 S  1.0  8.2
> 0:41.71 ./bitcoind -daemon
> 
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo { "size" : 1072, 
> "bytes" : 670000 }
> 
> 0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
> 
> 
> Here's another node I'm running that has been online longer, before
> restarting:
> 
> PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+
> Command [###] [XXXX]     20   0 4961M 3540M 11080 S  2.8 45.3
> 8h20:11 bin/bitcoind -daemon
> 
> XXXX@feather:~$ bin/bitcoin-cli getmempoolinfo { "size" : 3045, 
> "bytes" : 39656126 }
> 
> 39 MB mempool, 3540 MB total memory usage. After restarting
> bitcoind, I see:
> 
> [XXXX]@feather:~$ bin/bitcoin-cli stop Bitcoin server stopping 
> [XXXX]@feather:~$ bin/bitcoind -daemon Bitcoin server starting 
> [XXXX]@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo { "size"
> : 39, "bytes" : 47037 }
> 
> 
> PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+
> Command [###] [XXXX]     20   0 1640M  247M 67960 S  0.0  3.2
> 0:05.17 bin/bitcoind -daemon
> 
> 
> 
> 
> Does anybody have any guesses where we might be leaking memory, or
> what is using the additional 2.4 GB? I've been using
> minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a leak
> in the minrelaytxfee code path? Has anyone else seen something
> similar?
> 
> This issue appears to happen both with Bitcoin Core 0.10.1 and with
> Bitcoin XT 0.11B.
> 
> 
> 
> _______________________________________________ bitcoin-dev mailing
> list bitcoin-dev@lists.linuxfoundation.org 
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

- -- 
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJWHYqAAAoJEGxwq/inSG8Cp8YIAJa9xWA+rNY9ZOTWjOEZGc7Q
IpXqpIyZprhEog6By/rQ7Te+xdUuaIZrMPESEoQIdMlDjRu7V2CWKGr1LbbDf2v9
A6nMhE19tazDSMAcqHlcuWRfpex3C5QD93Oo7h0QvioNLc8cseNsYzqvOW40vIFL
SPJOfor6IFLEi6/0t7OBhVyaXZdhI7XD1IDxeD67IOafCDwHgixFZWS4aCpz4axj
i4B8DNsVgDdeYI2STBiqnL9Sopdnc1q2CwC1ENszR+sCXwIB9vdPOtIjhtWGk1gi
f+/I8IUP/jn2xIjAGixjEePCIFLFV3kI+B6akH8+TsXB4G9C3zHEHTsfCrmLblg=
=VMV3
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim (Toomim Bros)]

[-- Attachment #1.1: Type: text/plain, Size: 1539 bytes --]


On Oct 13, 2015, at 3:49 PM, odinn <odinn.cyberguerrilla@riseup.net> wrote:

> Signed PGP part
> It would also help to know what operating system(s) you are
> using for both the oldie and the freshie.

Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) x86_64 GNU/Linux
Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux

This excessive memory consumption was seen on 3 machines, all of which run Debian. All three machines run p2pool as well as bitcoind. Two run XT, one runs Core.

> 
> You should compare this to having set up a node on a completely clean
> computer.


I can't afford to do that. All of the servers I have are being used for something. Also, I'm not sure what it is you're trying to test for with that suggestion. The numbers I'm reporting are for bitcoind's resident set, not for the whole server's memory usage. I don't see how other processes running on the same machine are relevant unless you are suggesting that RPC calls (e.g. getblocktemplate) might be somehow responsible.

> 
> Also, dump your XT, is poo.


Not relevant. I addressed this message to both the Core and XT lists because the issue appears to affect both forks. Let's keep blocksize and governance debates to their own threads, please.

Repeating request: Has anyone else seen something similar? Can you report your mempool size and total bitcoind resident set size for your running full nodes?

[-- Attachment #1.2: Type: text/html, Size: 3401 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Dave Scotese]

[-- Attachment #1: Type: text/plain, Size: 2592 bytes --]

{
"size" : 1085,
"bytes" : 16151768
}

It has been running about a day.  I'll report tomorrow too.  This is a
Windows 8.1 box.

16 million divided by 1085 transactions is almost 15Kb per transaction =
unlikely, right?

On Tue, Oct 13, 2015 at 4:14 PM, Jonathan Toomim (Toomim Bros) <j@toom.im>
wrote:

>
> On Oct 13, 2015, at 3:49 PM, odinn <odinn.cyberguerrilla@riseup.net>
> wrote:
>
> Signed PGP part
> It would also help to know what operating system(s) you are
> using for both the oldie and the freshie.
>
>
> Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3
> (2015-08-04) x86_64 GNU/Linux
> Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
> Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux
>
> This excessive memory consumption was seen on 3 machines, all of which run
> Debian. All three machines run p2pool as well as bitcoind. Two run XT, one
> runs Core.
>
> You should compare this to having set up a node on a completely clean
> computer.
>
>
> I can't afford to do that. All of the servers I have are being used for
> something. Also, I'm not sure what it is you're trying to test for with
> that suggestion. The numbers I'm reporting are for bitcoind's resident set,
> not for the whole server's memory usage. I don't see how other processes
> running on the same machine are relevant unless you are suggesting that RPC
> calls (e.g. getblocktemplate) might be somehow responsible.
>
> Also, dump your XT, is poo.
>
>
> Not relevant. I addressed this message to both the Core and XT lists
> because the issue appears to affect both forks. Let's keep blocksize and
> governance debates to their own threads, please.
>
> Repeating request: Has anyone else seen something similar? Can you report
> your mempool size and total bitcoind resident set size for your running
> full nodes?
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy <http://www.litmocracy.com> and Meme Racing
<http://www.memeracing.net> (in alpha).
I'm the webmaster for The Voluntaryist <http://www.voluntaryist.com> which
now accepts Bitcoin.
I also code for The Dollar Vigilante <http://dollarvigilante.com/>.
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto

[-- Attachment #2: Type: text/html, Size: 5178 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim (Toomim Bros)]

[-- Attachment #1.1: Type: text/plain, Size: 834 bytes --]

> 16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?


The recent spam was about 15 kB per transaction, so that part sounds right.

The anomalous thing that I saw was that the total bitcoind process usage was about 50-100x higher than I would have expected if the mempool was the main determinant of memory usage scaling. Can you tell me how much memory Task Manager is reporting your bitcoin process as using both today and tomorrow?

On Oct 13, 2015, at 4:52 PM, Dave Scotese via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

> 
> 
> {
> "size" : 1085,
> "bytes" : 16151768
> }
> It has been running about a day.  I'll report tomorrow too.  This is a Windows 8.1 box.
> 16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?


[-- Attachment #1.2: Type: text/html, Size: 1954 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Dave Scotese]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

It was about 360MB (30 minutes ago?), but is now about 460MB.  I'm sure it
won't keep going up that fast.
{
"size" : 3413,
"bytes" : 41892350
}


On Tue, Oct 13, 2015 at 5:08 PM, Jonathan Toomim (Toomim Bros) <j@toom.im>
wrote:

> 16 million divided by 1085 transactions is almost 15Kb per transaction =
> unlikely, right?
>
>
> The recent spam was about 15 kB per transaction, so that part sounds right.
>
> The anomalous thing that I saw was that the total bitcoind process usage
> was about 50-100x higher than I would have expected if the mempool was the
> main determinant of memory usage scaling. Can you tell me how much memory
> Task Manager is reporting your bitcoin process as using both today and
> tomorrow?
>
> On Oct 13, 2015, at 4:52 PM, Dave Scotese via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>
> {
> "size" : 1085,
> "bytes" : 16151768
> }
> It has been running about a day.  I'll report tomorrow too.  This is a
> Windows 8.1 box.
> 16 million divided by 1085 transactions is almost 15Kb per transaction =
> unlikely, right?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy <http://www.litmocracy.com> and Meme Racing
<http://www.memeracing.net> (in alpha).
I'm the webmaster for The Voluntaryist <http://www.voluntaryist.com> which
now accepts Bitcoin.
I also code for The Dollar Vigilante <http://dollarvigilante.com/>.
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto

[-- Attachment #2: Type: text/html, Size: 3593 bytes --]

Re: [bitcoin-dev] Memory leaks?

[odinn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

You wanted advice... you got it

Jonathan Toomim (Toomim Bros):
> 
> On Oct 13, 2015, at 3:49 PM, odinn
> <odinn.cyberguerrilla@riseup.net> wrote:
> 
>> Signed PGP part It would also help to know what operating
>> system(s) you are using for both the oldie and the freshie.
> 
> Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3
> (2015-08-04) x86_64 GNU/Linux Linux server 3.2.0-4-amd64 #1 SMP
> Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux Linux prime 3.2.0-4-amd64
> #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux
> 
> This excessive memory consumption was seen on 3 machines, all of
> which run Debian. All three machines run p2pool as well as
> bitcoind. Two run XT, one runs Core.
> 
>> 
>> You should compare this to having set up a node on a completely
>> clean computer.
> 
> 
> I can't afford to do that. All of the servers I have are being used
> for something. Also, I'm not sure what it is you're trying to test
> for with that suggestion. The numbers I'm reporting are for
> bitcoind's resident set, not for the whole server's memory usage. I
> don't see how other processes running on the same machine are
> relevant unless you are suggesting that RPC calls (e.g.
> getblocktemplate) might be somehow responsible.
> 
>> 
>> Also, dump your XT, is poo.
> 
> 
> Not relevant. I addressed this message to both the Core and XT
> lists because the issue appears to affect both forks. Let's keep
> blocksize and governance debates to their own threads, please.
> 
> Repeating request: Has anyone else seen something similar? Can you
> report your mempool size and total bitcoind resident set size for
> your running full nodes?
> 

- -- 
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJWHZrZAAoJEGxwq/inSG8CZzQIAKsqKs//Wydv60nXgy5AWAPU
qZ9HuyyWXDKljxzv/Ky5jS7o7B8Ivhnt6zWvkpMTF/R9MLpGrS9jBxXZjHF//ET0
L+eoVrmxwt+rgSjIPSGU/ftF8Jnh1sELecR8FMuCaFR87xraR/7FsJF/233RLWFg
+scNiFEgttyizFNgSq2r1/N3G5e603qXfh0+reaabDX3E+8+PKyUqVaG5E+TUEW0
NIkqi7MuEYd+/Q0SGAYyY/j2BQnebsTB2TbupE/soJkAYqYbCQR8TtrctmwLXTL0
GN+WyWwLYpMio3+7a6oQJ67TBcFxCVmF81zxKM1VIoT0u39VVWeYD1YfxEYFN9Y=
=a6kH
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] Memory leaks?

[Pavel Janík]

> I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.
> 
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
>     "size" : 1896,
>     "bytes" : 37341328
> }
> 
> [total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]

The newer versions contain much more accurate reporting of memorypool usage that reflect the real memory usage:

{
  "size": 58536,
  "bytes": 770044257,
  "usage": 1652554832
}

See PR #6410 (Implement accurate memory accounting for mempool).
--  
Pavel Janík

Re: [bitcoin-dev] Memory leaks?

[Tom Zander]

On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim  via bitcoin-dev wrote:
> Does anybody have any guesses where we might be leaking memory, or what is
> using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or
> similar on my nodes. Maybe there's a leak in the minrelaytxfee code path?
> Has anyone else seen something similar?

I suggest running it in valgrind with --leak-check=full for 10 minutes.

  valgrind --leak-check=full src/bitcoind 2>&1 | tee out

This at least will show you any memory leaks at exit.
Naturally, the leaks you observe may just be design issues where cache can 
grow to much and when the cache is cleaned on shutdown you won't see it in the 
valgrind output.

Re: [bitcoin-dev] Memory leaks?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

Leaks are not the only explanation possible. Caches and fragmentation can
also give this sort of effect. Unfortunately the tools to debug this aren't
great. You could try a build with tcmalloc and use it to investigate heap
stats.

Odinn, trolling like a 3 year old will get you swiftly banned. Last warning.
On 14 Oct 2015 9:58 am, "Tom Zander" <tomz@freedommail.ch> wrote:

> On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim  via bitcoin-dev wrote:
> > Does anybody have any guesses where we might be leaking memory, or what
> is
> > using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or
> > similar on my nodes. Maybe there's a leak in the minrelaytxfee code path?
> > Has anyone else seen something similar?
>
> I suggest running it in valgrind with --leak-check=full for 10 minutes.
>
>   valgrind --leak-check=full src/bitcoind 2>&1 | tee out
>
> This at least will show you any memory leaks at exit.
> Naturally, the leaks you observe may just be design issues where cache can
> grow to much and when the cache is cleaned on shutdown you won't see it in
> the
> valgrind output.
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[-- Attachment #2: Type: text/html, Size: 1902 bytes --]

Re: [bitcoin-dev] Memory leaks?

[odinn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

For the record, Mr. Hearn, you do not own this list.  I submit to you
that you have very little to say on this matter at this stage and your
idle threats to "ban people" based on their preferences, suggestions,
or characterizations of your chosen software project are at best very
silly.

I am not and never was on bitcoin-xt@googlegroups.com and thus have
clipped that from the reply-to.

Mike Hearn via bitcoin-dev:
> Leaks are not the only explanation possible. Caches and
> fragmentation can also give this sort of effect. Unfortunately the
> tools to debug this aren't great. You could try a build with
> tcmalloc and use it to investigate heap stats.
> 
> Odinn, trolling like a 3 year old will get you swiftly banned. Last
> warning. On 14 Oct 2015 9:58 am, "Tom Zander" <tomz@freedommail.ch>
> wrote:
> 
>> On Tuesday 13 Oct 2015 14:56:08 Jonathan Toomim  via bitcoin-dev
>> wrote:
>>> Does anybody have any guesses where we might be leaking memory,
>>> or what
>> is
>>> using the additional 2.4 GB? I've been using
>>> minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a
>>> leak in the minrelaytxfee code path? Has anyone else seen
>>> something similar?
>> 
>> I suggest running it in valgrind with --leak-check=full for 10
>> minutes.
>> 
>> valgrind --leak-check=full src/bitcoind 2>&1 | tee out
>> 
>> This at least will show you any memory leaks at exit. Naturally,
>> the leaks you observe may just be design issues where cache can 
>> grow to much and when the cache is cleaned on shutdown you won't
>> see it in the valgrind output.
>> 
>> -- You received this message because you are subscribed to the
>> Google Groups "bitcoin-xt" group. To unsubscribe from this group
>> and stop receiving emails from it, send an email to
>> bitcoin-xt+unsubscribe@googlegroups.com. For more options, visit
>> https://groups.google.com/d/optout.
>> 
> 
> 
> 
> _______________________________________________ bitcoin-dev mailing
> list bitcoin-dev@lists.linuxfoundation.org 
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 

- -- 
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJWHnKFAAoJEGxwq/inSG8CoN8H/i4m748xSMkXLgsOO2nqghXr
K3EI98BucU5XF4M/0qW0EjN1PHdkl+EjtUt0POT3mG3hl66PaoA04nMgDrND7V+w
sEXICchAVNx5+AleT65U60iibASZZIlZaXmOtdtCgz7GulmMfsfnNV2IRRvsSO1A
Nl0PuEqPW1/rsJDA58tDb8y2ltMEo5Zi2AYDMvD/AfSuNBqdHM/2IrWSPUwDB7NN
TLq5WXyW5mv7qywIu3/8jk0za6RN4gc1DmpIJHjm4bO+4FoF0oytcaOg5X8uOC1B
pOxhvEM2fTjziXaBJVha/6lrGxfi8/mdLBE68hjB3Q6/KDF9VrugdG0JK0iuDW8=
=YqPh
-----END PGP SIGNATURE-----

Re: [bitcoin-dev] Memory leaks?

[Wladimir J. van der Laan]

On Tue, Oct 13, 2015 at 02:56:08PM -0700, Jonathan Toomim (Toomim Bros) via bitcoin-dev wrote:
> I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.
> 
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
>     "size" : 1896,
>     "bytes" : 37341328
> }
> 
> [total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]

This is *most likely* the mempool, but is just not reported correctly.

In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding

This will be the default in 0.11.1 and 0.10.3. 0.12.0 will have better mempool management, as well as more precise reporting (as paveljanik already mentions.)

Wladimir

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim]

[-- Attachment #1.1: Type: text/plain, Size: 3537 bytes --]

On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> wrote:
> This is *most likely* the mempool, but is just not reported correctly.

I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.

Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm guessing his bitcoind has been running longer than any of mine. His server definitely has more RAM. I don't know which email list he is subscribed to (probably XT), so I'm sharing it with both lists to make sure you're all aware of how big an issue this can be.

> In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding

I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.



Some additional notes on this issue:

1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.

I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.

2. valgrind did not show anything super promising. It did report this:

==6880== LEAK SUMMARY:
==6880==    definitely lost: 0 bytes in 0 blocks
==6880==    indirectly lost: 0 bytes in 0 blocks
==6880==      possibly lost: 288 bytes in 1 blocks
==6880==    still reachable: 10,552 bytes in 39 blocks
==6880==         suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)

and this:
==6778== LEAK SUMMARY:
==6778==    definitely lost: 0 bytes in 0 blocks
==6778==    indirectly lost: 0 bytes in 0 blocks
==6778==      possibly lost: 320 bytes in 1 blocks
==6778==    still reachable: 10,080 bytes in 32 blocks
==6778==         suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)

I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.

I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.



P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).

[-- Attachment #1.2: Type: text/html, Size: 5893 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Multipool Admin]

[-- Attachment #1: Type: text/plain, Size: 4124 bytes --]

My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
> wrote:
>
> This is *most likely* the mempool, but is just not reported correctly.
>
>
> I did some testing with PR #6410's better mempool reporting. The improved
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
> The excess memory usage that I'm seeing is on the order of 100x higher than
> the mempool "bytes": value. As such, I think it's unlikely that this is the
> mempool, or at least not normal/correct mempool behavior.
>
> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
> guessing his bitcoind has been running longer than any of mine. His server
> definitely has more RAM. I don't know which email list he is subscribed to
> (probably XT), so I'm sharing it with both lists to make sure you're all
> aware of how big an issue this can be.
>
> In the meantime you can mitigate the mempool growth by setting
> `-mintxfee`, see
>
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
>
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
> enough to exclude essentially all of the of the 14700-14800 byte flood
> transactions. My nodes' mempools only contain about one or two blocks'
> worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
> Core bitcoind process (commit d78a880) overnight with no mining connected
> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
> getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable,
> and record the results more carefully. Expect a follow-up from me in a day
> or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880==    definitely lost: 0 bytes in 0 blocks
> ==6880==    indirectly lost: 0 bytes in 0 blocks
> ==6880==      possibly lost: 288 bytes in 1 blocks
> ==6880==    still reachable: 10,552 bytes in 39 blocks
> ==6880==         suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778==    definitely lost: 0 bytes in 0 blocks
> ==6778==    indirectly lost: 0 bytes in 0 blocks
> ==6778==      possibly lost: 320 bytes in 1 blocks
> ==6778==    still reachable: 10,080 bytes in 32 blocks
> ==6778==         suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the
> multi-GB memory usage after running for a few days, but I could be missing
> it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll
> have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar
> fallout. While it's probably too late for this thread, I'll make sure to
> post in a manner that keeps the threads clearly separate in the future
> (e.g. different subject lines).
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 6690 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Multipool Admin]

[-- Attachment #1: Type: text/plain, Size: 4507 bytes --]

I should also mention that this is definitely not an attack coming from
connected nodes.  My node experiencing the issue is only connected to 3
other nodes, all of which I control (via connect=).

--Adam

On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <admin@multipool.us>
wrote:

> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880==    definitely lost: 0 bytes in 0 blocks
>> ==6880==    indirectly lost: 0 bytes in 0 blocks
>> ==6880==      possibly lost: 288 bytes in 1 blocks
>> ==6880==    still reachable: 10,552 bytes in 39 blocks
>> ==6880==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==    definitely lost: 0 bytes in 0 blocks
>> ==6778==    indirectly lost: 0 bytes in 0 blocks
>> ==6778==      possibly lost: 320 bytes in 1 blocks
>> ==6778==    still reachable: 10,080 bytes in 32 blocks
>> ==6778==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>

[-- Attachment #2: Type: text/html, Size: 7412 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 4675 bytes --]

OK, then running under Valgrind whilst sending gbt RPCs would be the next
step.

On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> wrote:

> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880==    definitely lost: 0 bytes in 0 blocks
>> ==6880==    indirectly lost: 0 bytes in 0 blocks
>> ==6880==      possibly lost: 288 bytes in 1 blocks
>> ==6880==    still reachable: 10,552 bytes in 39 blocks
>> ==6880==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==    definitely lost: 0 bytes in 0 blocks
>> ==6778==    indirectly lost: 0 bytes in 0 blocks
>> ==6778==      possibly lost: 320 bytes in 1 blocks
>> ==6778==    still reachable: 10,080 bytes in 32 blocks
>> ==6778==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[-- Attachment #2: Type: text/html, Size: 7777 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim]

[-- Attachment #1.1: Type: text/plain, Size: 5332 bytes --]

I did that Sunday twice. I'll report the results soon. Short version is that it looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, which is declared as a static pointer. Not exactly the multi-GB leak I'm looking for, but possibly related.

I've also got two bitcoind processes running on the same machine that I started at the same time, running on different ports, all with the same settings, but one of which is serving getblocktemplate every 5-6 seconds and the other is not, while logging RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, and for the serving node 1997 MB. Graphs coming soon.


On Oct 20, 2015, at 3:12 AM, Mike Hearn <hearn@vinumeris.com> wrote:

> OK, then running under Valgrind whilst sending gbt RPCs would be the next step.
> 
> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> wrote:
> My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server.
> 
> The instance only takes a few hours to get up to that memory usage.
> 
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists.linuxfoundation.org> wrote:
> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> wrote:
>> This is *most likely* the mempool, but is just not reported correctly.
> 
> I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.
> 
> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm guessing his bitcoind has been running longer than any of mine. His server definitely has more RAM. I don't know which email list he is subscribed to (probably XT), so I'm sharing it with both lists to make sure you're all aware of how big an issue this can be.
> 
>> In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
> 
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.
> 
> 
> 
> Some additional notes on this issue:
> 
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.
> 
> I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.
> 
> 2. valgrind did not show anything super promising. It did report this:
> 
> ==6880== LEAK SUMMARY:
> ==6880==    definitely lost: 0 bytes in 0 blocks
> ==6880==    indirectly lost: 0 bytes in 0 blocks
> ==6880==      possibly lost: 288 bytes in 1 blocks
> ==6880==    still reachable: 10,552 bytes in 39 blocks
> ==6880==         suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
> 
> and this:
> ==6778== LEAK SUMMARY:
> ==6778==    definitely lost: 0 bytes in 0 blocks
> ==6778==    indirectly lost: 0 bytes in 0 blocks
> ==6778==      possibly lost: 320 bytes in 1 blocks
> ==6778==    still reachable: 10,080 bytes in 32 blocks
> ==6778==         suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
> 
> I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.
> 
> I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.
> 
> 
> 
> P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> 
> --
> You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 


[-- Attachment #1.2: Type: text/html, Size: 8817 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim]

[-- Attachment #1.1: Type: text/plain, Size: 9305 bytes --]

More notes:

1. I ran a side-by-side comparison with two bitcoind processes (Core, same recent git commit as before) on the same computer with the same settings running on different ports. With both processes, I logged RSS (via /proc/$pid/status) every 6 seconds. With one of those processes, I also ran bitcoin-cli getblocktemplate > /dev/null every 6 seconds. I let that run for about 30 hours. A graph and links to the CSVs of raw data are below. Results seem pretty clear: the getblocktemplate RPC is implicated in this issue.


http://toom.im/files/memlog8518.csv
http://toom.im/files/memlog-nogbt-8503.csv
http://toom.im/files/bitcoind_memory_usage_gbt.png


2. I ran valgrind twice, for about 6 hours each, on bitcoind while hitting it with getblocktemplate every 6 hours. Full valgrind output can be found at these two URLs:

http://toom.im/files/valgrind-gbt-1.log
http://toom.im/files/valgrind-gbt-2.log

The summary:

==4064== LEAK SUMMARY:
==4064==    definitely lost: 0 bytes in 0 blocks
==4064==    indirectly lost: 0 bytes in 0 blocks
==4064==      possibly lost: 288 bytes in 1 blocks
==4064==    still reachable: 527,594 bytes in 4,367 blocks
==4064==         suppressed: 0 bytes in 0 blocks
The main components of that still reachable section seem to just be one output of CreateNewBlock that's cached in case another getblocktemplate request is received before any new transactions come in:

==4064== 98,304 bytes in 1 blocks are still reachable in loss record 39 of 40
==4064==    at 0x4C29180: operator new(unsigned long) (vg_replace_malloc.c:324)
==4064==    by 0x28EAA1: __gnu_cxx::new_allocator<CTransaction>::allocate(unsigned long, void const*) (new_allocator.h:104)
==4064==    by 0x27EE44: __gnu_cxx::__alloc_traits<std::allocator<CTransaction> >::allocate(std::allocator<CTransaction>&, unsigned long) (alloc_traits.h:182)
==4064==    by 0x26DFB0: std::_Vector_base<CTransaction, std::allocator<CTransaction> >::_M_allocate(unsigned long) (stl_vector.h:170)
==4064==    by 0x2D5BDE: std::vector<CTransaction, std::allocator<CTransaction> >::_M_insert_aux(__gnu_cxx::__normal_iterator<CTransaction*, std::vector<CTransaction, std::allocator<CTransaction> > >, CTransaction const&) (vector.tcc:353)
==4064==    by 0x2D3FF8: std::vector<CTransaction, std::allocator<CTransaction> >::push_back(CTransaction const&) (stl_vector.h:925)
==4064==    by 0x2D113E: CreateNewBlock(CScript const&) (miner.cpp:298)
==4064==    by 0x442D78: getblocktemplate(UniValue const&, bool) (rpcmining.cpp:513)
==4064==    by 0x390CEB: CRPCTable::execute(std::string const&, UniValue const&) const (rpcserver.cpp:526)
==4064==    by 0x41C5AB: HTTPReq_JSONRPC(HTTPRequest*, std::string const&) (httprpc.cpp:125)
==4064==    by 0x3559BD: boost::detail::function::void_function_invoker2<bool (*)(HTTPRequest*, std::string const&), void, HTTPRequest*, std::string const&>::invoke(boost::detail::function::function_buffer&, HTTPRequest*, std::string const&) (function_template.hpp:112)
==4064==    by 0x422520: boost::function2<void, HTTPRequest*, std::string const&>::operator()(HTTPRequest*, std::string const&) const (function_template.hpp:767)

There are a few other similar loss records (mostly referring to pblock or pblocktemplate in CreateNewBlock(...), but I see nothing that can explain the multi-GB memory consumption.

3. One user on the bitcointalk p2pool thread (https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791) claimed that he had this memory usage issue on Linux, but not on Mac OS X, under a GBT workload in both situations. If this is true, that would suggest this might be a fragmentation issue due to poor memory allocation. The other likely hypothesis is bloated caches. Looking into those two possibilities will be my next steps.



On Oct 20, 2015, at 5:39 AM, Jonathan Toomim <j@toom.im> wrote:

> I did that Sunday twice. I'll report the results soon. Short version is that it looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, which is declared as a static pointer. Not exactly the multi-GB leak I'm looking for, but possibly related.
> 
> I've also got two bitcoind processes running on the same machine that I started at the same time, running on different ports, all with the same settings, but one of which is serving getblocktemplate every 5-6 seconds and the other is not, while logging RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, and for the serving node 1997 MB. Graphs coming soon.
> 
> 
> On Oct 20, 2015, at 3:12 AM, Mike Hearn <hearn@vinumeris.com> wrote:
> 
>> OK, then running under Valgrind whilst sending gbt RPCs would be the next step.
>> 
>> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> wrote:
>> My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server.
>> 
>> The instance only takes a few hours to get up to that memory usage.
>> 
>> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> wrote:
>>> This is *most likely* the mempool, but is just not reported correctly.
>> 
>> I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.
>> 
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm guessing his bitcoind has been running longer than any of mine. His server definitely has more RAM. I don't know which email list he is subscribed to (probably XT), so I'm sharing it with both lists to make sure you're all aware of how big an issue this can be.
>> 
>>> In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
>>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>> 
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.
>> 
>> 
>> 
>> Some additional notes on this issue:
>> 
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.
>> 
>> I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.
>> 
>> 2. valgrind did not show anything super promising. It did report this:
>> 
>> ==6880== LEAK SUMMARY:
>> ==6880==    definitely lost: 0 bytes in 0 blocks
>> ==6880==    indirectly lost: 0 bytes in 0 blocks
>> ==6880==      possibly lost: 288 bytes in 1 blocks
>> ==6880==    still reachable: 10,552 bytes in 39 blocks
>> ==6880==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>> 
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==    definitely lost: 0 bytes in 0 blocks
>> ==6778==    indirectly lost: 0 bytes in 0 blocks
>> ==6778==      possibly lost: 320 bytes in 1 blocks
>> ==6778==    still reachable: 10,080 bytes in 32 blocks
>> ==6778==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>> 
>> I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.
>> 
>> I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.
>> 
>> 
>> 
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).
>> 
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> 
>> 
>> --
>> You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>> 
> 


[-- Attachment #1.2: Type: text/html, Size: 15807 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Tom Zander]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
>  claimed that he had this memory usage issue on Linux, but not on Mac OS X,
> under a GBT workload in both situations. If this is true, that would
> suggest this might be a fragmentation issue due to poor memory allocation.

Please make sure you measure your memory usage correctly on Linux, it is 
notoriously easy to get misleading info from tools like top.

I use this one on Linux. 

$cat ~/bin/showmemusage 
#!/bin/sh
if test -z "$1"; then
    echo "need a pid"
    exit
fi

mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

The method I was using was essentially

	grep VmRSS /proc/$pid/status

Comparing these two methods, I get

Your method (PSS):
	2408313
My method (RSS):
	VmRSS:	 2410396 kB

On Oct 21, 2015, at 12:29 AM, Tom Zander <tomz@freedommail.ch> wrote:

> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
> Please make sure you measure your memory usage correctly on Linux, it is
> notoriously easy to get misleading info from tools like top.
> 
> I use this one on Linux.
> 
> $cat ~/bin/showmemusage
> #!/bin/sh
> if test -z "$1"; then
>    echo "need a pid"
>    exit
> fi
> 
> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
> sed 's#^#+#' ) | bc`
> echo "$mem KB"



[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Multipool Admin]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

This is a real issue.  The bitcoind process is getting killed every few
days when it reaches around 55gb of usage on my server.
On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
> >  claimed that he had this memory usage issue on Linux, but not on Mac OS
> X,
> > under a GBT workload in both situations. If this is true, that would
> > suggest this might be a fragmentation issue due to poor memory
> allocation.
>
> Please make sure you measure your memory usage correctly on Linux, it is
> notoriously easy to get misleading info from tools like top.
>
> I use this one on Linux.
>
> $cat ~/bin/showmemusage
> #!/bin/sh
> if test -z "$1"; then
>     echo "need a pid"
>     exit
> fi
>
> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
> sed 's#^#+#' ) | bc`
> echo "$mem KB"
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 1662 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Jonathan Toomim]

[-- Attachment #1.1: Type: text/plain, Size: 1633 bytes --]

You may want to add a cron job to restart bitcoind every day or two as a damage control mechanism until we figure this out.

On Oct 22, 2015, at 9:06 AM, Multipool Admin <admin@multipool.us> wrote:

> This is a real issue.  The bitcoind process is getting killed every few days when it reaches around 55gb of usage on my server.
> 
> On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev" <bitcoin-dev@lists.linuxfoundation.org> wrote:
> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
> >  claimed that he had this memory usage issue on Linux, but not on Mac OS X,
> > under a GBT workload in both situations. If this is true, that would
> > suggest this might be a fragmentation issue due to poor memory allocation.
> 
> Please make sure you measure your memory usage correctly on Linux, it is
> notoriously easy to get misleading info from tools like top.
> 
> I use this one on Linux.
> 
> $cat ~/bin/showmemusage
> #!/bin/sh
> if test -z "$1"; then
>     echo "need a pid"
>     exit
> fi
> 
> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
> sed 's#^#+#' ) | bc`
> echo "$mem KB"
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> 
> 
> --
> You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


[-- Attachment #1.2: Type: text/html, Size: 2622 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] Memory leaks?

[Btc Drak]

I think this thread has gotten to the stage where it should be moved
to an issue on Github and not continue to CC the bitcoin-dev list (or
any other list tbh).

On Thu, Oct 22, 2015 at 5:27 PM, Jonathan Toomim via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> You may want to add a cron job to restart bitcoind every day or two as a
> damage control mechanism until we figure this out.
>
> On Oct 22, 2015, at 9:06 AM, Multipool Admin <admin@multipool.us> wrote:
>
> This is a real issue.  The bitcoind process is getting killed every few days
> when it reaches around 55gb of usage on my server.
>
> On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev"
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> On Tuesday 20 Oct 2015 20:01:16 Jonathan Toomim wrote:
>> >  claimed that he had this memory usage issue on Linux, but not on Mac OS
>> > X,
>> > under a GBT workload in both situations. If this is true, that would
>> > suggest this might be a fragmentation issue due to poor memory
>> > allocation.
>>
>> Please make sure you measure your memory usage correctly on Linux, it is
>> notoriously easy to get misleading info from tools like top.
>>
>> I use this one on Linux.
>>
>> $cat ~/bin/showmemusage
>> #!/bin/sh
>> if test -z "$1"; then
>>     echo "need a pid"
>>     exit
>> fi
>>
>> mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
>> sed 's#^#+#' ) | bc`
>> echo "$mem KB"
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

Re: [bitcoin-dev] Memory leaks?

[Rusty Russell]

Btc Drak via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> writes:
> I think this thread has gotten to the stage where it should be moved
> to an issue on Github and not continue to CC the bitcoin-dev list (or
> any other list tbh).

Agreed.  I couldn't see an issue, so I've opened one.  Let's track this
there, please.

https://github.com/bitcoin/bitcoin/issues/6876

Cheers,
Rusty.

Re: [bitcoin-dev] Memory leaks?

[Ross Bennetts]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

On Wed, Oct 21, 2015 at 2:01 PM, Jonathan Toomim <j@toom.im> wrote:

> 3. One user on the bitcointalk p2pool thread (
> https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791)
> claimed that he had this memory usage issue on Linux, but not on Mac OS X,
> under a GBT workload in both situations.
>
>
Thanks for the thorough investigation.
FYI, I believe I've experienced this issue on both Linux & Mac OSX.
I've moved my P2Pool/bitcoind node (rossbennetts.com:9332) from Ubuntu to
OSX since my P2Pool instability began and they both seem to become flaky
after a few hours of uptime (since ~v0.10.something). I've tried both core
and xt.
Two other bitcoin nodes, one connected to electrum-server and one
standalone, seem fine.

//long term lurker, first time poster...
   Ross...

-- 
Ross M. W. Bennetts
https://rossbennetts.com/

E-mail: ross.bennetts@gmail.com
Twitter: https://twitter.com/rmwb/
Flickr: https://flickr.com/photos/rmwb/
Instagram: https://instagram.com/rmwb/
Facebook: https://facebook.com/ross.bennetts

[-- Attachment #2: Type: text/html, Size: 2353 bytes --]

Re: [bitcoin-dev] Memory leaks?

[admin]

[-- Attachment #1.1: Type: text/plain, Size: 2890 bytes --]

I can do better than that:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 
                                                                            
                            
10625 ********  20   0 41.310g 0.035t  51172 S   1.0 57.6  47:11.17 
bitcoind     

Any idea what's going on here?

$ cat /proc/10625/status|grep -i VM
VmPeak: 43647420 kB
VmSize: 43319932 kB
VmLck:        60 kB
VmPin:         0 kB
VmHWM:  38286252 kB
VmRSS:  38014300 kB
VmData: 43103812 kB
VmStk:       236 kB
VmExe:      8844 kB
VmLib:    194656 kB
VmPTE:     79932 kB
VmSwap:        0 kB


On Tuesday, October 13, 2015 at 2:56:15 PM UTC-7, Jonathan Toomim (Toomim 
Bros) wrote:
>
> I just noticed that several of my running bitcoind processes were using 
> around 3+ GB of RAM, even though the mempool itself seemed to be under 
> control.
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
>     "size" : 1896,
>     "bytes" : 37341328
> }
>
> [total memory usage not shown -- I restarted bitcoind as soon as I 
> noticed, and didn't copy it down from top]
>
> 37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot of 
> unconfirmed txns floating around the network, memory usage is around 600 
> MB, so this is quite unusual.
>
> After restarting the process and letting it run for a few minutes, I get:
>
>   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> [###] [XXXX]     20   0 1402M  317M 49836 S  1.0  8.2  0:41.71 ./bitcoind 
> -daemon
>
> XXXX@prime:~/bin$ ./bitcoin-cli getmempoolinfo
> {
>     "size" : 1072,
>     "bytes" : 670000
> }
>
> 0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
>
>
> Here's another node I'm running that has been online longer, before 
> restarting:
>
>   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> [###] [XXXX]     20   0 4961M 3540M 11080 S  2.8 45.3  8h20:11 
> bin/bitcoind -daemon
>
> XXXX@feather:~$ bin/bitcoin-cli getmempoolinfo
> {
>     "size" : 3045,
>     "bytes" : 39656126
> }
>
> 39 MB mempool, 3540 MB total memory usage. After restarting bitcoind, I 
> see:
>
> [XXXX]@feather:~$ bin/bitcoin-cli stop
> Bitcoin server stopping
> [XXXX]@feather:~$ bin/bitcoind -daemon
> Bitcoin server starting
> [XXXX]@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo
> {
>     "size" : 39,
>     "bytes" : 47037
> }
>
>
>   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> [###] [XXXX]     20   0 1640M  247M 67960 S  0.0  3.2  0:05.17 
> bin/bitcoind -daemon
>
>
>
>
> Does anybody have any guesses where we might be leaking memory, or what is 
> using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or 
> similar on my nodes. Maybe there's a leak in the minrelaytxfee code path? 
> Has anyone else seen something similar?
>
> This issue appears to happen both with Bitcoin Core 0.10.1 and with 
> Bitcoin XT 0.11B.
>

[-- Attachment #1.2: Type: text/html, Size: 7883 bytes --]