memory leak?

[Skywalker]

Err, looks to me like it's only for x86 and ARM platforms.

[warewolf]

nuts, you're right. Do you want me to prod the author to see if he can come up with a sparc version?

[warewolf]

More searching around on task_struct leakage:

http://lkml.org/lkml/2006/8/22/246

I still havn't found a method to test or accelerate the speed of the memory leak

[audetto]

More searching around on task_struct leakage:

http://lkml.org/lkml/2006/8/22/246

I still havn't found a method to test or accelerate the speed of the memory leak

It is probably completely unrelated, but my NAS has been a very slow today.
Slower than usual in directory listing and in frontview. It is very likely a wireless congestion in the flat and nothing else

How would you detect memory leaks?

This is the output of "free"

total used free shared buffers cached
Mem: 487840 481552 6288 0 22816 416752
-/+ buffers/cache: 41984 445856
Swap: 767904 0 767904

and this is of /proc/meminfo

MemTotal: 487840 kB
MemFree: 6176 kB
Buffers: 22880 kB
Cached: 416752 kB
SwapCached: 0 kB
Active: 121952 kB
Inactive: 354336 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 487840 kB
LowFree: 6176 kB
SwapTotal: 767904 kB
SwapFree: 767904 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 50336 kB
Slab: 14400 kB
CommitLimit: 1060608 kB
Committed_AS: 131936 kB
PageTables: 0 kB
VmallocTotal: 131008 kB
VmallocUsed: 2096 kB
VmallocChunk: 128496 kB

I don't really know how to read those values tough.

[Skywalker]

total used free shared buffers cached
Mem: 487840 481552 6288 0 22816 416752
-/+ buffers/cache: 41984 445856
Swap: 767904 0 767904

That means your system has ~435MB of physical memory that it can use by clearing out page cache and/or buffers. And your system hasn't even touched swap, so your memory situation is looking pretty good to me.

[Skywalker]

More searching around on task_struct leakage:

http://lkml.org/lkml/2006/8/22/246

I still havn't found a method to test or accelerate the speed of the memory leak

I set up a script to kick over over 4,000 processes per minute, but I still haven't been able to reproduce a memory leak, even letting it run overnight.

[warewolf]

yeah, I'm trying to figure out if the leaking is occuring on a failed fork or what.

Basically, in regard to process churn I've got the following:

Stuff I've added that isn't per-se normal debian stuff:
Every day, a rather long and CPU intensive rsync backs up a remote colocated server.
Every day, another similarly intensive rsync backs up my personal file server at home.
Every 5 minutes, munin does some metric gathering. The munin daemon is a perl script, that kicks off other scripts (could be shell scripts, or perl scripts). The data is collected into RRDs on a remote server.

Normal debian stuff added through apt:
Every 10 minutes, the sysstat package collects SAR metrics.
Once a day, SAR metrics are summarized into daily reports.

The best way I can find to see the leak is downloading the latest version of procps, and compiling up the slabtop utility.

I've done a "firmware reinstall" multiple times, and it doesn't aleviate the problem. So obviously I'm doing something that is tickling the kernel bug, where ever it may be. I'll try turnin off munin for a few days (now that I have slabtop to actually see the status of the leak) and see how that looks.

[mstensgaard]

Did you solve the issue or find the reason for the memory leak?

I seem to have the same issue on my nv+, 1GB ram. I am only using the nfs service (Increased the number of nfsd to .
I am using radiator 4.1.4 with the only addition that I installed the ssh add-on and installed 'munin' to monitor the device.
The memory used by "apps" is slowly increasing by 100MB per day. It seems to be linearly so I have to reboot my device once a week.

[Skywalker]

I still haven't seen any systems die from OOM, and I have systems running with months of uptime. Wish I had some idea what could be going on here.

[mstensgaard]

I suspect that the 'munin-node' tiggles a kernel bug that leaks. This would also explain that the leak is so constant that the 'apps' memory consumption increases linearly.
I will try to stop munin for a day and will post the result in a day or two.

[mstensgaard]

It was the 'munin-node' that caused the memory leak! (At least the memory leak I have experienced).

Munin is just a small deamon and a collection of shell scripts that collects information about load, temperate, memory etc. I have tried to disable everything but the memory collector and this worked perfectly for 1 day without any leaks.
So the leak must be in one of the shell-scripts. I still think that the memory leak is in the kernel or a driver and that it is one of these shell scripts that tiggles the leak by system call.
I will try to see if I can find out what shell command that is really the causing the leak.

[Skywalker]

It was the 'munin-node' that caused the memory leak! (At least the memory leak I have experienced).

Interesting... Please post back if you narrow it down. It certainly does seem very strange.

[mstensgaard]

The problems seems to be in the bash shell, or at least it seems to be the bash shell that triggers the leak

This simple shell script leaks on my nv+.

Code: Select all

#!/bin/sh
for i in $(seq 1 10000); do
  echo "memleak test" | grep -v "//"
done


I then tried installing another shell, csh, and created a similar script

Code: Select all

#!/bin/csh
@ y = 1
while( $y < 10000 )
  echo "memleak test" | grep -v "//"
end


This script does not seem to leak.

I have tried to upgrade bash from version 2.06b to the version from sarge-backports 3.1.17, but the problem is still there.

Can you reproduce the problem?

[btm]

The problems seems to be in the bash shell, or at least it seems to be the bash shell that triggers the leak

Can you reproduce the problem?

ReadyNAS 1100
RAIDiator!!version=4.00c1-p2,time=1198024312
Linux nas-02 2.6.17.8ReadyNAS #1 Mon Dec 17 19:35:18 PST 2007 padre unknown

Note the increase in apps.value below. I stopped running munin-node on these boxes and I've mostly stopped using them. I was unable to resolve my leak so I've been migrating production data off ReadyNAS products here. Earlier thread is here. There's other data if you want it. Or if you'd like me to help you debug this let me know.

Code: Select all

./getdata.sh
./leak.sh
./getdata.sh
# about an hour later
./getdata.sh


Code: Select all

#!/bin/sh

DATE=`date +%Y%m%d-%H%M`
mkdir $DATE
uptime > $DATE/uptime
cat /etc/raidiator_version > $DATE/version
/usr/share/munin/plugins/memory > $DATE/memory
memstat > $DATE/memstat
cat /proc/meminfo > $DATE/meminfo
ps aux > $DATE/ps


Code: Select all

#!/bin/sh
for i in $(seq 1 10000); do
  echo "memleak test" | grep -v "//"
done


Code: Select all

nas-02:~# cat 20081022-1606/memory
slab.value 94191616
swap_cache.value 344064
page_tables.value 0
vmalloc_used.value 5095424
apps.value 74465280
free.value 11239424
buffers.value 595984384
cached.value 253657088
swap.value 458752
committed.value 65355776
mapped.value 42434560
active.value 455999488
inactive.value 420364288


Code: Select all

nas-02:~# cat 20081022-1629/memory
slab.value 94945280
swap_cache.value 344064
page_tables.value 0
vmalloc_used.value 5095424
apps.value 86933504
free.value 9846784
buffers.value 596115456
cached.value 241696768
swap.value 458752
committed.value 65355776
mapped.value 42434560
active.value 447397888
inactive.value 417136640


Code: Select all

nas-02:~# cat 20081022-1723/memory
slab.value 94928896
swap_cache.value 344064
page_tables.value 0
vmalloc_used.value 5095424
apps.value 87113728
free.value 10076160
buffers.value 596623360
cached.value 240795648
swap.value 458752
committed.value 65519616
mapped.value 42483712
active.value 447627264
inactive.value 416563200


[Skywalker]

Nice! Looks like I'm seeing the same thing using your script. I don't seem to be losing as much per run, but some does appear to be lost. Thanks for narrowing it down! Hopefully we can get a handle on this soon.