chirpa wrote:Do you have another way besides munin to watch the system resources? Awhile ago another user was running munin and also ran into out of memory issues, curious if it itself could be related somehow.
Sure, I can I just leave vmstat running on a terminal, which I'm doing anyways, but I when we started having memory issues yesterday I stopped munin-node and frontview, which temporarily give me some extra memory, but when the OOM killer went wild again was when I set vm.swappiness after realizing there was plenty of swap. Since munin-node wasn't running at that time, I'm reluctant to believe it was was munin. I originally though things were just slow because of the swapping causing disk contention on a machine that already has high disk I/O because it's used for common data storage for a web cluster.
the perl instance of munin hasn't gained any memory since last night:
- Code: Select all
nas-01:~# ps ax | grep mun
400 ? Ss 0:02 /usr/sbin/munin-node
28815 ? S 0:00 /usr/sbin/munin-node
28967 ? Rs 0:00 /usr/sbin/munin-node
28969 pts/1 R+ 0:00 grep mun
nas-01:~# grep perl *mem*
20080506-memstat: 3072k: PID 400 (/usr/bin/perl)
20080506-memstat: 64k: /usr/lib/perl/5.8.8/auto/Fcntl/Fcntl.so 400
20080506-memstat: 1296k: /usr/bin/perl 400
20080506-memstat: 64k: /usr/lib/perl/5.8.8/auto/IO/IO.so 400
20080506-memstat: 64k: /usr/lib/perl/5.8.8/auto/Socket/Socket.so 400
20080506-memstat: 1184k: /usr/lib/libperl.so.5.8.8 1509 1858
20080506-memstat: 576k: /usr/lib/apache2/modules/mod_perl.so 1509 1858
20080507-0924-memstat: 3072k: PID 400 (/usr/bin/perl)
20080507-0924-memstat: 64k: /usr/lib/perl/5.8.8/auto/Fcntl/Fcntl.so 400
20080507-0924-memstat: 1296k: /usr/bin/perl 400
20080507-0924-memstat: 64k: /usr/lib/perl/5.8.8/auto/IO/IO.so 400
20080507-0924-memstat: 64k: /usr/lib/perl/5.8.8/auto/Socket/Socket.so 400
20080507-0924-memstat: 1184k: /usr/lib/libperl.so.5.8.8 1509 1858 24791
20080507-0924-memstat: 576k: /usr/lib/apache2/modules/mod_perl.so 1509 1858 24791
The memory is all in cache now. Application memory gains appear to be in samba and apache. My first dump from memstat that I logged may have been before the box was completely up. I'll compare the output of memstat again later on.
- Code: Select all
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 0 97056 267520 469296 0 0 399 62 10 586 10 29 59 3
- Code: Select all
nas-01:~# diff 20080507-0924-memstat 20080506-memstat -y | more
480k: PID 1 (/sbin/init) 480k: PID 1 (/sbin/init)
3072k: PID 400 (/usr/bin/perl) 3072k: PID 400 (/usr/bin/perl)
672k: PID 710 (/usr/sbin/sshd) 672k: PID 710 (/usr/sbin/sshd)
496k: PID 1041 (/sbin/portmap) 496k: PID 1041 (/sbin/portmap)
496k: PID 1057 (/sbin/syslogd) 496k: PID 1057 (/sbin/syslogd)
496k: PID 1069 (/usr/sbin/atd) 496k: PID 1069 (/usr/sbin/atd)
496k: PID 1075 (/sbin/klogd) 496k: PID 1075 (/sbin/klogd)
480k: PID 1078 (/usr/sbin/inetd) 480k: PID 1078 (/usr/sbin/inetd)
496k: PID 1102 (/usr/sbin/cron) 496k: PID 1102 (/usr/sbin/cron)
1936k: PID 1109 (/usr/sbin/cupsd) 1936k: PID 1109 (/usr/sbin/cupsd)
752k: PID 1111 (/usr/sbin/avahi-daemon) 752k: PID 1111 (/usr/sbin/avahi-daemon)
1744k: PID 1200 (/usr/sbin/smbd) 1744k: PID 1200 (/usr/sbin/smbd)
1872k: PID 1208 (/usr/sbin/snmpd) 1872k: PID 1208 (/usr/sbin/snmpd)
1440k: PID 1221 (/usr/sbin/snmptrapd) 1440k: PID 1221 (/usr/sbin/snmptrapd)
1408k: PID 1225 (/usr/sbin/readynas-agent) 1408k: PID 1225 (/usr/sbin/readynas-agent)
480k: PID 1243 (/usr/sbin/rpc.rquotad) 480k: PID 1243 (/usr/sbin/rpc.rquotad)
496k: PID 1274 (/sbin/rpc.statd) | 480k: PID 1274 (/sbin/rpc.statd)
1744k: PID 1276 (/usr/sbin/smbd) 1744k: PID 1276 (/usr/sbin/smbd)
912k: PID 1298 (/usr/sbin/rpc.mountd) 912k: PID 1298 (/usr/sbin/rpc.mountd)
1136k: PID 1308 (/usr/sbin/proftpd) 1136k: PID 1308 (/usr/sbin/proftpd)
10976k: PID 1509 (/usr/sbin/apache2) 10976k: PID 1509 (/usr/sbin/apache2)
608k: PID 1517 (/usr/sbin/raidard) 608k: PID 1517 (/usr/sbin/raidard)
1760k: PID 1523 (/usr/sbin/winbindd) 1760k: PID 1523 (/usr/sbin/winbindd)
1760k: PID 1525 (/usr/sbin/winbindd) 1760k: PID 1525 (/usr/sbin/winbindd)
544k: PID 1530 (/usr/bin/rsync) 544k: PID 1530 (/usr/bin/rsync)
1760k: PID 1572 (/usr/sbin/winbindd) 1760k: PID 1572 (/usr/sbin/winbindd)
1536k: PID 1580 (/usr/sbin/nmbd) 1536k: PID 1580 (/usr/sbin/nmbd)
672k: PID 1641 (/frontview/bin/monitor_enclosure) 672k: PID 1641 (/frontview/bin/monitor_enclosure)
512k: PID 1857 (/sbin/getty) 512k: PID 1857 (/sbin/getty)
10992k: PID 1858 (/usr/sbin/apache2) 10992k: PID 1858 (/usr/sbin/apache2)
736k: PID 2320 (/usr/sbin/sshd) 736k: PID 2320 (/usr/sbin/sshd)
672k: PID 2322 (/bin/bash) 672k: PID 2322 (/bin/bash)
1760k: PID 9381 (/usr/sbin/winbindd) | 640k: PID 7171 (/usr/bin/memstat)
2144k: PID 18604 (/usr/sbin/smbd) | 64k: /dev/zero 1509 1858
3040k: PID 24778 (/usr/sbin/smbd) <
2160k: PID 24790 (/usr/sbin/smbd) <
11136k: PID 24791 (/usr/sbin/apache2) <
2128k: PID 25681 (/usr/sbin/smbd) <
2032k: PID 27526 (/usr/sbin/smbd) <
512k: PID 27876 (/usr/bin/vmstat) <
2032k: PID 28307 (/usr/sbin/smbd) <
2032k: PID 28308 (/usr/sbin/smbd) <
736k: PID 28309 (/usr/sbin/sshd) <
656k: PID 28311 (/bin/bash) <
768k: PID 28348 (/usr/bin/memstat) <
64k: /dev/zero 1509 1858 24791 <
1280k: /dev/zero 2320 1280k: /dev/zero 2320
1280k: /dev/zero 2320 1280k: /dev/zero 2320
1280k: /dev/zero 28309 | 64k: /lib/libacl.so.1.1.0 1200 1276 1530
1280k: /dev/zero 28309 <
64k: /lib/libacl.so.1.1.0 1200 1276 1530 18604 24778 247 <
64k: /lib/libpam.so.0.76 710 1102 1109 1200 1276 1298 13 64k: /lib/libpam.so.0.76 710 1102 1109 1200 1276 1298 13
64k: /lib/libattr.so.1.1.0 1200 1276 1530 18604 24778 24 | 64k: /lib/libattr.so.1.1.0 1200 1276 1530
64k: /lib/libcom_err.so.2.1 1200 1276 1298 1523 1525 157 | 64k: /lib/libcom_err.so.2.1 1200 1276 1298 1523 1525 157
64k: /lib/libcap.so.1.10 1200 1276 1298 1523 1525 1572 1 | 64k: /lib/libcap.so.1.10 1200 1276 1298 1523 1525 1572 1
320k: /lib/libdb2.so.2.7.7 1200 1276 1298 1523 1525 1572 | 320k: /lib/libdb2.so.2.7.7 1200 1276 1298 1523 1525 1572
288k: /lib/libncurses.so.5.4 2322 28311 | 288k: /lib/libncurses.so.5.4 2322
832k: /lib/libnss_wins.so.2 1298 24778 | 832k: /lib/libnss_wins.so.2 1298
64k: /lib/libnss_winbind.so.2 400 1111 1200 1276 1308 18 | 64k: /lib/libnss_winbind.so.2 400 1111 1200 1276 1308 18
64k: /lib/libpopt.so.0.0.0 1200 1276 1523 1525 1530 1572 | 64k: /lib/libpopt.so.0.0.0 1200 1276 1523 1525 1530 1572
64k: /lib/libuuid.so.1.2 1298 1509 1858 24791 | 64k: /lib/libuuid.so.1.2 1298 1509 1858
64k: /lib/libwrap.so.0.7.6 710 1041 1208 1221 1225 1243 | 64k: /lib/libwrap.so.0.7.6 710 1041 1208 1221 1225 1243
112k: /lib/ld-2.3.2.so 1 400 710 1041 1057 1069 1075 1078 112k: /lib/ld-2.3.2.so 1 400 710 1041 1057 1069 1075 1078
64k: /lib/libcrypt-2.3.2.so 400 710 1109 1200 1276 1298 64k: /lib/libcrypt-2.3.2.so 400 710 1109 1200 1276 1298
1280k: /lib/libc-2.3.2.so 1 400 710 1041 1057 1069 1075 10 1280k: /lib/libc-2.3.2.so 1 400 710 1041 1057 1069 1075 10
64k: /lib/libnss_compat-2.3.2.so 400 710 1069 1102 1109 64k: /lib/libnss_compat-2.3.2.so 400 710 1069 1102 1109
64k: /lib/libdl-2.3.2.so 400 710 1102 1109 1111 1200 120 64k: /lib/libdl-2.3.2.so 400 710 1102 1109 1111 1200 120
528k: /lib/libm-2.3.2.so 400 1208 1221 1225 1509 1858 247 | 528k: /lib/libm-2.3.2.so 400 1208 1221 1225 1509 1858
128k: /lib/libnsl-2.3.2.so 400 710 1041 1069 1102 1109 11 128k: /lib/libnsl-2.3.2.so 400 710 1041 1069 1102 1109 11
64k: /lib/libnss_files-2.3.2.so 400 710 1057 1069 1102 1 64k: /lib/libnss_files-2.3.2.so 400 710 1057 1069 1102 1
64k: /lib/libnss_dns-2.3.2.so 1200 1274 1276 1298 1523 1 | 64k: /lib/libnss_dns-2.3.2.so 1200 1276 1298 1523 1525 1
64k: /lib/libnss_nis-2.3.2.so 400 710 1069 1102 1109 111 64k: /lib/libnss_nis-2.3.2.so 400 710 1069 1102 1109 111
64k: /lib/libpthread-0.10.so 1111 1509 1858 24791 | 64k: /lib/libpthread-0.10.so 1111 1509 1858
64k: /lib/libresolv-2.3.2.so 710 1200 1274 1276 1298 152 | 64k: /lib/libresolv-2.3.2.so 710 1200 1276 1298 1523 152
64k: /lib/librt-2.3.2.so 1509 1858 24791 | 64k: /lib/librt-2.3.2.so 1509 1858
64k: /lib/libutil-2.3.2.so 710 2320 28309 | 64k: /lib/libutil-2.3.2.so 710 2320
64k: /lib/libblkid.so.1.0 1298 64k: /lib/libblkid.so.1.0 1298
64k: /lib/libproc.so.3.2.1 27876 | 608k: /bin/bash 2322
608k: /bin/bash 2322 28311 | 2048k: /usr/lib/locale/locale-archive 1509 1523 1525 1530
2048k: /usr/lib/locale/locale-archive 1509 1523 1525 1530 | 32k: /etc/samba/secrets.tdb 1200 1276 1523 1525 1572
32k: /etc/samba/secrets.tdb 1200 1276 1523 1525 1572 938 | 64k: /usr/lib/samba/valid.dat 1200 1276 1523 1525 1572 1
64k: /usr/lib/samba/valid.dat 1200 1276 1523 1525 1572 1 <
64k: /usr/lib/perl/5.8.8/auto/Fcntl/Fcntl.so 400 64k: /usr/lib/perl/5.8.8/auto/Fcntl/Fcntl.so 400
1296k: /usr/bin/perl 400 1296k: /usr/bin/perl 400
304k: /usr/bin/rsync 1530 304k: /usr/bin/rsync 1530
16k: /usr/bin/vmstat 27876 | 16k: /usr/bin/memstat 7171
16k: /usr/bin/memstat 28348 | 64k: /var/lib/samba/winbindd_cache.tdb 1523 1525 1572
64k: /var/lib/samba/winbindd_cache.tdb 1523 1525 1572 93 | 48k: /var/lib/samba/idmap_cache.tdb 1523 1525 1572
48k: /var/lib/samba/idmap_cache.tdb 1523 1525 1572 9381 <
16k: /var/lib/samba/notify.tdb 18604 24778 24790 25681 2 <
16k: /var/lib/samba/unexpected.tdb 1580 16k: /var/lib/samba/unexpected.tdb 1580
16k: /var/lib/samba/group_mapping.tdb 1200 1276 1572 186 | 16k: /var/lib/samba/group_mapping.tdb 1200 1276
16k: /var/lib/samba/account_policy.tdb 1200 1276 18604 2 | 16k: /var/lib/samba/account_policy.tdb 1200 1276
16k: /var/lib/samba/messages.tdb 1200 1276 1523 1525 157 | 16k: /var/lib/samba/messages.tdb 1200 1276 1523 1525 157
32k: /var/lib/samba/sessionid.tdb 1200 1276 18604 24778 | 16k: /var/lib/samba/sessionid.tdb 1200 1276
32k: /var/lib/samba/connections.tdb 1200 1276 18604 2477 | 16k: /var/lib/samba/connections.tdb 1200 1276
48k: /var/lib/samba/brlock.tdb 1200 1276 18604 24778 247 | 48k: /var/lib/samba/brlock.tdb 1200 1276
176k: /var/lib/samba/locking.tdb 1200 1276 18604 24778 24 | 48k: /var/lib/samba/locking.tdb 1200 1276
16k: /var/lib/samba/share_info.tdb 18604 24778 24790 256 | 16k: /var/lib/samba/gencache.tdb 1200 1276 1523 1525 157
16k: /var/lib/samba/gencache.tdb 1200 1276 1523 1525 157 | 16k: /var/lib/samba/ntdrivers.tdb 1200 1276
16k: /var/lib/samba/ntdrivers.tdb 1200 1276 18604 24778 | 16k: /var/lib/samba/ntprinters.tdb 1200 1276
16k: /var/lib/samba/ntprinters.tdb 1200 1276 18604 24778 | 16k: /var/lib/samba/ntforms.tdb 1200 1276
16k: /var/lib/samba/ntforms.tdb 1200 1276 18604 24778 24 | 80k: /var/lib/samba/winbindd_idmap.tdb 1523 1572
80k: /var/lib/samba/winbindd_idmap.tdb 1523 1572 9381 | 48k: /var/lib/samba/netsamlogon_cache.tdb 1523 1525 1572
48k: /var/lib/samba/netsamlogon_cache.tdb 1523 1525 1572 <
64k: /usr/lib/perl/5.8.8/auto/IO/IO.so 400 64k: /usr/lib/perl/5.8.8/auto/IO/IO.so 400
64k: /usr/lib/perl/5.8.8/auto/Socket/Socket.so 400 64k: /usr/lib/perl/5.8.8/auto/Socket/Socket.so 400
64k: /frontview/bin/monitor_enclosure 1641 64k: /frontview/bin/monitor_enclosure 1641
64k: /lib/security/pam_env.so 2320 28309 | 64k: /lib/security/pam_env.so 2320
64k: /lib/security/pam_lastlog.so 2320 28309 | 64k: /lib/security/pam_lastlog.so 2320
64k: /lib/security/pam_limits.so 2320 28309 | 64k: /lib/security/pam_limits.so 2320
64k: /lib/security/pam_mail.so 2320 28309 | 64k: /lib/security/pam_mail.so 2320
64k: /lib/security/pam_motd.so 2320 28309 | 64k: /lib/security/pam_motd.so 2320
64k: /lib/security/pam_nologin.so 2320 28309 | 64k: /lib/security/pam_nologin.so 2320
64k: /lib/security/pam_unix.so 2320 28309 | 64k: /lib/security/pam_unix.so 2320
96k: /sbin/init 1 96k: /sbin/init 1
80k: /sbin/getty 1857 80k: /sbin/getty 1857
32k: /sbin/klogd 1075 32k: /sbin/klogd 1075
16k: /sbin/portmap 1041 16k: /sbin/portmap 1041
48k: /sbin/rpc.statd 1274 48k: /sbin/rpc.statd 1274
32k: /sbin/syslogd 1057 32k: /sbin/syslogd 1057
64k: /usr/lib/gconv/IBM437.so 1200 1276 1523 1525 1572 1 | 64k: /usr/lib/gconv/IBM437.so 1200 1276 1523 1525 1572 1
64k: /usr/lib/gconv/UTF-16.so 1200 1276 1523 1525 1572 1 | 64k: /usr/lib/gconv/UTF-16.so 1200 1276 1523 1525 1572 1
128k: /usr/lib/libcups.so.2 1109 1200 1276 18604 24778 24 | 128k: /usr/lib/libcups.so.2 1109 1200 1276
64k: /usr/lib/libgdbm.so.3.0.0 400 1509 1858 24791 | 64k: /usr/lib/libgdbm.so.3.0.0 400 1509 1858
944k: /usr/lib/libdb-4.2.so 1509 1858 24791 | 944k: /usr/lib/libdb-4.2.so 1509 1858
144k: /usr/lib/libexpat.so.1.0.0 1111 1509 1858 24791 | 144k: /usr/lib/libexpat.so.1.0.0 1111 1509 1858
112k: /usr/lib/liblber.so.2.0.15 1200 1276 1298 1523 1525 | 112k: /usr/lib/liblber.so.2.0.15 1200 1276 1298 1523 1525
128k: /usr/lib/libgssapi_krb5.so.2.2 1200 1276 1298 1523 | 128k: /usr/lib/libgssapi_krb5.so.2.2 1200 1276 1298 1523
192k: /usr/lib/libk5crypto.so.3.0 1200 1276 1298 1523 152 | 192k: /usr/lib/libk5crypto.so.3.0 1200 1276 1298 1523 152
512k: /usr/lib/libkrb5.so.3.2 1200 1276 1298 1523 1525 15 | 512k: /usr/lib/libkrb5.so.3.2 1200 1276 1298 1523 1525 15
64k: /usr/lib/libsasl.so.7.1.10 1200 1276 1298 1523 1525 | 64k: /usr/lib/libsasl.so.7.1.10 1200 1276 1298 1523 1525
64k: /usr/lib/libdaemon.so.0.2.1 1111 64k: /usr/lib/libdaemon.so.0.2.1 1111
384k: /usr/lib/libsqlite3.so.0.8.6 1509 1858 24791 | 384k: /usr/lib/libsqlite3.so.0.8.6 1509 1858
128k: /usr/lib/libz.so.1.2.2 710 1109 1208 1221 1225 1308 | 128k: /usr/lib/libz.so.1.2.2 710 1109 1208 1221 1225 1308
1184k: /usr/lib/libperl.so.5.8.8 1509 1858 24791 | 1184k: /usr/lib/libperl.so.5.8.8 1509 1858
224k: /usr/lib/libldap.so.2.0.15 1200 1276 1298 1523 1525 | 224k: /usr/lib/libldap.so.2.0.15 1200 1276 1298 1523 1525
64k: /usr/lib/libkrb5support.so.0.0 1200 1276 1298 1523 | 64k: /usr/lib/libkrb5support.so.0.0 1200 1276 1298 1523
64k: /usr/lib/libavahi-common.so.3.4.3 1111 64k: /usr/lib/libavahi-common.so.3.4.3 1111
256k: /usr/lib/libavahi-core.so.4.0.4 1111 256k: /usr/lib/libavahi-core.so.4.0.4 1111
128k: /usr/lib/libpcre.so.3.12.0 1509 1858 24791 | 128k: /usr/lib/libpcre.so.3.12.0 1509 1858
128k: /usr/lib/libaprutil-1.so.0.2.7 1509 1858 24791 | 128k: /usr/lib/libaprutil-1.so.0.2.7 1509 1858
144k: /usr/lib/libapr-1.so.0.2.7 1509 1858 24791 | 144k: /usr/lib/libapr-1.so.0.2.7 1509 1858
64k: /usr/lib/libapreq2.so.3.6.0 1509 1858 24791 | 64k: /usr/lib/libapreq2.so.3.6.0 1509 1858
272k: /usr/lib/libnetsnmpagent.so.9.0.1 1208 1221 1225 272k: /usr/lib/libnetsnmpagent.so.9.0.1 1208 1221 1225
256k: /usr/lib/libssl.so.0.9.8 1308 1509 1858 24791 | 256k: /usr/lib/libssl.so.0.9.8 1308 1509 1858
1296k: /usr/lib/libcrypto.so.0.9.8 710 1208 1221 1225 1308 | 1296k: /usr/lib/libcrypto.so.0.9.8 710 1208 1221 1225 1308
528k: /usr/lib/libnetsnmp.so.9.0.1 1208 1221 1225 528k: /usr/lib/libnetsnmp.so.9.0.1 1208 1221 1225
128k: /usr/lib/libnetsnmphelpers.so.9.0.1 1208 1221 1225 128k: /usr/lib/libnetsnmphelpers.so.9.0.1 1208 1221 1225
880k: /usr/lib/libnetsnmpmibs.so.9.0.1 1208 1221 1225 880k: /usr/lib/libnetsnmpmibs.so.9.0.1 1208 1221 1225
64k: /usr/lib/libnetsnmptrapd.so.9.0.1 1221 64k: /usr/lib/libnetsnmptrapd.so.9.0.1 1221
576k: /usr/lib/apache2/modules/mod_perl.so 1509 1858 2479 | 576k: /usr/lib/apache2/modules/mod_perl.so 1509 1858
64k: /usr/lib/apache2/modules/mod_apreq2.so 1509 1858 24 | 64k: /usr/lib/apache2/modules/mod_apreq2.so 1509 1858
64k: /usr/lib/apache2/modules/mod_auth_pam.so 1509 1858 | 64k: /usr/lib/apache2/modules/mod_auth_pam.so 1509 1858
64k: /usr/lib/apache2/modules/mod_auth_sys_group.so 1509 | 64k: /usr/lib/apache2/modules/mod_auth_sys_group.so 1509
48k: /usr/sbin/avahi-daemon 1111 48k: /usr/sbin/avahi-daemon 1111
32k: /usr/sbin/atd 1069 32k: /usr/sbin/atd 1069
224k: /usr/sbin/cupsd 1109 224k: /usr/sbin/cupsd 1109
32k: /usr/sbin/cron 1102 32k: /usr/sbin/cron 1102
32k: /usr/sbin/inetd 1078 32k: /usr/sbin/inetd 1078
32k: /usr/sbin/raidard 1517 32k: /usr/sbin/raidard 1517
912k: /usr/sbin/nmbd 1580 912k: /usr/sbin/nmbd 1580
64k: /usr/sbin/rpc.mountd 1298 64k: /usr/sbin/rpc.mountd 1298
112k: /usr/sbin/rpc.rquotad 1243 112k: /usr/sbin/rpc.rquotad 1243
3136k: /usr/sbin/smbd 1200 1276 18604 24778 24790 25681 27 | 3136k: /usr/sbin/smbd 1200 1276
32k: /usr/sbin/snmpd 1208 32k: /usr/sbin/snmpd 1208
32k: /usr/sbin/snmptrapd 1221 32k: /usr/sbin/snmptrapd 1221
288k: /usr/sbin/sshd 710 2320 28309 | 288k: /usr/sbin/sshd 710 2320
2112k: /usr/sbin/winbindd 1523 1525 1572 9381 | 2112k: /usr/sbin/winbindd 1523 1525 1572
688k: /usr/sbin/apache2 1509 1858 24791 | 688k: /usr/sbin/apache2 1509 1858
544k: /usr/sbin/proftpd 1308 544k: /usr/sbin/proftpd 1308
16k: /usr/sbin/readynas-agent 1225 16k: /usr/sbin/readynas-agent 1225
-------- --------
118640k | 85296k