Memory usage Retrieval on Linux ( process wise and general )

Number of posts : 11 Age : 40 Registration date : 2007-04-13

Note : This guide is applicable to Linux kernels 2.6.14 and above, which adds support for smaps, per-mapping data, including data on each mapping's rss usage.

Most Admins and Developers are used to retrieve memory usage information of a process with VMSTAT or RSS, but these are not accurate,

VMSIZE counts pages that are not mapped in to memory.

RSS multi-counts shared pages, and hence is also more or less useless.

But fortunately, form kernel 2.6.14 (November 2005) onwards, has support for smaps, per-mapping data, including data on each mapping's rss usage.

This mapping data resides in /proc/$pid/smaps.

But it is not easily comprehend-able and lists all private regions and public region usage separately.

we can use this following script written by Ben Maurer, undergraduate at Carnegie Mellon University ( can reach him at, bmaurer@andrew.cmu.edu), many thanks to him.

The perl script follows below, which uses the following perl modules, But before that make sure that you have the following perl module, which is perls interface for smap facility.

Linux::Smaps ( this in turn depends on Class::Member 1.3 )

So if the modules are not there in your system, than get them form the following links.

1.) http://search.cpan.org/CPAN/authors/id/O/OP/OPI/Linux-Smaps-0.01.tar.gz

2.) http://search.cpan.org/CPAN/authors/id/O/OP/OPI/Class-Member-1.3.tar.gz

to install these two , decompress both as follows :

Code:: bash # gunzip -v ./Class-Member-1.3.tar.gz ; tar xvf ./Class-Member-1.3.tar
bash # cd ./Class-Member-1.3
bash # perl Makefile.PL
bash # make
bash # make test
bash # make install

bash # gunzip -v ./Linux-Smaps-0.01.tar.gz ; tar xvf ./Linux-Smaps-0.01.tar
bash # cd ./Linux-Smaps-0.01
bash # perl Makefile.PL
bash # make
bash # make test
bash # make install

Now you can run the following script, with pid as the argument name. To get the complete memory statistics of a particular process.

Code:: #!/usr/bin/perl
#smaps.pl
#typical usage is as follows :
# smaps.pl pid

use Linux::Smaps;

my $pid=shift @ARGV;
unless ($pid) {
print "./smem.pl <pid>\n";
exit 1;
}
my $map=Linux::Smaps->new($pid);
my @VMAs = $map->vmas;

format STDOUT =
VMSIZE: @######## kb
$map->size
RSS: @######## kb total
$map->rss
@######## kb shared
$map->shared_clean + $map->shared_dirty
@######## kb private clean
$map->private_clean
@######## kb private dirty
$map->private_dirty
.

write;

printPrivateMappings ();
printSharedMappings ();

sub sharedMappings () {
return grep { ($_->shared_clean + $_->shared_dirty) > 0 } @VMAs;
}

sub privateMappings () {
return grep { ($_->private_clean + $_->private_dirty) > 0 } @VMAs;
}

sub printPrivateMappings ()
{
$TYPE = "PRIVATE MAPPINGS";
$^ = 'SECTION_HEADER';
$~ = 'SECTION_ITEM';
$- = 0;
$= = 100000000;
foreach $vma (sort {-($a->private_dirty <=> $b->private_dirty)}
privateMappings ()) {
$size = $vma->size;
$dirty = $vma->private_dirty;
$clean = $vma->private_clean;
$file = $vma->file_name;
write;
}
}

sub printSharedMappings ()
{
$TYPE = "SHARED MAPPINGS";
$^ = 'SECTION_HEADER';
$~ = 'SECTION_ITEM';
$- = 0;
$= = 100000000;

foreach $vma (sort {-(($a->shared_clean + $a->shared_dirty)
<=>
($b->shared_clean + $b->shared_dirty))}
sharedMappings ()) {

$size = $vma->size;
$dirty = $vma->shared_dirty;
$clean = $vma->shared_clean;
$file = $vma->file_name;
write;

}
}

format SECTION_HEADER =
@<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$TYPE
@>>>>>>>>>> @>>>>>>>>>> @>>>>>>>>> @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
"vmsize" "rss clean" "rss dirty" "file"
.

format SECTION_ITEM =
@####### kb @####### kb @####### kb @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$size $clean $dirty $file
.

save this file as SCRIPTNAME.pl . I have saved this script as smaps.pl.

Now assume that, you want to see memory usage statistics for the Firefox process running on the system ( useful to detect firefox memory leak problem .. . Very Happy

) .

Make sure smaps.pl is in your path and Just run

Code:: [bash] # pgrep firefox | xargs smaps.pl

you get the following output :

Code:: [root@chandan-gateway chandan]# pgrep firefox | xargs ./smaps.pl
VMSIZE: 240304 kb
RSS: 97392 kb total
6512 kb shared
21952 kb private clean
68928 kb private dirty
PRIVATE MAPPINGS
vmsize rss clean rss dirty file
54624 kb 0 kb 54384 kb
10316 kb 0 kb 10316 kb
376 kb 0 kb 376 kb /SYSV00000000 (deleted)
376 kb 0 kb 376 kb /SYSV00000000 (deleted)
372 kb 0 kb 372 kb /usr/lib/firefox-2.0.0.4/components/libgklayout.so
296 kb 0 kb 296 kb /SYSV00000000 (deleted)
24 kb 0 kb 24 kb /usr/lib/firefox-2.0.0.4/components/libuconv.so
20 kb 0 kb 20 kb /usr/lib/firefox-2.0.0.4/libmozjs.so
16 kb 0 kb 16 kb /usr/lib/firefox-2.0.0.4/components/libxpconnect.so
4 kb 0 kb 4 kb /usr/lib/firefox-2.0.0.4/components/libtxmgr.so
4 kb 0 kb 4 kb /lib/libnss_dns-2.6.so
4 kb 0 kb 4 kb /lib/libnss_dns-2.6.so
4 kb 0 kb 4 kb /usr/lib/firefox-2.0.0.4/components/libnecko2.so
4 kb 0 kb 4 kb /usr/lib/firefox-2.0.0.4/components/libremoteservice.so
60 kb 44 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libmyspell.so
16 kb 12 kb 0 kb /usr/lib/firefox-2.0.0.4/libgtkxtbin.so
24 kb 16 kb 0 kb /usr/lib/libpangoxft-1.0.so.0.1600.4
104 kb 68 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libchrome.so
120 kb 80 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libcaps.so
80 kb 60 kb 0 kb /usr/lib/libbonobo-activation.so.4.0.0
8 kb 8 kb 0 kb /usr/lib/libavahi-glib.so.1.0.1
248 kb 168 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libwidget_gtk2.so
444 kb 268 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libtoolkitcomps.so
16 kb 12 kb 0 kb /usr/lib/libORBitCosNaming-2.so.0.1.0
88 kb 56 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libpref.so
1100 kb 624 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libnecko.so
132 kb 64 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libappcomps.so
52 kb 32 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libcookie.so
44 kb 32 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libpipboot.so
100 kb 48 kb 0 kb /usr/lib/firefox-2.0.0.4/libjsj.so
812 kb 124 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libuconv.so
424 kb 260 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libhtmlpars.so
52 kb 20 kb 0 kb /usr/lib/libgnome-keyring.so.0.0.1
960 kb 296 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libeditor.so
28 kb 20 kb 0 kb /usr/lib/firefox-2.0.0.4/components/libtxmgr.so
112 kb 60 kb 0 kb /usr/share/fonts/liberation/LiberationSans-Regular.ttf
144 kb 144 kb 0 kb /usr/share/fonts/default/Type1/n021003l.pfb
428 kb 20 kb 0 kb /usr/share/fonts/dejavu-lgc/DejaVuLGCSans.ttf
100 kb 100 kb 0 kb /usr/share/fonts/default/Type1/n019003l.pfb
408 kb 12 kb 0 kb /usr/share/fonts/dejavu-lgc/DejaVuLGCSans-Bold.ttf
60 kb 16 kb 0 kb /usr/share/fonts/liberation/LiberationMono-Regular.ttf
72 kb 28 kb 0 kb /usr/share/fonts/liberation/LiberationSans-Italic.ttf
68 kb 40 kb 0 kb /usr/share/fonts/liberation/LiberationSans-Bold.ttf
28 kb 4 kb 0 kb /usr/lib/gconv/gconv-modules.cache
. . . . .
. . . . .
. . . . .

SHARED MAPPINGS
vmsize rss clean rss dirty file
1336 kb 660 kb 0 kb /lib/libc-2.6.so
1016 kb 524 kb 0 kb /usr/lib/libX11.so.6.2.0
384 kb 0 kb 384 kb /SYSV00000000 (deleted)
896 kb 360 kb 0 kb /usr/lib/libstdc++.so.6.0.8
536 kb 340 kb 0 kb /usr/lib/libfreetype.so.6.3.15
632 kb 336 kb 0 kb /lib/libglib-2.0.so.0.1200.11
248 kb 220 kb 0 kb /lib/libgobject-2.0.so.0.1200.11
504 kb 216 kb 0 kb /usr/lib/libnss3.so
384 kb 0 kb 192 kb /SYSV00000000 (deleted)
312 kb 188 kb 0 kb /usr/lib/libORBit-2.so.0.1.0
3676 kb 176 kb 0 kb /usr/lib/libgtk-x11-2.0.so.0.1000.13
212 kb 152 kb 0 kb /usr/lib/libnspr4.so
328 kb 148 kb 0 kb /usr/lib/libsoftokn3.so
156 kb 124 kb 0 kb /usr/lib/libfontconfig.so.1.2.0
204 kb 112 kb 0 kb /usr/lib/libgconf-2.so.4.1.2
108 kb 100 kb 0 kb /lib/ld-2.6.so
148 kb 100 kb 0 kb /usr/lib/libpng12.so.0.16.0
868 kb 92 kb 0 kb /lib/libasound.so.2.0.0
124 kb 88 kb 0 kb /lib/libexpat.so.0.5.0
1136 kb 84 kb 0 kb /lib/libcrypto.so.0.9.8b
428 kb 72 kb 0 kb /usr/share/fonts/dejavu-lgc/DejaVuLGCSans.ttf
552 kb 68 kb 0 kb /usr/lib/libgdk-x11-2.0.so.0.1000.13
572 kb 68 kb 0 kb /usr/lib/libkrb5.so.3.3
80 kb 64 kb 0 kb /lib/libpthread-2.6.so
64 kb 64 kb 0 kb /var/cache/fontconfig/d066cc093611c74861a22d5ef60b042c-x86.ca
232 kb 60 kb 0 kb /usr/lib/libfreebl3.so
408 kb 60 kb 0 kb /usr/share/fonts/dejavu-lgc/DejaVuLGCSans-Bold.ttf
2048 kb 52 kb 0 kb /usr/lib/locale/locale-archive
156 kb 48 kb 0 kb /lib/libm-2.6.so
72 kb 44 kb 0 kb /lib/libz.so.1.2.3
260 kb 44 kb 0 kb /lib/libssl.so.0.9.8b
. . . . .
. . . . .
. . . . .

The starting four lines is enough to see how much memory is it actually using, The next details if useful, if you want to pin point which library or usually plugin lib, which is eating memory , and other analysis.

So just to get the Jist of process memory usage , do the following :

Code:: VMSIZE: 247496 kb
RSS: 104740 kb total
6512 kb shared
21956 kb private clean
76272 kb private dirty

here u see that actually, over 90.5 MB ( that is 21956 kb private clean + 76272 kb private dirty ) Is used by firefox , as reserved private region, exclusively. and only over 6 mb is occupied by shared libs, used by many processes along with Firefox.

actually, Though this is a better solution, it is not ideal. See for example, All this shared libraries is used only by this process, or else just by 2 or 3 processes, than this one is not ideal.

Hope this is helpful.

Regards,

-- Chandan

Chandan.Maddanna@gmail.com

If any queries etc, Reply to this Message by CLICKING HERE

.

Guest

Just what i was looking for , i am trying to use this script as a part of another memory tracer and reporter script. Will surely let you know my mileage.

Mark

Number of posts : 11 Age : 40 Registration date : 2007-04-13

Wow,

Quite Surprising, because even i am designing one. Also i am thinking of this script monitoring and also sending alerts through paging etc.

I will discuss it with you once i am done.

Regards,

-- Chandan

Mark wrote:: Just what i was looking for , i am trying to use this script as a part of another memory tracer and reporter script. Will surely let you know my mileage.

Mark