Overly Sharpened Blog

Swap Usage Summary

Top is full of lies. And so is every other process monitor out there, probably.

They tell you the amount of swap allocated for a process, not the amount of swap actually in _use_ by that process. I believe the trouble is in the distinction between pages which have been written out to swap (but are still active in memory) and those which have been read back from swap (but where the copy in swap still exists because the page hasn't been modified).

In both of these cases, the tool is counting pages as being in swap, even though there is no cost to "pulling" them back out of swap. This gives a false impression of the swap usage on a machine, often to the point of completely obscuring which processes actually have swapped-out pages.

Enter a horrible python one-liner:

for line in ('%5s %20s %8dk' % row for row in sorted([(d, open(d+'/stat').read().split()[1][1:-1], sum(int(line.split()[1]) for line in open(d+'/smaps').read().splitlines() if line.startswith('Swap:'))) for d in __import__('os').listdir('.') if d.isdigit()], key=lambda k: k[2]) if row[2]): print line

Um, right, and ignore the abuse of __import__ there. :p

Saner, more useful source is listed below, but first, a quick look at the output from my slow and therefore unloaded home computer:

 2634      gdu-notificatio       28k
 3022      chromium-browse      108k
 1906      gnome-keyring-d      144k
 2112          dbus-daemon      148k
 2200                gvfsd      156k
 2633         clock-applet      176k
 2205      gvfs-fuse-daemo      244k
 2102          dbus-launch      252k
 2371             nautilus      264k
 2617      e-calendar-fact      360k
 2611      gnome-screensav      416k
 2341           pulseaudio      504k
 2593      bonobo-activati      532k
 2366          gnome-panel      576k
 2369         gconf-helper      700k
 1940        gnome-session      752k
 2336      polkit-gnome-au      828k
 2195      gnome-settings-      832k
 2178             gconfd-2     1044k
 2373      gnome-volume-ma     1104k
 2339      evolution-alarm     1160k
 2361            nm-applet     1304k
 2190      gnome-power-man     1360k
 2491      ubuntu-sso-logi     4884k
 2209               compiz     6324k

Compared the top output:

top - 07:13:42 up 1 day,  5:50,  8 users,  load average: 0.21, 0.38, 0.45
Tasks: 230 total,   1 running, 229 sleeping,   0 stopped,   0 zombie
Cpu(s): 28.1%us,  7.9%sy,  1.3%ni, 62.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2578192k total,  2474400k used,   103792k free,       44k buffers
Swap:  1959920k total,    43944k used,  1915976k free,   946496k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND                              
 3193 cwillu    20   0  466m  75m  19m S  7.9  3.0  17:48.60 391m rhythmbox                            
 3022 cwillu    20   0  361m  90m  26m S  0.3  3.6  17:52.99 270m chromium-browse                      
 2341 cwillu     9 -11  158m 4720 3800 S  3.3  0.2  23:21.64 154m pulseaudio                           
 3367 cwillu    20   0  204m  51m  14m S  4.6  2.0  62:35.57 153m chromium-browse                      
 3260 cwillu    20   0  222m  70m  14m S  0.0  2.8   0:49.22 151m chromium-browse                      
 2195 cwillu    20   0  158m  13m  10m S  0.0  0.5   0:42.28 145m gnome-settings-                      
 3187 cwillu    20   0  165m  30m  12m S  2.0  1.2  30:27.93 135m chromium-browse                      
 3772 cwillu    20   0  191m  59m  14m S  0.0  2.4   1:34.67 132m chromium-browse                      
 3269 cwillu    25   5  171m  39m  14m S  0.0  1.6   0:08.32 131m chromium-browse                      
 3263 cwillu    20   0  186m  57m  14m S  0.0  2.3   0:35.36 128m chromium-browse                      
 3303 cwillu    25   5  183m  55m  14m S  0.0  2.2   0:50.63 128m chromium-browse                      
 3293 cwillu    20   0  177m  49m  14m S  0.0  2.0   1:09.17 127m chromium-browse                      
 3272 cwillu    20   0  176m  51m  14m S  0.0  2.0   0:29.34 125m chromium-browse                      
 2361 cwillu    20   0  132m 9396 8116 S  0.0  0.4   0:00.53 123m nm-applet                            
 4837 cwillu    20   0  154m  36m  11m S  0.0  1.5   1:00.19 118m gimp-2.7                             
 2371 cwillu    20   0  156m  38m  17m S  0.0  1.5   1:27.77 118m nautilus                             
 2967 cwillu    20   0  145m  27m  14m S  0.0  1.1   6:47.36 117m xchat   
Full of lies it is! Notice how even though rhythmbox doesn't have a single byte in swap, top reports 391MB; and likewise chromium has only 108k in swap, in one single process, while top reports 270MB!

My much faster, and therefore heavily loaded, work machine:

27664                  top       36k
21973                  top       76k
27428       update-manager      148k
32176              evinced      160k
10954                 less      188k
22822                pager      188k
17777                 bash      192k
 2730      mission-control      204k
 2268       gvfsd-metadata      212k
 1989          dbus-launch      256k
31588      telepathy-gabbl      256k
 1990          dbus-daemon      260k
 2023      gvfs-gdu-volume      296k
31483                gvfsd      332k
 2071           gvfsd-burn      340k
 2028      gvfs-afc-volume      352k
 9476                  ssh      356k
 2031      gvfs-gphoto2-vo      364k
13950             synergys      396k
 2052      bonobo-activati      428k
 2236                 bash      428k
 6810                  ssh      468k
26032       gvfsd-computer      468k
25875          gvfsd-trash      520k
31488      gvfs-fuse-daemo      532k
27345           gvfsd-http      560k
 2286      indicator-messa      580k
 2017         gconf-helper      584k
 2275      indicator-sessi      584k
 2284      indicator-appli      600k
 2270      indicator-me-se      608k
 2016           pulseaudio      656k
22811                  man      664k
 2288      indicator-sound      688k
 3771                  ssh      692k
31510            gvfsd-smb     1168k
 1993             gconfd-2     1216k
17015                 bash     1252k
 2250      gdu-notificatio     1268k
 1313      evolution-data-     1436k
 2063      multiload-apple     1468k
 1947        gnome-session     1516k
 2048      bluetooth-apple     1560k
 3200      update-notifier     1640k
25667               python     1640k
 2293          wnck-applet     1736k
 2065      notification-ar     1888k
19523               python     1912k
 3258      chromium-browse     2028k
 2038      gnome-power-man     2044k
 2036      polkit-gnome-au     2288k
 2039      gtk-window-deco     2364k
 4010                 bash     2364k
19089               python     2364k
 2047            nm-applet     2392k
 2592      desktopcouch-se     2432k
 2068      indicator-apple     2492k
26786                 bash     2592k
22606                 bash     2604k
24060                 bash     2604k
28553                 bash     2604k
 2307                 bash     2608k
 3332                 bash     2612k
 3751                 bash     2612k
 3773                 bash     2612k
 3774                 bash     2612k
 4526                 bash     2612k
 3007                 bash     2616k
 4569                 bash     2616k
13504                 bash     2616k
21531                 bash     2616k
22830                 bash     2616k
24539                 bash     2616k
 2786                 bash     2620k
11753                 bash     2620k
12855                 bash     2620k
13345                 bash     2620k
14810                 bash     2620k
15032                 bash     2620k
18097                 bash     2620k
25626                 bash     2620k
 8808                 bash     2624k
27332                 bash     2624k
29694                 bash     2624k
30185                 bash     2624k
 2561                 bash     2628k
17641                 bash     2628k
 8014                 bash     2636k
30091                 bash     2752k
25483                 bash     2760k
 1318      evolution-excha     2884k
 3818                xchat     3360k
 3415      chromium-browse     4100k
 1929      gnome-keyring-d     4516k
 2000      gnome-settings-     5056k
 2271      gnome-screensav     6808k
 2256      indicator-apple     6856k
 2347      desktopcouch-se     7240k
14681                 bash     7364k
 2334               python     8092k
 2345      gwibber-service    10652k
 2014          gnome-panel    11700k
 2344      ubuntuone-syncd    11976k
 2043           bzr-notify    12968k
 2295         clock-applet    13896k
 2305       gnome-terminal    14544k
17782            banshee-1    16792k
 2032               compiz    18784k
 3845      chromium-browse    22804k
12355               evince    28856k
32174               evince    29112k
 2728              empathy    33888k
 2033             nautilus    36504k
15344          soffice.bin    39176k
28693           notify-osd   101668k
 3256      chromium-browse   111280k
 3905             inkscape   170060k
31523               evince   193888k
 3351                gedit   254276k
Did I ever mention that swap on a good SSD drive is pure win? inkscape will swap in 100MB in a split-second on this, as opposed to 10-15 minutes on the same machine on a conventional drive. Not noticing a couple hundred MB swapping in and out when switching projects is simply wonderful.

(source):

import os
import sys

def get_swap_table(pids=None):
  if pids:
    pids = set(pids)
    
  table = []
  for pid in os.listdir('/proc/'):
    if not pid.isdigit():
      continue
    if pids and pid not in pids:
      continue

    with open(os.path.join('/proc', pid, 'stat')) as stat:
      name = stat.read().split()[1][1:-1]

    with open(os.path.join('/proc', pid, 'smaps')) as smaps:
      swap_mappings = []
      for line in smaps:
        if not line.startswith('Swap:'):
          continue
        swap_mappings.append(int(line.split()[1]))

    table.append((pid, name, sum(swap_mappings)))

  return table
    
def main(*pids):
  summary = get_swap_table(pids)
  summary.sort(key=lambda (pid, name, size): size)
  for pid, name, size in summary:
    if not size and not pids:
      continue
    print '%5s %20s %8dk' % (pid, name, size)
    
if __name__ == "__main__":
  main(*sys.argv[1:])