Plot performance counters in realtime with feedgnuplot

I recently discovered Feedgnuplot. Feedgnuplot is basically a wrapper around Gnuplot, that makes Gnuplot able to plot data fed through a pipe in realtime. Feedgnuplot is thus able to plot realtime performance counters from headless sytems like routers, firewalls and storage systems. In this short article I will give a few examples on how to use Feedgnuplot for various purposes.

Sure: There are a lot of graphing tools and network monitoring tools available for free. But for quick and dirty troubleshooting, feedgnuplot has quite some advantages, which I will get back to.

Feedgnuplot and data format

First of all: Data needs to be prepped before being fed into Feedgnuplot. All unneeded data needs to be removed, leaving only the values. And the values needs to be timestamped. In short: the data needs to look something like this, when piped into feedgnuplot:

The first column is the timestamp in epoch format (seconds since 1st of January 1970). The second column is the value to plot. There can even be several value columns. If so, every value column will be plotted with an unique graph. Plotting the above data with feedgnuplot will look like this:

The above graph is plotted running feedgnuplot with the following arguments:

--domain -- X-data is time
--lines -- Plot data with lines instead of points
--stream -- Plot data in realtime.
--exit -- End gnuplot, when pipe is broken.
--timefmt %s -- Timestamps are in epoch-format.

Plotting ping reply times as a function of time

A simple, but very useful use of feedgnuplot is to plot reply times from ping a a function of time. The output of ping will look like this and is not ready to be plotted. In order to make the result plot-able, the following needs to be done:

Strip out lines not having a reply time value.
Prepend each line with a timestamp
Cut out the timestamp and result value for each result line

The three steps have been illustrated below:

Awk is just the tool for this purpose.. The following awk script is able to do the above with a minimalistic block of code:

BEGIN {
    FS="[ =]"               # Set field separator to be both space and equal sign.
}

/^64 bytes/ {               # Only act on lines starting with the string "64 bytes"
    print systime(), $10    # Print system time in epoch format followed by the time value
}

So ping combined with the awk re-formation script and feedgnuplot will result in this command:

ping 8.8.8.8 | gawk -F "[ =]" '{print systime(),$10}' | 
  feedgnuplot --stream --exit --domain --timefmt %s --lines

And the result will look like this - quite useful to display possible jitter.

Note that gawk is used specifically. There are two main versions of awk:

mawk - A minimalistic awk, which cannot be used in the above example, as it does not have the systime() function
gawk - A more powerful implementation of awk, which has the systime() function

Perl could be used as well in loop mode, if one prefer perl over awk. I do prefer the simplicity of awk, though. If you do not know either awk yet and work with Linux on daily basis: Learn awk!

Plotting disk performance

Disk throughput and transactions per second are valuable performance indicators. The command iostat will provide just those numbers. The command iostat -y 1 will continuously print performance stats every second. The -y parameter prevents accumulated statistics to be printed as the first line (we don't need those). And just like with the ping example, we need to:

Only concentrate on lines with performance counters for the disk we want to monitor
Prepend lines with a timestamp
Cut out the timestamp and the result values for each result line

And again awk is the right tool. The following awk script will timestamp and prepare the output for feedgnuplot

/^sda/ {
    print systime(),$2,$3,$4
}

Combined into a one-liner:

iostat -y 1 | gawk '/^sda/{print systime(),$2,$3,$4}' |
  feedgnuplot --lines --timefmt %s --domain --exit --stream

And the result will look something like this when running dbench and find/ simultaneously:

Oh - I did actually add these parameters to feedgnuplot as well:

--legend 0 tps
--legend 1 "kB_-read/s
--lengend "kB_-write/s

Plotting number of running processes

Both ping and iostat will print the counters at a desired interval. But with other commands, one needs to be a little creative. As an example, I will show how to plot the number of processes on a running system. The command ps -ef | wc -l will result in the number of running process plus one (the first line of ps -ef is a headline, not a process). We can use bash to execute the command in a loop. And awk to subtract one from the process count.

while true; do ps -ef | wc -l; sleep 1; done | awk '{print systime(),$1-1}' | 
  feedgnuplot --lines --domain --timefmt %s --stream --exit

And the result looks something like this on my desktop computer:

Monitor activity in log files

Activity in log files can be a useful overall diagnostic tool as well. Unusual activity might result in much more lines being written to the log file per second, and something one needs to act on. With a little help from some classic Unix utilities, it is possible to plot number of lines written to a log file per second. I will try to explain each step below

Every time a file is written to a log file, we will only need to print a timestamp like:

tail -f -n 0 /var/log/syslog | gawk '{print systime()}

And the result will just be a lot of timestamps:

With uniq -c, we can count the number of timestamps that are not unique. The result will be something like this:

      3 1515099147
      6 1515099153
      6 1515099154
     12 1515099160
     12 1515099168
      6 1515099170
      6 1515099176
      6 1515099191

And then it is just about getting awk to reorder the two fields and feed the data to feedgnuplot. The command line should look something like this:

tail -f -n 0 /var/log/syslog | gawk '{print systime()} | uniq -c | awk '{print $2,$1}' |
  feedgnuplot --stream --exit --domain --timefmt %s --lines

And the result will look something like this after some time:

The spikes are great examples of abnormalities in my log file. In this case, they are a reflection of the iwlwifi-driver crashing, causing the logfile to be flooded with messages from the kernel.

Final words

So with a little bit of ingenuity and just a few commands, it is possible to make performance graphs of almost anything on a Linux system. Just imagine the number of interesting values you will be able to find in the /proc and /sys file system.

Feedgnuplot is of course not a replacement for graphing tools like PRTG or Graphviz. But I often have to troubleshoot on systems not connected to any useful graphing systems at all. Also: These graphing tools normally works at a low sample rate. With feedgnuplot, you can specify the sample rate as you wish, making it very useful for network troubleshooting.

In my next article I will show how to collect performance counters from headless systems and plot them with feedgnuplot. I will also demonstrate how feedgnuplot can be used for network troubleshooting in combination with tshark.