How monitor WSO2 server CPU usage and generate thread dump on high CPU usage using simple shell script

When we deployed WSO2 servers in production deployments we may need to monitor them for high CPU and memory usages. So in this article i will describe how we can use simple shell script to monitor server CPU usage and generate thread dump using jstack command.

First you need to create test.sh script using following command.

 vi test.sh 

Then paste following script content.

#!/bin/bash
# 1: ['command\ name' or PID number(,s)] 2: MAX_CPU_PERCENT
[[ $# -ne 2 ]] && exit 1
PID_NAMES=$1
# get all PIDS as nn,nn,nn
if [[ ! "$PID_NAMES" =~ ^[0-9,]+$ ]] ; then
    PIDS=$(pgrep -d ',' -x $PID_NAMES)
else
    PIDS=$PID_NAMES
fi
#  echo "$PIDS $MAX_CPU"
MAX_CPU="$2"
MAX_CPU="$(echo "($MAX_CPU+0.5)/1" | bc)"
LOOP=1
while [[ $LOOP -eq 1 ]] ; do
    sleep 0.3s
    # Depending on your 'top' version and OS you might have
    #   to change head and tail line-numbers
    LINE="$(top -b -d 0 -n 1 -p $PIDS | head -n 8 \
        | tail -n 1 | sed -r 's/[ ]+/,/g' | \
        sed -r 's/^\,|\,$//')"
    # If multiple processes in $PIDS, $LINE will only match\
    #   the most active process
    CURR_PID=$(echo "$LINE" | cut -d ',' -f 1)
    # calculate cpu limits
    CURR_CPU_FLOAT=$(echo "$LINE"| cut -d ',' -f 9)
    CURR_CPU=$(echo "($CURR_CPU_FLOAT+0.5)/1" | bc)
    echo "PID $CURR_PID: $CURR_CPU""%"
    if [[ $CURR_CPU -ge $MAX_CPU ]] ; then
        now="$(date)"
        echo "PID $CURR_PID ($PID_NAMES) went over $MAX_CPU""%" on $now
        jstack $CURR_PID > ./$now+jlog.txt
        echo "[[ $CURR_CPU""% -ge $MAX_CPU""% ]]"
        LOOP=0
        break
    fi
done
echo "Stopped"

Then we need to get process id of running WSO2 server by running following command.

sanjeewa@sanjeewa-ThinkPad-T530:~/work$ jps
30755 Bootstrap
8543 Jps
4892 Main

Now we know carbon server running with process ID 30755. Then we can start our script by providing init parameters(process ID and CPU limit). So it will keep printing CPU usage in terminal and once it reached limit it will take thread dump using jstack command. It will create new file with with embedding current date time and push Jstack output to it.

We can start scritp like this.

 sh test.sh <processId> <CPU Limit>

sanjeewa@sanjeewa-ThinkPad-T530:~/work$ sh test.sh 30755 10
PID 30755: 0%
PID 30755: 0%
PID 30755: 0%
PID 30755: 0%
PID 30755 (30755) went over 10% on 2015 පෙබරවාරි 19 වැනි බ්‍රහස්පතින්දා 14:44:55 +0530
[[ 13% -ge 10% ]]
Stopped

As you can see when CPU goes above 10% it will create log file and append thread dump.