Taking thread dump on CPU High usage in WSO2 products

Please follow the commands provided here to take the thread dump.

 

First you need to create test.sh script using following command.
     vi test.sh

Then paste following script content.

    #!/bin/bash
    # 1: ['command\ name' or PID number(,s)] 2: MAX_CPU_PERCENT
    [[ $# -ne 2 ]] && exit 1
    PID_NAMES=$1
    # get all PIDS as nn,nn,nn
    if [[ ! "$PID_NAMES" =~ ^[0-9,]+$ ]] ; then
        PIDS=$(pgrep -d ',' -x $PID_NAMES)
    else
        PIDS=$PID_NAMES
    fi
    # echo "$PIDS $MAX_CPU"
    MAX_CPU="$2"
    MAX_CPU="$(echo "($MAX_CPU+0.5)/1" | bc)"
    LOOP=1
    while [[ $LOOP -eq 1 ]] ; do
        sleep 0.3s
        # Depending on your 'top' version and OS you might have
        # to change head and tail line-numbers
        LINE="$(top -b -d 0 -n 1 -p $PIDS | head -n 8 \
            | tail -n 1 | sed -r 's/[ ]+/,/g' | \
            sed -r 's/^\,|\,$//')"
        # If multiple processes in $PIDS, $LINE will only match\
        # the most active process
        CURR_PID=$(echo "$LINE" | cut -d ',' -f 1)
        # calculate cpu limits
        CURR_CPU_FLOAT=$(echo "$LINE"| cut -d ',' -f 9)
        CURR_CPU=$(echo "($CURR_CPU_FLOAT+0.5)/1" | bc)
        echo "PID $CURR_PID: $CURR_CPU""%"
        if [[ $CURR_CPU -ge $MAX_CPU ]] ; then
            now="$(date)"
            echo "PID $CURR_PID ($PID_NAMES) went over $MAX_CPU""%" on $now
            jstack $CURR_PID > ./$now+jlog.txt
            echo "[[ $CURR_CPU""% -ge $MAX_CPU""% ]]"
            LOOP=0
            break
        fi
    done
    echo "Stopped"


Then we need to get process id of running WSO2 server by running following command.
    sanjeewa@sanjeewa-ThinkPad-T530:~/work$ jps
    30755 Bootstrap
    8543 Jps
    4892 Main


Now we know carbon server running with process ID 30755. Then we can start our script by providing init parameters(process ID and CPU limit). So it will keep printing CPU usage in terminal and once it reached limit it will take thread dump using jstack command. It will create new file with with embedding current date time and push Jstack output to it.

We can start scritp like this.
    sanjeewa@sanjeewa-ThinkPad-T530:~/work$ sh test.sh 30755 99

As you can see when CPU goes above 99% it will create log file and append thread dump.