Linux集群和自动化维2.6.4　开发类脚本

xiaoxiao2024-01-23 27

2.6.4　开发类脚本

业务需求在不断地变化，有时候互联网上的开源方案并不能全部解决，这个时候就需要自己写一些开发类的脚本来满足工作中的需求了，虽然很多时候脚本都可以独立运行，但笔者的做法还是尽量将其return结果写成Nagios能够识别的格式，以便配合Nagios发送报警邮件和信息。

1.监测redis是否正常运行

笔者接触的线上NoSQL业务主要是redis数据库，多用于处理大量数据的高访问负载需求。为了最大化地利用资源，每个redis实例分配的内存并不是很大，有时候程序组的同事导入数据量大的IP list时会导致redis实例崩溃，所以笔者开发了一个redis监测脚本并配合Nagios进行工作，脚本内容如下所示（此脚本在Amazon Linux AMI x86_64下已测试通过）：

#!/usr/bin/python

#Check redis Nagios Plungin,Please install the redis-py module.

import redis

import sys

STATUS_OK = 0

STATUS_WARNING = 1

STATUS_CRITICAL = 2

HOST = sys.argv[1]

PORT = int(sys.argv[2])

WARNING = float(sys.argv[3])

CRITICAL = float(sys.argv[4])

def connect_redis(host, port):

r = redis.Redis(host, port, socket_timeout = 5, socket_connect_timeout = 5)

return r

def main():

r = connect_redis(HOST, PORT)

try:

r.ping()

except:

print HOST,PORT,'down'

sys.exit(STATUS_CRITICAL)

redis_info = r.info()

used_mem = redis_info['used_memory']/1024/1024/1024.0

used_mem_human = redis_info['used_memory_human']

if WARNING <= used_mem < CRITICAL:

print HOST,PORT,'use memory warning',used_mem_human

sys.exit(STATUS_WARNING)

elif used_mem >= CRITICAL:

print HOST,PORT,'use memory critical',used_mem_human

sys.exit(STATUS_CRITICAL)

else:

print HOST,PORT,'use memory ok',used_mem_human

sys.exit(STATUS_OK)

if __name__ == '__main__':

main()

2.监测机器的IP连接数

需求其实比较简单，先统计IP连接数，如果ip_conns值小于15 000则显示为正常，介于15 000至20 000之间为警告，如果超过20 000则报警，脚本内容如下所示（此脚本在Amazon Linux AMI x86_64下已测试通过）：

#!/bin/bash

#Nagios plugin For ip connects

#$1 = 15000 $2 = 20000

ip_conns=`netstat -an | grep tcp | grep EST | wc -l`

messages=`netstat -ant | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'|tr -s '\n' ',' | sed -r 's/(.*),/\1\n/g' `

if [ $ip_conns -lt $1 ]

then

echo "$messages,OK -connect counts is $ip_conns"

exit 0

if [ $ip_conns -gt $1 -a $ip_conns -lt $2 ]

then

echo "$messages,Warning -connect counts is $ip_conns"

exit 1

if [ $ip_conns -gt $2 ]

then

echo "$messages,Critical -connect counts is $ip_conns"

exit 2

3.监测机器的CPU利用率脚本

线上的bidder业务机器，在业务繁忙的高峰期会出现CPU利用率达到100%（sys%+ user%），导致后面的流量打在上面却完全进不去的情况，但此时机器、系统负载及Nginx+Lua进程都是完全正常的，所以这种情况下需要开发一个CPU利用率脚本，在超过自定义阀值时报警，方便运维人员批量添加bidder AMI机器以应对峰值，AWS EC2实例机器是可以以小时来计费的，大家在这里也要注意分清系统负载和CPU利用率之间的区别。脚本内容如下所示（此脚本在Amazon Linux AMI x86_64下已测试通过）：

#!/bin/bash

# ==============================================================================

# CPU Utilization Statistics plugin for Nagios

# USAGE : ./check_cpu_utili.sh [-w <user,system,iowait>] [-c <user,system,iowait>] ( [ -i <intervals in second> ] [ -n <report number> ])

# Exemple: ./check_cpu_utili.sh

# ./check_cpu_utili.sh -w 70,40,30 -c 90,60,40

# ./check_cpu_utili.sh -w 70,40,30 -c 90,60,40 -i 3 -n 5

#-------------------------------------------------------------------------------

# Paths to commands used in this script. These may have to be modified to match your system setup.

IOSTAT="/usr/bin/iostat"

# Nagios return codes

STATE_OK=0

STATE_WARNING=1

STATE_CRITICAL=2

STATE_UNKNOWN=3

# Plugin parameters value if not define

LIST_WARNING_THRESHOLD="70,40,30"

LIST_CRITICAL_THRESHOLD="90,60,40"

INTERVAL_SEC=1

NUM_REPORT=1

# Plugin variable description

PROGNAME=$(basename $0)

if [ ! -x $IOSTAT ]; then

echo "UNKNOWN: iostat not found or is not executable by the nagios user."

exit $STATE_UNKNOWN

print_usage() {

echo ""

echo "$PROGNAME $RELEASE - CPU Utilization check script for Nagios"

echo ""

echo "Usage: check_cpu_utili.sh -w -c (-i -n)"

echo ""

echo " -w Warning threshold in % for warn_user,warn_system,warn_iowait CPU (default : 70,40,30)"

echo " Exit with WARNING status if cpu exceeds warn_n"

echo " -c Critical threshold in % for crit_user,crit_system,crit_iowait CPU (default : 90,60,40)"

echo " Exit with CRITICAL status if cpu exceeds crit_n"

echo " -i Interval in seconds for iostat (default : 1)"

echo " -n Number report for iostat (default : 3)"

echo " -h Show this page"

echo ""

echo "Usage: $PROGNAME"

echo "Usage: $PROGNAME --help"

echo ""

exit 0

}

print_help() {

print_usage

echo ""

echo "This plugin will check cpu utilization (user,system,CPU_Iowait in %)"

echo ""

exit 0

}

# Parse parameters

while [ $# -gt 0 ]; do

case "$1" in

-h | --help)

print_help

exit $STATE_OK

;;

-v | --version)

print_release

exit $STATE_OK

;;

-w | --warning)

shift

LIST_WARNING_THRESHOLD=$1

;;

-c | --critical)

shift

LIST_CRITICAL_THRESHOLD=$1

;;

-i | --interval)

shift

INTERVAL_SEC=$1

;;

-n | --number)

shift

NUM_REPORT=$1

;;

*) echo "Unknown argument: $1"

print_usage

exit $STATE_UNKNOWN

;;

esac

shift

done

# List to Table for warning threshold (compatibility with

TAB_WARNING_THRESHOLD=(`echo $LIST_WARNING_THRESHOLD | sed 's/,/ /g'`)

if [ "${#TAB_WARNING_THRESHOLD[@]}" -ne "3" ]; then

echo "ERROR : Bad count parameter in Warning Threshold"

exit $STATE_WARNING

else

USER_WARNING_THRESHOLD=`echo ${TAB_WARNING_THRESHOLD[0]}`

SYSTEM_WARNING_THRESHOLD=`echo ${TAB_WARNING_THRESHOLD[1]}`

IOWAIT_WARNING_THRESHOLD=`echo ${TAB_WARNING_THRESHOLD[2]}`

# List to Table for critical threshold

TAB_CRITICAL_THRESHOLD=(`echo $LIST_CRITICAL_THRESHOLD | sed 's/,/ /g'`)

if [ "${#TAB_CRITICAL_THRESHOLD[@]}" -ne "3" ]; then

echo "ERROR : Bad count parameter in CRITICAL Threshold"

exit $STATE_WARNING

else

USER_CRITICAL_THRESHOLD=`echo ${TAB_CRITICAL_THRESHOLD[0]}`

SYSTEM_CRITICAL_THRESHOLD=`echo ${TAB_CRITICAL_THRESHOLD[1]}`

IOWAIT_CRITICAL_THRESHOLD=`echo ${TAB_CRITICAL_THRESHOLD[2]}`

if [ ${TAB_WARNING_THRESHOLD[0]} -ge ${TAB_CRITICAL_THRESHOLD[0]} -o ${TAB_WARNING_THRESHOLD[1]} -ge ${TAB_CRITICAL_THRESHOLD[1]} -o ${TAB_WARNING_THRESHOLD[2]} -ge ${TAB_CRITICAL_THRESHOLD[2]} ]; then

echo "ERROR : Critical CPU Threshold lower as Warning CPU Threshold "

exit $STATE_WARNING

CPU_REPORT=`iostat -c $INTERVAL_SEC $NUM_REPORT | sed -e 's/,/./g' | tr -s ' ' ';' | sed '/^$/d' | tail -1`

CPU_REPORT_SECTIONS=`echo ${CPU_REPORT} | grep ';' -o | wc -l`

CPU_USER=`echo $CPU_REPORT | cut -d ";" -f 2`

CPU_SYSTEM=`echo $CPU_REPORT | cut -d ";" -f 4`

CPU_IOWAIT=`echo $CPU_REPORT | cut -d ";" -f 5`

CPU_STEAL=`echo $CPU_REPORT | cut -d ";" -f 6`

CPU_IDLE=`echo $CPU_REPORT | cut -d ";" -f 7`

NAGIOS_STATUS="user=${CPU_USER}%,system=${CPU_SYSTEM}%,iowait=${CPU_IOWAIT}%,idle=${CPU_IDLE}%"

NAGIOS_DATA="CpuUser=${CPU_USER};${TAB_WARNING_THRESHOLD[0]};${TAB_CRITICAL_THRESHOLD[0]};0"

CPU_USER_MAJOR=`echo $CPU_USER| cut -d "." -f 1`

CPU_SYSTEM_MAJOR=`echo $CPU_SYSTEM | cut -d "." -f 1`

CPU_IOWAIT_MAJOR=`echo $CPU_IOWAIT | cut -d "." -f 1`

CPU_IDLE_MAJOR=`echo $CPU_IDLE | cut -d "." -f 1`

# Return

if [ ${CPU_USER_MAJOR} -ge $USER_CRITICAL_THRESHOLD ]; then