对 journal 信息永久化保存方法

针对 systemd-journald.service 服务默认只把信息存放与内存进行优化, 改变当服务器长期启动, 无法利用 journald 命令获取服务状态信息 修改 /etc/systemd/journald.conf 添加 Storage=persistent 可利用系统文件对 journal 信息进行永久保存

优化 sysct.conf

/proc/sys/kernel/panic 10 (kernel panic 重启时间) /proc/sys/kernel/perf_event_max_sample_rate 100000 (提高 kernel 中断) /proc/sys/kernel/printk 7 4 1 7 (内核调试信息打印) /proc/sys/net/netfilter/nf_conntrack_max 4194304 (增加 iptables 链路) echo $[128*1024*1024*1024/16384/2] /proc/sys/net/netfilter/nf_conntrack_buckets 524288 (hash size) echo $[128*1024*1024*1024/131072/2] /proc/sys/vm/dirty_ratio 30 /proc/sys/vm/swappiness 10 /proc/sys/vm/overcommit_memory 2 (拒绝内存超配) /proc/sys/vm/max_map_count 可 > 65535 (某个进程可能使用的最大内存映射区域)

PCP 使用参考

参考: http://www.pcp.io/docs/guide.html

PCP主要用于对系统进程进行分析, 显示当前进程资源使用情况, 可以根据返回值对进程资源使用进行判断

查询可用项目 pminfo -h localhost

例子: 查询磁盘启动到现在的read 信息

[root@gx-yun-084036 .ssh]# pminfo -h localhost -dfmtT disk.partitions.read_bytes disk.partitions.read_bytes PMID: 60.10.6 [number of bytes read for storage partitions] Data Type: 32-bit unsigned int InDom: 60.10 0xf00000a Semantics: counter Units: Kbyte Help: Cumulative number of bytes read since system boot time (subject to counter wrap) for individual disk partitions or logical volumes. inst [0 or "sda1"] value 22367 inst [1 or "sda2"] value 592513 inst [2 or "sda3"] value 1424 inst [3 or "sdb1"] value 557470 inst [4 or "sdc1"] value 1568

持续地观察当前磁盘的读写状态

[root@gx-yun-084036 .ssh]# pmval -t 2sec -f 3 disk.partitions.write -h localhost metric: disk.partitions.write host: gx-yun-084036.vclound.com semantics: cumulative counter (converting to rate) units: count (converting to count / sec) samples: all sda1 sda2 sda3 sdb1 sdc1 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 2.498 0.000 0.500 0.500 0.000 2.498 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

另外一个例

[root@gx-yun-084036 .ssh]# pmdumptext -Xlimu -t 2sec 'kernel.all.load[1]' mem.util.used disk.partitions.write -h localhost [ 1] localhost:kernel.all.load["1 minute"] [ 2] localhost:mem.util.used [ 3] localhost:disk.partitions.write["sda1"] [ 4] localhost:disk.partitions.write["sda2"] [ 5] localhost:disk.partitions.write["sda3"] [ 6] localhost:disk.partitions.write["sdb1"] [ 7] localhost:disk.partitions.write["sdc1"] Column 1 2 3 4 5 6 7 Source localh localh localh localh localh localh localh Metric load used write write write write write Inst 1 minu n/a sda1 sda2 sda3 sdb1 sdc1 Units none b c/s c/s c/s c/s c/s Mon Sep 26 16:20:07 0.08 14.78G ? ? ? ? ? Mon Sep 26 16:20:09 0.08 14.78G 0.00 0.00 0.00 0.00 0.00 Mon Sep 26 16:20:11 0.08 14.78G 0.00 36.00 0.00 0.00 0.00 Mon Sep 26 16:20:13 0.07 14.78G 0.00 2.50 0.00 0.50 0.50 Mon Sep 26 16:20:15 0.07 14.78G 0.00 0.00 0.00 0.50 0.00 Mon Sep 26 16:20:17 0.07 14.78G 0.00 0.00 0.00 0.00 0.00

pcp atop 可实时监控当前资源(RHEL7.2 以上可用)

实时监控系统资源

[root@gx-yun-084036 .ssh]# pmcollectl #<---------CPU---------><------------Disks----------><--------Network----------> #cpu sys inter ctxsw KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut 0 0 888 1773 0 0 8 6 54 81 7 73 0 0 620 1712 0 0 4 3 55 96 7 88 0 0 736 1539 0 0 0 0 187 229 32 208 0 0 647 1222 0 0 0 0 210 206 18 159 0 0 555 1061 0 0 64 2 61 90 10 83 0 0 432 840 0 0 0 0 84 110 11 101 0 0 511 1025 0 0 0 0 7 49 7 49

stap 监控参考

stap 一个不错的系统监控工具, 能够满足对进程的监控

yum install -y systemtap*

其中 systemtap-client 软件包中包含了一些常用的管理脚本, 当然, 可以自行进行编程实现对系统的监控显示 需要安装下面软件包才可以满足 stap 命令使用

rpm -ivh kernel-debuginfo-3.10.0-327.el7.x86_64.rpm kernel-debuginfo-common-x86_64-3.10.0-327.el7.x86_64.rpm

当前系统进程 IO 最猛的几个进程

iostop.stp

#!/usr/bin/stap global io_stat,device global read_bytes,write_bytes probe vfs.read.return { if ($return>0) { if (devname!="N/A") {/*skip read from cache*/ io_stat[pid(),execname(),uid(),ppid(),"R"] += $return device[pid(),execname(),uid(),ppid(),"R"] = devname read_bytes += $return } } } probe vfs.write.return { if ($return>0) { if (devname!="N/A") { /*skip update cache*/ io_stat[pid(),execname(),uid(),ppid(),"W"] += $return device[pid(),execname(),uid(),ppid(),"W"] = devname write_bytes += $return } } } probe timer.ms(5000) { /* skip non-read/write disk */ if (read_bytes+write_bytes) { printf("\n%-25s, %-8sMKb/sec, %-7smKb, %-7smKb\n\n", ctime(gettimeofday_s()), "Average:", ((read_bytes+write_bytes)/1024)/5, "Read:",read_bytes/1024, "Write:",write_bytes/1024) /* print header */ printf("%8s %8s %8s %s %8s %4s s\n", "UID","PID","PPID","CMD","DEVICE","T","BYTES") } /* print top ten I/O */ foreach ([process,cmd,userid,parent,action] in io_stat- limit 10) printf("