参考 ceph 环境
cluster dc4f91c1-8792-4948-b68f-2fcea75f53b9 health HEALTH_WARN 15 requests are blocked > 32 sec; clock skew detected on mon.hh-yun-ceph-cinder025-128075 monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0 ,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128 076=240.30.128.76:6789/0}, election epoch 168, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-y un-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076 osdmap e27430: 100 osds: 100 up, 100 in pgmap v11224834: 20544 pgs, 2 pools, 70255 GB data, 17678 kobjects 205 TB used, 157 TB / 363 TB avail 20540 active+clean 4 active+clean+scrubbing+deep client io 57251 kB/s rd, 44602 kB/s wr, 3797 op/s参考 ceph health detail 返回结果
1. mon.hh-yun-ceph-cinder025-128075 addr 240.30.128.75:6789/0 clock skew 0.919947s > max 0.05s (latency 0.000544046s) 2. 15 requests are blocked这里是具有两个常见错误 1. 时间不同步导致 mon 报警 2. 由于有硬件故障, 网络延时, 或其他原因导致客户端访问 ceph 存储超时
问题解决 (时间同步) 当前系统中的环境设定
[root@hh-yun-ceph-cinder015-128055 ceph]# ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep clock "mon_clock_drift_allowed": "0.05", <- 当 mon 时间偏移 0.05 秒则不正常 "mon_clock_drift_warn_backoff": "5", <- 当出现 5 次偏移, 则报警 "clock_offset": "0", <- mon 节点的时间偏移默认值问题定位 检测各个机器的时间, 发现 hh-yun-ceph-cinder025-128075 节点时间偏移 修正方法
systemctl stop chronyd ntpdate 10.199.129.21 systemctl start chronyd当同步了时间并验证后, 需重启 mon 节点
/etc/init.d/ceph stop mon /etc/init.d/ceph start mon因为 mon 节点与客户非常连接, 因此, 在确保 mon 节点具有冗余情况下, 可以在生产时间段进行快速重启
问题解决 (15 requests are blocked)
参考信息
ceph health detail HEALTH_WARN 14 requests are blocked > 32 sec; 11 osds have slow requests 7 ops are blocked > 536871 sec 2 ops are blocked > 268435 sec 2 ops are blocked > 67108.9 sec 3 ops are blocked > 33554.4 sec 1 ops are blocked > 536871 sec on osd.0 1 ops are blocked > 536871 sec on osd.10 2 ops are blocked > 536871 sec on osd.12 2 ops are blocked > 268435 sec on osd.18 1 ops are blocked > 536871 sec on osd.31 1 ops are blocked > 536871 sec on osd.38 1 ops are blocked > 67108.9 sec on osd.38 1 ops are blocked > 33554.4 sec on osd.48 1 ops are blocked > 67108.9 sec on osd.52 1 ops are blocked > 536871 sec on osd.63 1 ops are blocked > 33554.4 sec on osd.64 1 ops are blocked > 33554.4 sec on osd.69 11 osds have slow requests上述信息, 发现, 访问被 block 而且时间很久, 解决方法
对上述 osd 进行一个一个的重启, 切记, 一个重启后, 数据 recovery 正常后才可以执行下一次的 osd 重启 /etc/init.d/ceph stop osd.0 /etc/init.d/ceph start osd.0 待数据恢复后才能够执行下一个 OSD 重启恢复后解决
cluster dc4f91c1-8792-4948-b68f-2fcea75f53b9 health HEALTH_OK monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0 ,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128 076=240.30.128.76:6789/0}, election epoch 170, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-y un-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076 osdmap e27495: 100 osds: 100 up, 100 in pgmap v11231620: 20544 pgs, 2 pools, 70294 GB data, 17688 kobjects 206 TB used, 157 TB / 363 TB avail 20539 active+clean 5 active+clean+scrubbing+deep client io 973 kB/s rd, 22936 kB/s wr, 1334 op/s 相关资源：ceph心跳丢失问题分析