openstack 与 ceph (monitor初始化)

Monitor 节点初始化 执行下面步骤

操作系统初始化 当前环境中, ceph 存储集群统一部署 centos 7.1 操作系统, 并在 /etc/hosts 中登记集群中的服务器主机名, 并完成时间同步

主机名验证

[root@hh-yun-ceph-cinder015-128055 ~]# hostname hh-yun-ceph-cinder015-128055.vclound.com [root@hh-yun-ceph-cinder017-128057 ~]# hostname hh-yun-ceph-cinder017-128057.vclound.com [root@hh-yun-ceph-cinder024-128074 ~]# hostname hh-yun-ceph-cinder024-128074.vclound.com

cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 240.30.128.55 hh-yun-ceph-cinder015-128055.vclound.com hh-yun-ceph-cinder015-128055 240.30.128.56 hh-yun-ceph-cinder016-128056.vclound.com hh-yun-ceph-cinder016-128056 240.30.128.57 hh-yun-ceph-cinder017-128057.vclound.com hh-yun-ceph-cinder017-128057 240.30.128.73 hh-yun-ceph-cinder023-128073.vclound.com hh-yun-ceph-cinder023-128073 240.30.128.74 hh-yun-ceph-cinder024-128074.vclound.com hh-yun-ceph-cinder024-128074 240.30.128.75 hh-yun-ceph-cinder025-128075.vclound.com hh-yun-ceph-cinder025-128075 240.30.128.76 hh-yun-ceph-cinder026-128076.vclound.com hh-yun-ceph-cinder026-128076

时间同步验证

[root@hh-yun-ceph-cinder024-128074 ~]# chronyc tracking Reference ID : 10.199.129.21 (hh-yun-129021.vclound.com) Stratum : 12 Ref time (UTC) : Tue Jul 21 07:45:32 2015 System time : 0.000002526 seconds fast of NTP time Last offset : 0.000005405 seconds RMS offset : 0.007192081 seconds Frequency : 18.428 ppm slow Residual freq : 0.001 ppm Skew : 0.060 ppm Root delay : 0.000141 seconds Root dispersion : 0.011696 seconds Update interval : 386.4 seconds Leap status : Normal ceph 软件安装 注: 假如之前安装过 ceph, 并计划清除之前的集群配置, 那么执行下面命令可以把 ceph 配置进行安装 清除方法: ceph-deploy purgedata {node-name}, ceph-deploy purge {node-name}

yum 源

yum install -y epel-release.noarch echo "sslverify=0" >> /etc/yum.conf

ceph 软件安装

yum install -y ceph* bash-completion 创建配置文件, 默认是 /etc/ceph/ceph.conf vim /etc/ceph/ceph.conf [global] fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9 mon initial members = hh-yun-ceph-cinder015-128055, hh-yun-ceph-cinder017-128057, hh-yun-ceph-cinder024-128074 mon host = 240.30.128.55, 240.30.128.57, 240.30.128.74 public network = 240.30.128.0/21 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 filestore xattr use omap = true osd pool default size = 2 osd pool default min size = 1 osd pool default pg num = 333 osd pool default pgp num = 333 osd crush chooseleaf type = 1 创建集群密钥对创建监控密钥 [root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' creating /etc/ceph/ceph.mon.keyring [root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring [mon.] key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w== caps mon = "allow *"

注意: ceph-0.87 版本需要把文件 /etc/ceph/ceph.mon.keyring 重命名为 /etc/ceph/ceph.keyring 或 /etc/ceph/keyring 或 /etc/ceph/ceph.mon.(hostname).keyring 否则无法执行注册

生成管理员密钥, 生成 client.admin 用户并增加用户密钥对 [root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow' creating /etc/ceph/ceph.client.admin.keyring [root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring -rw------- 1 root root 137 7月 23 16:20 /etc/ceph/ceph.client.admin.keyring [root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.client.admin.keyring [client.admin] key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw== auid = 0 caps mds = "allow" caps mon = "allow *" caps osd = "allow *" 增加 client.admin 密钥对到 ceph.mon.keyring 中 [root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring [root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.mon.keyring -rw------- 1 root root 137 7月 23 16:20 /etc/ceph/ceph.client.admin.keyring -rw------- 1 root root 214 7月 23 16:20 /etc/ceph/ceph.mon.keyring

注意, 这里 /etc/ceph/ceph.mon.keyring 从之前的 77 字节变成 214 字节, 并自动添加如下的内容

[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring [mon.] key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w== caps mon = "allow *" [client.admin] key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw== auid = 0 caps mds = "allow" caps mon = "allow *" caps osd = "allow *" 使用主机名, IP 地址及 FSID 生成 monitor map 保存到 /etc/ceph/monmap 注意, 命令创建时候需要一次过把所有的 mon 节点都直接进行定义 [root@hh-yun-ceph-cinder015-128055 tmp]# monmaptool --create --add hh-yun-ceph-cinder015-128055 240.30.128.55 --add hh-yun-ceph-cinder017-128057 240.30.128.57 --add hh-yun-ceph-cinder024-128074 240.30.128.74 --fsid dc4f91c1-8792-4948-b68f-2fcea75f53b9 /etc/ceph/monmap monmaptool: monmap file /etc/ceph/monmap monmaptool: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9 monmaptool: writing epoch 0 to /etc/ceph/monmap (3 monitors) 在对应的监控主机上创建默认数据目录 [root@hh-yun-ceph-cinder015-128055 /]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055 [root@hh-yun-ceph-cinder017-128057 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057 [root@hh-yun-ceph-cinder024-128074 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074 定义监控进程与监控 MAP 及密钥对 [root@hh-yun-ceph-cinder015-128055 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder015-128055 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9 ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055 for mon.hh-yun-ceph-cinder015-128055

把 /etc/ceph 目录下的文件: ceph.conf ceph.client.admin.keyring ceph.mon.keyring monmap 同步到另外两台 mon 节点中

[root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder017-128057:/etc/ceph/. ceph.client.admin.keyring 100% 137 0.1KB/s 00:00 ceph.conf 100% 544 0.5KB/s 00:00 ceph.mon.keyring 100% 214 0.2KB/s 00:00 monmap 100% 550 0.5KB/s 00:00 [root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder024-128074:/etc/ceph/. ceph.client.admin.keyring 100% 137 0.1KB/s 00:00 ceph.conf 100% 544 0.5KB/s 00:00 ceph.mon.keyring 100% 214 0.2KB/s 00:00 monmap 100% 550 0.5KB/s 00:00

为另外两台 mon 节点进行监控数据初始化

[root@hh-yun-ceph-cinder017-128057 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder017-128057 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9 ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057 for mon.hh-yun-ceph-cinder017-128057 [root@hh-yun-ceph-cinder024-128074 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder024-128074 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9 ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074 for mon.hh-yun-ceph-cinder024-128074 分别启动三台 mon 节点 [root@hh-yun-ceph-cinder015-128055 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/sysvinit [root@hh-yun-ceph-cinder017-128057 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/sysvinit [root@hh-yun-ceph-cinder024-128074 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/sysvinit [root@hh-yun-ceph-cinder015-128055 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder015-128055 === mon.hh-yun-ceph-cinder015-128055 === Starting Ceph mon.hh-yun-ceph-cinder015-128055 on hh-yun-ceph-cinder015-128055... Running as unit run-53061.service. Starting ceph-create-keys on hh-yun-ceph-cinder015-128055... [root@hh-yun-ceph-cinder017-128057 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder017-128057 === mon.hh-yun-ceph-cinder017-128057 === Starting Ceph mon.hh-yun-ceph-cinder017-128057 on hh-yun-ceph-cinder017-128057... Running as unit run-52736.service. Starting ceph-create-keys on hh-yun-ceph-cinder017-128057... [root@hh-yun-ceph-cinder024-128074 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder024-128074 === mon.hh-yun-ceph-cinder024-128074 === Starting Ceph mon.hh-yun-ceph-cinder024-128074 on hh-yun-ceph-cinder024-128074... Running as unit run-51440.service. Starting ceph-create-keys on hh-yun-ceph-cinder024-128074...