前提

配置 nova compute, 令其他与 ceph 集群进行连接 最终目的能够允许 instances 利用 ceph RBD 当作外部卷使用 nova compute 默认已经能够正常工作, 当前只添加 ceph 连接部分

安装软件

yum install -y python-ceph ceph

额外的目录

mkdir -p /var/run/ceph/guests/ /var/log/qemu/ chown qemu:qemu /var/run/ceph/guests /var/log/qemu/

确认版本

确保 qemu 支持 rbd 协议, 利用 ldd 检查是否支持 librbd.so 动态库

[root@hh-yun-compute-130133 ~]# ldd /usr/bin/qemu-img | grep rbd librbd.so.1 => /lib64/librbd.so.1 (0x00007fa708216000)

也可以通过命令行进行检测

[root@hh-yun-compute-130133 ~]# qemu-img -h | grep rbd Supported formats: vvfat vpc vmdk vhdx vdi sheepdog rbd raw host_cdrom host_floppy host_device file qed qcow2 qcow parallels nbd iscsi gluster dmg cloop bochs blkverify blkdebug

当前使用的版本

qemu-img-1.5.3-86.el7_1.2.x86_64 qemu-kvm-1.5.3-86.el7_1.2.x86_64 qemu-kvm-common-1.5.3-86.el7_1.2.x86_64

配置

修改 nova compute 下的 ceph.conf

[global] fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9 mon initial members = XXX.XXXX.XXXXX mon host = XXX.XXX.XXX public network = XXX.XXX.XXX.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 filestore xattr use omap = true osd pool default size = 3 osd pool default min size = 1 osd pool default pg num = 10240 osd pool default pgp num = 10240 osd crush chooseleaf type = 1 [client] rbd cache = true rbd cache writethrough until flush = true admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok log file = /var/log/qemu/qemu-guest-$pid.log rbd concurrent management ops = 20

修改 nova compute 连接 ceph 方法配置

[libvirt] libvirt_images_type = rbd libvirt_images_rbd_pool = volumes libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf libvirt_disk_cachemodes="network=writeback" rbd_user = cinder rbd_secret_uuid = dc4f91c1-8792-4948-b68f-2fcea75f53b9 libvirt_inject_password = false libvirt_inject_key = false libvirt_inject_partition = -2 libvirt_live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"

复制 ceph 服务器中 ceph.client.cinder.keyring 到 /etc/ceph 目录下 (参考上面 rbd_user=cinder)

添加 ceph 配置信息

/etc/libvirt/secrets/dc4f91c1-8792-4948-b68f-2fcea75f53b9.base64 (连接 ceph 的用户密钥定义)

AQADxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==

/etc/libvirt/secrets/dc4f91c1-8792-4948-b68f-2fcea75f53b9.xml (连接 ceph 的用户定义)

<secret ephemeral="no" private="no"> <uuid>dc4f91c1-8792-4948-b68f-2fcea75f53b9</uuid> <usage type="ceph"> <name>client.volumes secret</name> </usage> </secret>

重启服务

分别重启下面两个服务

systemctl restart libvirtd systemctl restart openstack-nova-compute