ceph monitor功能的兼容性管理

xiaoxiao2024-05-29 104

软件需要兼容旧版本

ceph是在一个不断进化的软件，会增加功能也会取消旧的功能，如何在ceph monitor的不同版本间保持兼容或者防止不兼容而产生错误，是需要认真思考的问题。而我们阅读代码时往往忽略了兼容性管理的代码，而重点关心它的业务代码，所以这里特别写一篇文章谈谈兼容性的管理。

ceph monitor兼容主要包括两方面:

通讯时检查对方的兼容性代码访问本机数据时的兼容性

Messenger的创建和功能位features的设置

ceph使用原生的Messenger的方式来通讯，在通讯开始前需要创建一个Messenger, 例如ceph monitor创建monitor之间通讯的messenger:

Messenger *msgr = Messenger::create(g_ceph_context, g_conf->ms_type, entity_name_t::MON(rank), "mon", 0);

ceph在建立连接时会告诉对方自己是什么类型的节点，主要有几种类型: monitor, osd, mds, client，上面的代码就说明自己是一个Monitor,而任何一个Messenger都会准对某一种节点设置一个Policy, Policy的内容与兼容有关的是：

/// Specify features supported locally by the endpoint. uint64_t features_supported; /// Specify features any remotes must have to talk to this endpoint. int64_t features_required;

其中features_support表示本节点支持的功能，features_required表示对方必须具备的功能,每个功能一个bit位。缺省的Policy把featuires_supported设置成当前代码支持的全部功能。即：CEPH_FEATURES_ALL。

而ceph为monitor之间设置的Policy:

msgr->set_policy(entity_name_t::TYPE_MON, Messenger::Policy::lossless_peer_reuse( supported, CEPH_FEATURE_UID | CEPH_FEATURE_MON_SINGLE_PAXOS));

初始设置的features_required仅仅包含了CEPH_FEATUIRE_UID和CEPH_FEATURE_MON_SINGLE_PAXOS，而features_supported则是全部功能, monitor之间的兼容性是在随后的通讯过程中逐渐被检测的。

通讯时的兼容性检查

在连接建立时就检查对方的功能位设置，在ceph messenger通讯协议中，双方提供给对方支持的功能集，并且与本地Policy中设置的必需的功能位进行比较，例如:

ceph的simple messenger中，connect发起方会提供supported features:

while (1) { delete authorizer; authorizer = msgr->get_authorizer(peer_type, false); bufferlist authorizer_reply; ceph_msg_connect connect; connect.features = policy.features_supported;

当接收到应答后，又会验证对方支持的功能:

if (reply.tag == CEPH_MSGR_TAG_READY || reply.tag == CEPH_MSGR_TAG_SEQ) { uint64_t feat_missing = policy.features_required & ~(uint64_t)reply.features; if (feat_missing) { ldout(msgr->cct,1) << "missing required features " << std::hex << feat_missing << std::dec << dendl; goto fail_locked; }

查看reply的feature与本地必需的功能，如果缺少就会失败。

一些内部兼容性的表示方法

ceph使用一个CompatSet的数据结构来表示功能集合:

struct CompatSet { struct Feature { uint64_t id; string name; Feature(uint64_t _id, const char *_name) : id(_id), name(_name) {} Feature(uint64_t _id, const string& _name) : id(_id), name(_name) {} }; struct FeatureSet { uint64_t mask; map <uint64_t,string> names; }; FeatureSet compat; FeatureSet ro_compat; FeatureSet incompat; };

mask中的每一位代表代表一个功能, 兼容测试主要判断是否可读可写。测试是否可读是通过readable成员函数来实现:

bool readable(CompatSet const& other) const { return !((other.incompat.mask ^ incompat.mask) & other.incompat.mask); }

这个函数的意思是如果我的incompat不能全部包含对方的位域，我就无法读取对方数据。

测试是否可写是用writable成员函数来实现:

bool writeable(CompatSet const& other) const { return readable(other) && !((other.ro_compat.mask ^ ro_compat.mask) & other.ro_compat.mask); }

这个函数的意思就是：除了readable，我在ro_compat全部的位域包含了对方的位域才能写数据。

ceph monitor的内部兼容的保护

每个Monitor被创建时，都需要初始化本地数据，其中mkfs函数是被调用的重要一环。首先Monitor::mkfs会在本地写入一个兼容性集合，用以记录用当前代码生成这些数据库的时候，具备什么功能： int Monitor::mkfs() { MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction); // verify cluster fsid int r = check_fsid(); if (r < 0 && r != -ENOENT) return r; bufferlist magicbl; magicbl.append(CEPH_MON_ONDISK_MAGIC); magicbl.append("\n"); t->put(MONITOR_NAME, "magic", magicbl); features = get_initial_supported_features(); <<<<============== write_features(t); 当Monitor启动时，代码检查本地文件系统上的数据结构是否兼容当前代码，注意因为本地文件不被修改的情况下，ceph程序依然可以被升级或者用其他方法替换，所以程序启动检查本地数据是否兼容是必须的, ceph_mon.cc会调用check_features(),它检查本地数据文件格式是否和当前代码兼容： int Monitor::check_features(MonitorDBStore *store) { CompatSet required = get_supported_features(); CompatSet ondisk; read_features_off_disk(store, &ondisk); if (!required.writeable(ondisk)) { CompatSet diff = required.unsupported(ondisk); generic_derr << "ERROR: on disk data includes unsupported features: " << diff << dendl; return -EPERM; } return 0; }

而get_supported_features()就是当前Monitor代码能支持的所有功能，read_features_off_disk()则把write_features()的数据读出来，我们看到它用writable（）测试当前代码是否有能力可以写本地文件系统上的数据。

read_features_off_disk读出write_feature()生成的数据：

void Monitor::read_features_off_disk(MonitorDBStore *store, CompatSet *features) { bufferlist featuresbl; store->get(MONITOR_NAME, COMPAT_SET_LOC, featuresbl); if (featuresbl.length() == 0) { generic_dout(0) << "WARNING: mon fs missing feature list.\n" << "Assuming it is old-style and introducing one." << dendl; //we only want the baseline ~v.18 features assumed to be on disk. //If new features are introduced this code needs to disappear or //be made smarter. *features = get_legacy_features(); bufferlist bl; features->encode(bl); MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction); t->put(MONITOR_NAME, COMPAT_SET_LOC, bl); store->apply_transaction(t); } else { bufferlist::iterator it = featuresbl.begin(); features->decode(it); } }

作为一种特殊情况，如果数据是旧版的ceph monitor生成的，因为旧版没有写features到本地文件系统，所以read_features_off_disk会调用get_legacy_features()函数得到旧版本ceph monitor的功能集，这只是一个简单的构造:

CompatSet Monitor::get_legacy_features() { CompatSet::FeatureSet ceph_mon_feature_compat; CompatSet::FeatureSet ceph_mon_feature_ro_compat; CompatSet::FeatureSet ceph_mon_feature_incompat; ceph_mon_feature_incompat.insert(CEPH_MON_FEATURE_INCOMPAT_BASE); return CompatSet(ceph_mon_feature_compat, ceph_mon_feature_ro_compat, ceph_mon_feature_incompat); } Monitor运行时检查和设置features

一旦ceph_mon.cc决定运行Monitor, 首先会调用成员函数preinit(), 而preinit的一项工作就时调用read_features()把本地文件中记录的feature读入到成员变量features中:

void Monitor::read_features() { read_features_off_disk(store, &features); dout(10) << "features " << features << dendl; apply_compatset_features_to_quorum_requirements(); dout(10) << "required_features " << required_features << dendl; }

当然它不会忘记按照本地数据中保存的功能位，要求monitor paxos集群的所有法人必须有对应的功能位：

void Monitor::apply_compatset_features_to_quorum_requirements() { required_features = 0; if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_OSD_ERASURE_CODES)) { required_features |= CEPH_FEATURE_OSD_ERASURE_CODES; } if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_OSDMAP_ENC)) { required_features |= CEPH_FEATURE_OSDMAP_ENC; } if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_ERASURE_CODE_PLUGINS_V2)) { required_features |= CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2; } dout(10) << __func__ << " required_features " << required_features << dendl; }

required_features的设置，可以防止不兼容的Monitor构成一个paxos集群，有几个地方通过required_features阻断这些不兼容的monitor之间的通讯:

收到一个探测包，发现对方不能提供相关的功能位，则阻断通讯:

void Monitor::handle_probe_probe(MMonProbe *m) { MMonProbe *r; dout(10) << "handle_probe_probe " << m->get_source_inst() << *m << " features " << m->get_connection()->get_features() << dendl; uint64_t missing = required_features & ~m->get_connection()->get_features(); if (missing) { dout(1) << " peer " << m->get_source_addr() << " missing features " << missing << dendl; if (m->get_connection()->has_feature(CEPH_FEATURE_OSD_PRIMARY_AFFINITY)) { MMonProbe *r = new MMonProbe(monmap->fsid, MMonProbe::OP_MISSING_FEATURES, name, has_ever_joined); m->required_features = required_features; m->get_connection()->send_message(r); } goto out; }

获取数据复制的cookie时的，发现对方不能提供相关的功能位，阻断通讯：

void Monitor::handle_sync_get_cookie(MMonSync *m) { if (is_synchronizing()) { _sync_reply_no_cookie(m); return; } assert(g_conf->mon_sync_provider_kill_at != 1); // make sure they can understand us. if ((required_features ^ m->get_connection()->get_features()) & required_features) { <<======================= dout(5) << " ignoring peer mon." << m->get_source().num() << " has features " << std::hex << m->get_connection()->get_features() << " but we require " << required_features << std::dec << dendl; return; } paxos集群形成时的功能集兼容

一个接收到要求投票的请求的选举器,检查功能位是否兼容：通过获得当前Monitor对兼容性的要求和对方能支持的功能集的比较来决定是否还要继续：

void Elector::handle_propose(MMonElection *m) { ... uint64_t required_features = mon->get_required_features(); dout(10) << __func__ << " required features " << required_features << ", peer features " << m->get_connection()->get_features() << dendl; if ((required_features ^ m->get_connection()->get_features()) & required_features) { dout(5) << " ignoring propose from mon" << from << " without required features" << dendl; nak_old_peer(m); return;

一个选举器在接收到选举应答时，检查功能位是否兼容：通过获得当前Monitor对兼容性的要求和对方能支持的功能集的比较来决定是否还要继续：

void Elector::handle_ack(MMonElection *m) { ... uint64_t required_features = mon->get_required_features(); if ((required_features ^ m->get_connection()->get_features()) & required_features) { dout(5) << " ignoring ack from mon" << from << " without required features" << dendl; m->put(); return; }

一个提出选举的Monitor，会在选举过程中收集与各个monitor连接时对方提供的features, 记录在案，在获胜后，会求出这些Monitor共同支持的功能集:

void Elector::victory() { leader_acked = -1; electing_me = false; uint64_t features = CEPH_FEATURES_ALL; set<int> quorum; for (map<int, uint64_t>::iterator p = acked_me.begin(); p != acked_me.end(); ++p) { quorum.insert(p->first); features &= p->second; } ... mon->win_election(epoch, quorum, features, cmds, cmdsize, ©_classic_mons);

最后得到的features变量包含这些monitor共同支持的集合，同时把这个features传给Monitor类记录在案。

而Monitor::win_election把features保存在qurum_features后调用finish_election, finish_election调用apply_quorum_to_compatset_features(), apply_quorum_to_compatset_features就是把paxos集群中的monitor的共同的功能集合保存在本地文件中，以备下次ceph mon启动时通过read_features读回来：

void Monitor::apply_quorum_to_compatset_features() { CompatSet new_features(features); if (quorum_features & CEPH_FEATURE_OSD_ERASURE_CODES) { new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_OSD_ERASURE_CODES); } if (quorum_features & CEPH_FEATURE_OSDMAP_ENC) { new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_OSDMAP_ENC); } if (quorum_features & CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2) { new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_ERASURE_CODE_PLUGINS_V2); } if (new_features.compare(features) != 0) { CompatSet diff = features.unsupported(new_features); dout(1) << __func__ << " enabling new quorum features: " << diff << dendl; features = new_features; MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction); write_features(t); <<<<<<<<<<<<<<保存 store->apply_transaction(t); apply_compatset_features_to_quorum_requirements(); } }

总结

ceph monitor在通讯初始化时，声明需要最小的功能集，它绕开了Messenger中对required_features的过分依赖，而是在通讯建立后动态地检查是否兼容。