文章目录
前言链接：[基于kubesphere的k8s环境部署单点版本的rook-ceph](/baidu_35848778/article/details/131050918) 一、rook-ceph是什么？二、开始部署2.1 环境准备2.2 软件包准备，计算\存储节点执行2.3 下载rook-ceph文件2.4 部署operator2.5 创建ceph集群2.6 创建工具容器，检查集群状态2.7 准备dashboard的nodeport端口映射服务2.8 准备prometheus的metric端口映射服务 三、创建存储类3.1 创建cephrbd存储类3.2 创建cephfs存储类3.3 查看创建结果 补充总结前言
这次实验记录的是使用虚拟机搭建的使用了三个计算\存储节点的rook-ceph集群，模拟真实的使用场景。补充之前发的只有单点的部署方式。
链接：基于kubesphere的k8s环境部署单点版本的rook-ceph
一、rook-ceph是什么？
Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for Ceph storage to natively integrate with cloud-native environments.
Rook是一个开源的云原生存储编排器，为Ceph存储提供平台、框架和支持，以便与云原生环境进行本地集成。
二、开始部署
2.1 环境准备
虚拟机数量：四台
虚拟机镜像类型：CentOS-7-x86_64-Minimal-.iso
k8s环境：一套
k8s环境:1.23.6
机器列表如下
2.2 软件包准备，计算\存储节点执行
安装软件包，加载rbd模块
#软件包装备yum install -y git lvm2 gdisk#内核加载rbd模块modprobe rbdlsmod | grep rbd
备注：删除残留数据，如果部署失败，一定清理下数据，不清理的话会影响下一次的部署
删除配置文件目录rm -rf /var/lib/rook/格式化磁盘gdisk --zap-all /dev/vdagdisk --zap-all /dev/vdbdd if=/dev/zero of=/dev/vda bs=1M count=100 oflag=direct,dsyncdd if=/dev/zero of=/dev/vdb bs=1M count=100 oflag=direct,dsync
2.3 下载rook-ceph文件
下载文件并提取核心文件到自己的部署文件夹
cd /data/yum install -y gitgit clone --single-branch --branch v1.11.6 /rook/rook.git# 提取部署文件mkdir -p /data/rook-ceph/cp /data/rook/deploy/examples/crds.yaml /data/rook-ceph/crds.yamlcp /data/rook/deploy/examples/common.yaml /data/rook-ceph/common.yamlcp /data/rook/deploy/examples/operator.yaml /data/rook-ceph/operator.yamlcp /data/rook/deploy/examples/cluster.yaml /data/rook-ceph/cluster.yamlcp /data/rook/deploy/examples/filesystem.yaml /data/rook-ceph/filesystem.yamlcp /data/rook/deploy/examples/toolbox.yaml /data/rook-ceph/toolbox.yamlcp /data/rook/deploy/examples/csi/rbd/storageclass.yaml /data/rook-ceph/storageclass-rbd.yamlcp /data/rook/deploy/examples/csi/cephfs/storageclass.yaml /data/rook-ceph/storageclass-cephfs.yamlcp /data/rook/deploy/examples/csi/nfs/storageclass.yaml /data/rook-ceph/storageclass-nfs.yamlcd /data/rook-ceph
2.4 部署operator
修改镜像仓库信息，operator.yaml中镜像仓库修改为阿里云的镜像仓库配置
ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.8.0"ROOK_CSI_REGISTRAR_IMAGE: "-/google_containers/csi-node-driver-registrar:v2.7.0"ROOK_CSI_RESIZER_IMAGE: "-/google_containers/csi-resizer:v1.7.0"ROOK_CSI_PROVISIONER_IMAGE: "-/google_containers/csi-provisioner:v3.4.0"ROOK_CSI_SNAPSHOTTER_IMAGE: "-/google_containers/csi-snapshotter:v6.2.1"ROOK_CSI_ATTACHER_IMAGE: "-/google_containers/csi-attacher:v4.1.0"
执行部署
# 开始部署cd /data/rook-cephkubectl create -f crds.yamlkubectl create -f common.yamlkubectl create -f operator.yaml# 检查operator的创建运行状态kubectl -n rook-ceph get pod# 输出NAME READY STATUS RESTARTS AGErook-ceph-operator-585f6875d-qjhdn 1/1Running 04m36s
2.5 创建ceph集群
修改cluster.yaml，配置osd对应的数据盘,这里截取修改的部分，主要是nodes这里的配置。我这里按照集群的现有硬盘，将三个节点的数据盘都直接配置上了。没有使用规则发现或是全部发现。
priorityClassNames:mon: system-node-criticalosd: system-node-criticalmgr: system-cluster-criticaluseAllNodes: falseuseAllDevices: falsedeviceFilter: config:nodes:- name: "kubeworker01"devices: - name: "vda" - name: "vdb" - name: "kubeworker02"devices:- name: "vda" - name: "vdb" - name: "kubeworker03"devices:- name: "vda" - name: "vdb"
执行部署cluster-test.yaml
kubectl create -f cluster.yaml# 会部署一段时间kubectl -n rook-ceph get pod# 查看部署结果，当全部为Running之后部署工具容器进行集群确认NAME READY STATUSRESTARTS AGEcsi-cephfsplugin-7qk262/2Running034mcsi-cephfsplugin-dp8zx2/2Running034mcsi-cephfsplugin-fb6rh2/2Running034mcsi-cephfsplugin-provisioner-5549b4bcff-56ntx 5/5Running034mcsi-cephfsplugin-provisioner-5549b4bcff-m5j76 5/5Running034mcsi-rbdplugin-d829n 2/2Running034mcsi-rbdplugin-provisioner-bcff85bf9-7thl75/5Running034mcsi-rbdplugin-provisioner-bcff85bf9-cctkc5/5Running034mcsi-rbdplugin-rj9wp 2/2Running034mcsi-rbdplugin-zs6s2 2/2Running034mrook-ceph-crashcollector-kubeworker01-794647548b-bdrcx 1/1Running091srook-ceph-crashcollector-kubeworker02-d97cfb685-ss2sl 1/1Running086srook-ceph-crashcollector-kubeworker03-9d65c8dd8-zrv5x 1/1Running022mrook-ceph-mgr-a-6fccb8744f-5zdvf3/3Running023mrook-ceph-mgr-b-7c4bbbfcf4-fhxm93/3Running023mrook-ceph-mon-a-56dc4dfb8d-4j2bz2/2Running034mrook-ceph-mon-b-7d6d96649b-spz4p2/2Running033mrook-ceph-mon-c-759c774dc7-8hftq2/2Running028mrook-ceph-operator-f45db9b9f-knbx4 1/1Running02m9srook-ceph-osd-0-86cd7776c8-bm7642/2Running091srook-ceph-osd-1-7686cf9757-ss9z22/2Running086srook-ceph-osd-2-5bc55847d-g2z6l2/2Running091srook-ceph-osd-3-998bccb64-rq9cf2/2Running083srook-ceph-osd-4-5c7c7f555b-djdvl2/2Running086srook-ceph-osd-5-69976f85fc-9xz942/2Running083srook-ceph-osd-prepare-kubeworker01-qlvcp 0/1Completed 0104srook-ceph-osd-prepare-kubeworker02-mnhcj 0/1Completed 0100srook-ceph-osd-prepare-kubeworker03-sbk76 0/1Completed 097srook-ceph-tools-598b59df89-77sm71/1Running07m43s
2.6 创建工具容器，检查集群状态
# 创建工具容器kubectl apply -f toolbox.yaml# 执行命令查询kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s# 输出：cluster:id:3a04d434-a2ac-4f2a-a231-a08ca46c6df3health: HEALTH_OKservices:mon: 3 daemons, quorum a,b,c (age 20m)mgr: a(active, since 4m), standbys: bosd: 6 osds: 6 up (since 3m), 6 in (since 4m)data:pools: 1 pools, 1 pgsobjects: 2 objects, 449 KiBusage: 521 MiB used, 119 GiB / 120 GiB availpgs:1 active+clean# 或是进入工具容器内执行命令kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash# 查看集群状态bash-4.4$ ceph -s cluster:id:3a04d434-a2ac-4f2a-a231-a08ca46c6df3health: HEALTH_OKservices:mon: 3 daemons, quorum a,b,c (age 21m)mgr: a(active, since 5m), standbys: bosd: 6 osds: 6 up (since 5m), 6 in (since 5m)data:pools: 1 pools, 1 pgsobjects: 2 objects, 449 KiBusage: 521 MiB used, 119 GiB / 120 GiB availpgs:1 active+clean
2.7 准备dashboard的nodeport端口映射服务
cat > /data/rook-ceph/dashboard-external-https.yaml <<EOFapiVersion: v1kind: Servicemetadata:name: rook-ceph-mgr-dashboard-external-httpsnamespace: rook-cephlabels:app: rook-ceph-mgrrook_cluster: rook-cephspec:ports:- name: dashboardport: 8443protocol: TCPtargetPort: 8443nodePort: 30808selector:app: rook-ceph-mgrrook_cluster: rook-cephsessionAffinity: Nonetype: NodePortEOF# 这里的nodeport端口建议更换为适合自己环境规划的端口kubectl apply -f dashboard-external-https.yaml# 输出service/rook-ceph-mgr-dashboard-external-https created# 获取admin用户密码kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
使用浏览器访问端口192.168.150.61:30808，使用admin用户登陆，登陆后可以修改密码，也可以新建用户
成功登陆
2.8 准备prometheus的metric端口映射服务
cat > /data/rook-ceph/metric-external-https.yaml <<EOFapiVersion: v1kind: Servicemetadata:name: rook-ceph-mgr-metric-external-httpsnamespace: rook-cephlabels:app: rook-ceph-mgrrook_cluster: rook-cephspec:ports:- name: metricport: 9283protocol: TCPtargetPort: 9283nodePort: 30809selector:app: rook-ceph-mgrrook_cluster: rook-cephsessionAffinity: Nonetype: NodePortEOF# 这里的nodeport端口建议更换为适合自己环境规划的端口kubectl apply -f metric-external-https.yaml# 输出service/rook-ceph-mgr-metric-external-https created
使用浏览器访问端口192.168.150.61:30809
三、创建存储类
3.1 创建cephrbd存储类
kubectl apply -f storageclass-rbd.yaml
创建测试用的pvc
apiVersion: v1kind: PersistentVolumeClaimmetadata:name: test-rbd-pv-claimspec:storageClassName: rook-ceph-blockaccessModes:- ReadWriteOnceresources:requests:storage: 10G
3.2 创建cephfs存储类
kubectl apply -f filesystem.yamlkubectl apply -f storageclass-cephfs.yaml# 创建filesystem.yaml之后会生成rook-ceph-mds-myfs的工作负载kubectl -n rook-ceph get pod |grep mds# 输出rook-ceph-mds-myfs-a-5d5754b77-nlcb9 2/2Running097srook-ceph-mds-myfs-b-9f9dd7f6-sc6qm 2/2Running096s
apiVersion: v1kind: PersistentVolumeClaimmetadata:name: test-cephfs-pv-claimspec:storageClassName: rook-cephfsaccessModes:- ReadWriteManyresources:requests:storage: 10G
3.3 查看创建结果
查询存储
kubectl get storageclass# 输出NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODEALLOWVOLUMEEXPANSION AGElocal (default) openebs.io/localDeleteWaitForFirstConsumer false 21drook-ceph-block rook-ceph.rbd.DeleteImmediate true 4srook-cephfs rook-ceph.cephfs. DeleteImmediate true 4m44s
查询pvc
kubectl get pvc -o wide# 输出NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASSAGE VOLUMEMODEtest-cephfs-pv-claim Bound pvc-3cdd9e88-2ae2-4e23-9f23-13e095707964 10Gi RWX rook-cephfs 7s Filesystemtest-rbd-pv-claimBound pvc-55a57b74-b595-4726-8b82-5257fd2d279a 10Gi RWO rook-ceph-block 6s Filesystem
补充
当集群没有按照预期成功部署的情况下，可以执行如下的命令重新部署或是查看异常的原因
# 重启ceph operator调度，重新部署kubectl rollout restart deploy rook-ceph-operator -n rook-ceph#注：如果新osd pod无法执行起来可以通过查询osd prepare日志找问题kubectl -n rook-ceph logs rook-ceph-osd-prepare-nodeX-XXXXX provision
总结
正式的集群版本用起来还是很方便的，后续会测试一些加盘和减盘的动作，模拟一下日常可能会遇到的问题。ceph只是初级使用还是很香的。
