cephFS的使用以及K8S对接cephFS
1 cephFS快速搭建1.创建两个存储池分别用于存储mds的元数据和数据
# ceph osd pool create cephfs_data
pool 'cephfs_data' created
# ceph osd pool create cephfs_metadata
pool 'cephfs_metadata' created
2.创建一个文件系统，名称为"zhiyong18-cephfs"
# ceph fs new zhiyong18-cephfs cephfs_metadata cephfs_data
Pool 'cephfs_data' (id '10') has pg autoscale mode 'on' but is not marked as bulk.
Consider setting the flag by running
# ceph osd pool set cephfs_data bulk true
new fs with metadata pool 11 and data pool 10
3.查看创建的存储池
# ceph fs ls
name: zhiyong18-cephfs, metadata pool: cephfs_metadata, data pools:
# ceph mds stat
zhiyong18-cephfs:0
# ceph -s
cluster:
id: 12fad866-9aa0-11ef-8656-6516a17ad6dd
health: HEALTH_ERR
1 filesystem is offline
1 filesystem is online with fewer MDS than max_mds
1 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 73m)
mgr: ceph141.yvswvf(active, since 73m), standbys: ceph142.gtcikx
mds: 0/0 daemons up
osd: 7 osds: 7 up (since 73m), 7 in (since 38h)
data:
volumes: 1/1 healthy
pools: 6 pools, 129 pgs
objects: 50 objects, 18 MiB
usage: 335 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 129 active+clean
发现：存储池的状态无法正常使用，而且集群是有错误的（HEALTH_ERR），因此我们须要先办理这个题目
在web页面中也有提示：All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline.
https://i-blog.csdnimg.cn/direct/2114723232c0490f85a201de1d7fb207.png#pic_center
以是先开启mds服务，元数据服务
4.应用mds的文件系统
# ceph orch apply mds zhiyong18-cephfs
Scheduled mds.zhiyong18-cephfs update...
5.添加第1个mds服务器，ceph143节点
# ceph orch daemon add mds zhiyong18-cephfs ceph143
Deployed mds.zhiyong18-cephfs.ceph143.eltfff on host 'ceph143'
# ceph mds stat
zhiyong18-cephfs:1 {0=zhiyong18-cephfs.ceph143.eltfff=up:active} 1 up:standby
添加第2个mds，ceph141节点
# ceph orch daemon add mds zhiyong18-cephfs ceph141
Deployed mds.zhiyong18-cephfs.ceph141.geohlx on host 'ceph141'
6.目前活跃提供服务是ceph143，备用的是ceph141
# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANKSTATE MDS ACTIVITY DNS INOS DIRS CAPS
0 activezhiyong18-cephfs.ceph143.eltfffReqs: 0 /s 10 13 12 0
POOL TYPE USEDAVAIL
cephfs_metadatametadata96.0k1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph141.geohlx
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
7.再次查看集群状态ERR状态消散。此处ERR状态是因为，之前的rbd题目（可以忽略）
# ceph -s
cluster:
id: 12fad866-9aa0-11ef-8656-6516a17ad6dd
health: HEALTH_WARN
1 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 84m)
mgr: ceph141.yvswvf(active, since 84m), standbys: ceph142.gtcikx
mds: 1/1 daemons up, 1 standby
osd: 7 osds: 7 up (since 84m), 7 in (since 39h)
data:
volumes: 1/1 healthy
pools: 6 pools, 129 pgs
objects: 72 objects, 18 MiB
usage: 335 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 129 active+clean
8.验证mds的高可用性
直接给ceph143节点关机：init 0，然后查看cephfs的状态照旧正常的
# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANKSTATE MDS ACTIVITY DNS INOS DIRS CAPS
0 activezhiyong18-cephfs.ceph141.geohlxReqs: 0 /s 10 13 12 0
POOL TYPE USEDAVAIL
cephfs_metadatametadata96.0k1625G
cephfs_data data 0 1083G
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
2 cephFS配置多活
1.查看之前的状态
# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANKSTATE MDS ACTIVITY DNS INOS DIRS CAPS
0 activezhiyong18-cephfs.ceph141.geohlxReqs: 0 /s 10 13 12 0
POOL TYPE USEDAVAIL
cephfs_metadatametadata 104k1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph143.eltfff
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
2.查看默认的mds数量为1，同时仅有一个对外提供服务。修改为2，以让2个mds同时对外提供服务
# ceph fs get zhiyong18-cephfs | grepmax_mds
max_mds 1
# ceph fs set zhiyong18-cephfsmax_mds 2
# ceph fs get zhiyong18-cephfs | grepmax_mds
max_mds 2
3.再次查看cephFS 。此时ceph141和ceph143同时对外提供服务，任何一个mds挂掉，都导致集群不可用！
# ceph orch daemon add mds zhiyong18-cephfs ceph142
# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANKSTATE MDS ACTIVITY DNS INOS DIRS CAPS
0 activezhiyong18-cephfs.ceph141.geohlxReqs: 0 /s 10 13 12 0
1 activezhiyong18-cephfs.ceph143.eltfffReqs: 0 /s 10 13 11 0
POOL TYPE USEDAVAIL
cephfs_metadatametadata 176k1083G
cephfs_data data 0 1083G
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
4.添加一个备用的mds，ceph142充当。添加后再次查看cephFS状态
# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANKSTATE MDS ACTIVITY DNS INOS DIRS CAPS
0 activezhiyong18-cephfs.ceph141.geohlxReqs: 0 /s 10 13 12 0
1 activezhiyong18-cephfs.ceph143.eltfffReqs: 0 /s 10 13 11 0
POOL TYPE USEDAVAIL
cephfs_metadatametadata 176k1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph142.igigvp
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
5.验证是否生效。关闭ceph143节点：init 0。然后发现ceph142节点切换过去。
# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANKSTATE MDS ACTIVITY DNS INOS DIRS CAPS
0 activezhiyong18-cephfs.ceph141.geohlxReqs: 0 /s 10 13 12 0
1 activezhiyong18-cephfs.ceph142.igigvpReqs: 0 /s 10 13 11 0
POOL TYPE USEDAVAIL
cephfs_metadatametadata 183k1625G
cephfs_data data 0 1083G
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
# ceph -s
cluster:
id: 12fad866-9aa0-11ef-8656-6516a17ad6dd
health: HEALTH_WARN
failed to probe daemons or devices
insufficient standby MDS daemons available
1/3 mons down, quorum ceph141,ceph142
2 osds down
1 host (2 osds) down
Degraded data redundancy: 91/273 objects degraded (33.333%), 50 pgs degraded
1 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph141,ceph142 (age 36s), out of quorum: ceph143
mgr: ceph141.yvswvf(active, since 7h), standbys: ceph142.gtcikx
mds: 2/2 daemons up
osd: 7 osds: 5 up (since 35s), 7 in (since 80m)
data:
volumes: 1/1 healthy
pools: 6 pools, 129 pgs
objects: 91 objects, 18 MiB
usage: 367 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 91/273 objects degraded (33.333%)
79 active+undersized
50 active+undersized+degraded
6.再把ceph143添加到集群
# ceph orch daemon add mds zhiyong18-cephfs ceph143
最后发现添加不成功，大概是mds不支持3个，最后把max_mds改为了1
3 客户端使用cephFS
01 服务端准备工作
1.确认cephFS的可用
# ceph fs ls
name: zhiyong18-cephfs, metadata pool: cephfs_metadata, data pools:
# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANKSTATE MDS ACTIVITY DNS INOS DIRS CAPS
0 activezhiyong18-cephfs.ceph141.geohlxReqs: 0 /s 10 13 12 0
POOL TYPE USEDAVAIL
cephfs_metadatametadata 249k1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph142.oszjfe
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
# ceph fs get zhiyong18-cephfs | grep max_mds
max_mds 1
2.管理节点创建用户并导出钥匙环和key文件，发到其他节点
# ceph auth add client.wenzhiyongfs mon 'allow r' mds 'allow rw' osd 'allow rwx'
added key for client.wenzhiyongfs
# ceph auth get client.wenzhiyongfs
key = AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rwx"
# ceph auth get client.wenzhiyongfs > ceph.client.wenzhiyongfs.keyring
# ceph authprint-key client.wenzhiyongfs > wenzhiyongfs.key
# more wenzhiyongfs.key
AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
# scp ceph.client.wenzhiyongfs.keyring wenzhiyongfs.key ceph143:/etc/ceph/
# scp ceph.client.wenzhiyongfs.keyringceph142:/etc/ceph/
3.客户端确认本地的认证文件
# ll | grep wen
-rw-r--r-- 1 root root 139 Nov6 22:53 ceph.client.wenzhiyongfs.keyring
-rw-r--r-- 1 root root40 Nov6 22:53 wenzhiyongfs.key
02 客户端使用cephFS
1.客户端ceph141基于key文件举行挂载并尝试写入数据
# df -h | grep mnt
# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt \
-o name=wenzhiyongfs,secretfile=/etc/ceph/wenzhiyongfs.key
# df -h | grep mnt
10.0.0.141:6789,10.0.0.142:6789,127.0.1.1:6789,10.0.0.143:6789:/1.1T 01.1T 0% /mnt
# cp /etc/os-release /etc/fstab /etc/hosts /mnt/
# ls /mnt/
fstabhostsos-release
2.启动另一个客户端ceph142测试，测试时发现数据和ceph143同步了：都能看到全部节点上传的4个文件
# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt \
-o name=wenzhiyongfs,secret=AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
# df -h | grep mnt
10.0.0.141:6789,127.0.1.1:6789,10.0.0.142:6789,10.0.0.143:6789:/1.1T 01.1T 0% /mnt
# cp /etc/hostname /mnt/
# ls /mnt/
fstabhostnamehostsos-release
ceph143节点查看：
# ls /mnt/
fstabhostnamehostsos-release
3.把认证文件删除后ceph.client.wenzhiyongfs.keyring，固然不影响挂载，但是会有提示。
# ls
ceph.client.admin.keyringceph.client.wenzhiyongfs.keyringceph.confrbdmap
# rm -rf ceph.client.wenzhiyongfs.keyring
# umount /mnt
# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt -o name=wenzhiyongfs,secret=AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
2024-11-06T23:35:58.145+0800 7f21d1629100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.wenzhiyongfs.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
# df -h | grep mnt
10.0.0.141:6789,127.0.1.1:6789,10.0.0.142:6789,10.0.0.143:6789:/1.1T 01.1T 0% /mnt
4.ceph142节点仍旧可以正常写入。用起来就和NFS差不多
# cp /etc/netplan/00-installer-config.yaml /mnt/
4 K8S对接cephFS
4.1 cephFS作为volumes
1.Ubuntu默认安装的最新版本是17.2.7，也可以使用拷贝ceph的APT源，实现安装最新版。
全部K8S节点安装ceph-common：
apt -y install ceph-common
ceph节点把认证文件传输到全部K8S节点：
scp admin.secret 10.0.0.231:/etc/ceph/
scp admin.secret 10.0.0.232:/etc/ceph/
scp admin.secret 10.0.0.233:/etc/ceph/
1.编写资源清单
# cat 01-k8s-use-cephfs.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-volume-cephfs-admin-secretfile
spec:
replicas: 3
selector:
matchLabels:
apps: ceph-fs
template:
metadata:
labels:
apps: ceph-fs
spec:
volumes:
- name: data
# 指定存储卷的类型是cephFS
cephfs:
monitors:
- 10.0.0.141:6789
- 10.0.0.142:6789
- 10.0.0.143:6789
# 指定引用的cephFS的路径，若不指定默认为"/"
path: /
# 对于Pod而言，无需修改文件，因此设置为true
readOnly: true
# 指定连接ceph集群的用户，若不指定，默认为admin
user: admin
# 指定admin用户对应的认证文件所在路径
secretFile: "/etc/ceph/admin.secret"
containers:
- name: c1
image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
volumeMounts:
- name: data
mountPath: /wenzhiyong-data
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: svc-cephfs-secretfile
spec:
type: NodePort
selector:
apps: ceph-fs
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 30011
2.进入容器测试，可以看到之前ceph节点写入的数据；因为设置了只读，以是不能写入数据。pod如果要有写入权限，就把volumes中改为：readOnly: false
kubectl exec deploy-volume-cephfs-admin-secretfile-57f6cf7df5-wmfg5 -- sh -c 'ls /wenzhiyong-data'
00-installer-config.yaml
fstab
hostname
hosts
os-release
# 进入pod测试
/ # cd /wenzhiyong-data/
/wenzhiyong-data # ls
00-installer-config.yamlfstab hostname hosts os-release
/wenzhiyong-data # echo 11 >> pods.txt
sh: can't create pods.txt: Read-only file system
4.2 cephFS作为PVC
apiVersion: v1
kind: PersistentVolume
metadata:
name: zhiyong18-cephfs-pv
labels:
school: zhiyong18
spec:
accessModes:
- ReadWriteMany
cephfs:
monitors:
- 10.0.0.141:6789
- 10.0.0.141:6789
- 10.0.0.141:6789
path: /
readOnly: false
user: admin
secretFile: "/etc/ceph/admin.secret"
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: zhiyong18-cephfs-pvc
spec:
# 选择上一步创建的PV
volumeName: zhiyong18-cephfs-pv
accessModes:
- ReadWriteMany
resources:
limits:
storage: 2Gi
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-pvc-cephfs-secretfile
spec:
replicas: 3
selector:
matchLabels:
apps: cephfs
template:
metadata:
labels:
apps: cephfs
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: zhiyong18-cephfs-pvc
containers:
- name: c1
image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
volumeMounts:
- name: data
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: svc-cephfs-secretfile
spec:
type: NodePort
selector:
apps: cephfs
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 30012
[*]PV 的 accessModes 表现该卷支持的访问模式范围。ReadWriteMany` 表现该 PV 可以支持多个节点同时读写该卷
[*]PVC 的 accessModes 表现 PVC 向PV请求的访问模式。如果 PVC 请求的是 ReadWriteMany，那么它会尝试绑定一个支持该模式的 PV
2.运用后查看pod目次下的文件，另有之前ceph集群测试时留下来的文件。在nginx pod内创建目次和写入文件。
# kubectl exec -it deploy-pvc-cephfs-secretfile-6cfcdbfcf-r6sbt -- sh
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # ls
00-installer-config.yaml fstab hostname hosts os-release
/usr/share/nginx/html # mkdir nginx
/usr/share/nginx/html # echo wzywzy.fun > nginx/index.html
3.回到宿主机测试，访问任意一个pod都能看到返回准确的内容，说明：3个pod在共享这个cephFS
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deploy-pvc-cephfs-secretfile-6cfcdbfcf-r6sbt 1/1 Running 0 7m57s 10.100.2.37 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-v55zb 1/1 Running 0 7m57s 10.100.2.38 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-vzzbf 1/1 Running 0 7m57s 10.100.1.19 worker233 <none> <none>
# curl 10.100.2.37/nginx/index.html
wzywzy.fun
# curl 10.100.2.38/nginx/index.html
wzywzy.fun
# curl 10.100.1.19/nginx/index.html
wzywzy.fun
ning 0 7m57s 10.100.2.37 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-v55zb 1/1 Running 0 7m57s 10.100.2.38 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-vzzbf 1/1 Running 0 7m57s 10.100.1.19 worker233 <none> <none>
# curl 10.100.2.37/nginx/index.html
wzywzy.fun
# curl 10.100.2.38/nginx/index.html
wzywzy.fun
# curl 10.100.1.19/nginx/index.html
wzywzy.fun
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
页:
[1]