Milvus向量数据库备份方案

背景

向量数据库集群只有一个，如果集群出问题将影响整个业务，所以需要对向量数据库中的数据做定时备份
现有两个milvus集群，方案如下

通过对milvus集群的了解发现其数据的存储是集群中的minio组件，所以做的整个数据备份及恢复是基于minio来做的。
首先是在需要做备份的集群中对minio数据做备份（milvus-backup），然后通过rclone将备份数据传输到备份集群的minio集群中，然后使用py脚本来对备份集群进行数据清理，然后恢复最新的备份数据到集群中，最后加载到内存中并进行数据的校验和对比最后告警出来
milvus-backup备份

工具地点milvus-backup下载地点
需要配置好config文件
这里配置两个，一个是备份生产数据的，一个是在备份集群中恢复数据的
此处只做一个示例

1. # Configures the system log output.
2. log:
3.   level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
4.   console: true # whether print log to console
5.   file:
6.     rootPath: "logs/backup.log"
8. http:
9.   simpleResponse: true
11. # milvus proxy address, compatible to milvus.yaml
12. milvus:
13.   address:
14.   port: 19530
15.   authorizationEnabled: false
16.   # tls mode values [0, 1, 2]
17.   # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
18.   tlsMode: 0
19.   user: "root"
20.   password: "Milvus"
22. # Related configuration of minio, which is responsible for data persistence for Milvus.
23. minio:
24.   # cloudProvider: "minio" # deprecated use storageType instead
25.   storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
27.   address:  # Address of MinIO/S3
28.   port: 9000   # Port of MinIO/S3
29.   accessKeyID: minioadmin  # accessKeyID of MinIO/S3
30.   secretAccessKey: minioadmin # MinIO/S3 encryption string
31.   useSSL: false # Access to MinIO/S3 with SSL
32.   useIAM: false
33.   iamEndpoint: ""
35.   bucketName: "milvus-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
36.   rootPath: "file" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance
38.   # only for azure
39.   backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
40.   backupSecretAccessKey: minioadmin # MinIO/S3 encryption string
41.   backupBucketName: "backup" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
42.   backupRootPath: "file" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

复制代码

主要更改地点及端口
创建备份文件

1. ./milvus-backup create -n backup_${datetime} --config backup-prod.yaml

复制代码

会在prod的minio集群中生成一个备份数据
备份数据传输

生产备份好的数据是在生产milvus集群中的minio中，所以需要工具进行对象存储的传输，此处使用rclone
此处rclone配置文件

1. [prod-minio]
2. type = s3
3. provider = Minio
4. env_auth = false
5. access_key_id = minioadmin
6. secret_access_key = minioadmin
7. endpoint =
8. [backup-minio]
9. type = s3
10. provider = Minio
11. env_auth = false
12. access_key_id = minioadmin
13. secret_access_key = minioadmin
14. endpoint =

复制代码

此处主要修改endpoint，为minio的地点
备份数据传输

1. rclone copy -P prod-minio:backup/file/backup_${datetime} backup-minio:backup/file/backup_${datetime}

复制代码

清理备份集群中现有数据

1. #!/usr/bin/env python3.8
2. from pymilvus import (
3. connections,
4. utility,
5. )
6. # backup-server proxy地址
7. hosts = "XXXXXX"
8. # 连接milvus服务器
9. connections.connect("default", host=hosts, port="19530")
10. # 获取collection列表
11. conn_lst = utility.list_collections()
12. for i in conn_lst:
13.     utility.drop_collection(i)

复制代码

备份数据恢复到备份集群并加载至内存

恢复数据使用backup工具实现

1. ./milvus-backup restore --restore_index -n backup_${datetime} --config restore-prod.yaml

复制代码

加载collections使用py脚本实现

1. #!/usr/bin/env python3.8
2. from pymilvus import (
3. connections,
4. utility,
5. Collection,
6. )
7. hosts="XXXXXX"
8. # 连接milvus服务器
9. connections.connect("default", host=hosts, port="19530")
10. # 获取collection列表
11. conn_lst = utility.list_collections()
12. # 遍历collection列表
13. for i in conn_lst:
14.     collection = Collection(i)
15.     collection.load()

复制代码

校验数据并告警

此处同样用py脚本去实现校验
最后便是实现完全的自动化，在服务器上创建定时使命并编写自动化shell脚本，使其在每晚两点开始执行备份

1. #!/bin/bash# 创建完整备份文件echo -e "\033[32m----------start backup prod milvus data to minio----------\033[0m"datetime=$(date "+%Y%m%d")cd /opt/milvus-backups./milvus-backup create -n backup_${datetime} --config backup-prod.yaml
2. echo -e "\033[32m----------backup prod milvus data to minio finish----------\033[0m"# 拷贝备份文件到备份机器echo -e "\033[32m----------copy backup files to backup minio server----------\033[0m"rclone copy -P prod-minio:backup/file/backup_${datetime} backup-minio:backup/file/backup_${datetime}
3. echo -e "\033[32m----------copy backup files finish----------\033[0m"# 清理备份集群现有数据echo -e "\033[32m----------clean backup-server data----------\033[0m"python3 /opt/milvus-backups/delete.pyecho -e "\033[32m----------clean  data finish------------\033[0m"# 从备份数据恢复至备份集群echo -e "\033[32m----------start restore backup files----------\033[0m"./milvus-backup restore --restore_index -n backup_${datetime} --config restore-prod.yaml
4. echo -e "\033[32m----------restore backup files finish----------\033[0m"# 加载collectionsecho -e "\033[32m----------load collection------------\033[0m"python3 /opt/milvus-backups/load.pyecho -e "\033[32m----------load finish------------\033[0m"# 对比数据并上传qwpython3 /opt/milvus-backups/alert.py

复制代码

实现效果

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。