背景
向量数据库集群只有一个,如果集群出问题将影响整个业务,所以需要对向量数据库中的数据做定时备份
现有两个milvus集群,方案如下
通过对milvus集群的了解发现其数据的存储是集群中的minio组件,所以做的整个数据备份及恢复是基于minio来做的。
首先是在需要做备份的集群中对minio数据做备份(milvus-backup),然后通过rclone将备份数据传输到备份集群的minio集群中,然后使用py脚本来对备份集群进行数据清理,然后恢复最新的备份数据到集群中,最后加载到内存中并进行数据的校验和对比最后告警出来
milvus-backup备份
工具地点milvus-backup下载地点
需要配置好config文件
这里配置两个,一个是备份生产数据的,一个是在备份集群中恢复数据的
此处只做一个示例
- # Configures the system log output.
- log:
- level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
- console: true # whether print log to console
- file:
- rootPath: "logs/backup.log"
- http:
- simpleResponse: true
- # milvus proxy address, compatible to milvus.yaml
- milvus:
- address:
- port: 19530
- authorizationEnabled: false
- # tls mode values [0, 1, 2]
- # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
- tlsMode: 0
- user: "root"
- password: "Milvus"
- # Related configuration of minio, which is responsible for data persistence for Milvus.
- minio:
- # cloudProvider: "minio" # deprecated use storageType instead
- storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
- address: # Address of MinIO/S3
- port: 9000 # Port of MinIO/S3
- accessKeyID: minioadmin # accessKeyID of MinIO/S3
- secretAccessKey: minioadmin # MinIO/S3 encryption string
- useSSL: false # Access to MinIO/S3 with SSL
- useIAM: false
- iamEndpoint: ""
- bucketName: "milvus-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
- rootPath: "file" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance
- # only for azure
- backupAccessKeyID: minioadmin # accessKeyID of MinIO/S3
- backupSecretAccessKey: minioadmin # MinIO/S3 encryption string
- backupBucketName: "backup" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
- backupRootPath: "file" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath
复制代码 主要更改地点及端口
创建备份文件
- ./milvus-backup create -n backup_${datetime} --config backup-prod.yaml
复制代码 会在prod的minio集群中生成一个备份数据
备份数据传输
生产备份好的数据是在生产milvus集群中的minio中,所以需要工具进行对象存储的传输,此处使用rclone
此处rclone配置文件
- [prod-minio]
- type = s3
- provider = Minio
- env_auth = false
- access_key_id = minioadmin
- secret_access_key = minioadmin
- endpoint =
- [backup-minio]
- type = s3
- provider = Minio
- env_auth = false
- access_key_id = minioadmin
- secret_access_key = minioadmin
- endpoint =
复制代码 此处主要修改endpoint,为minio的地点
备份数据传输
- rclone copy -P prod-minio:backup/file/backup_${datetime} backup-minio:backup/file/backup_${datetime}
复制代码 清理备份集群中现有数据
- #!/usr/bin/env python3.8
- from pymilvus import (
- connections,
- utility,
- )
- # backup-server proxy地址
- hosts = "XXXXXX"
- # 连接milvus服务器
- connections.connect("default", host=hosts, port="19530")
- # 获取collection列表
- conn_lst = utility.list_collections()
- for i in conn_lst:
- utility.drop_collection(i)
复制代码 备份数据恢复到备份集群并加载至内存
恢复数据使用backup工具实现
- ./milvus-backup restore --restore_index -n backup_${datetime} --config restore-prod.yaml
复制代码 加载collections使用py脚本实现
- #!/usr/bin/env python3.8
- from pymilvus import (
- connections,
- utility,
- Collection,
- )
- hosts="XXXXXX"
- # 连接milvus服务器
- connections.connect("default", host=hosts, port="19530")
- # 获取collection列表
- conn_lst = utility.list_collections()
- # 遍历collection列表
- for i in conn_lst:
- collection = Collection(i)
- collection.load()
复制代码 校验数据并告警
此处同样用py脚本去实现校验
最后便是实现完全的自动化,在服务器上创建定时使命并编写自动化shell脚本,使其在每晚两点开始执行备份
- #!/bin/bash# 创建完整备份文件echo -e "\033[32m----------start backup prod milvus data to minio----------\033[0m"datetime=$(date "+%Y%m%d")cd /opt/milvus-backups./milvus-backup create -n backup_${datetime} --config backup-prod.yaml
- echo -e "\033[32m----------backup prod milvus data to minio finish----------\033[0m"# 拷贝备份文件到备份机器echo -e "\033[32m----------copy backup files to backup minio server----------\033[0m"rclone copy -P prod-minio:backup/file/backup_${datetime} backup-minio:backup/file/backup_${datetime}
- echo -e "\033[32m----------copy backup files finish----------\033[0m"# 清理备份集群现有数据echo -e "\033[32m----------clean backup-server data----------\033[0m"python3 /opt/milvus-backups/delete.pyecho -e "\033[32m----------clean data finish------------\033[0m"# 从备份数据恢复至备份集群echo -e "\033[32m----------start restore backup files----------\033[0m"./milvus-backup restore --restore_index -n backup_${datetime} --config restore-prod.yaml
- echo -e "\033[32m----------restore backup files finish----------\033[0m"# 加载collectionsecho -e "\033[32m----------load collection------------\033[0m"python3 /opt/milvus-backups/load.pyecho -e "\033[32m----------load finish------------\033[0m"# 对比数据并上传qwpython3 /opt/milvus-backups/alert.py
复制代码 实现效果
免责声明:如果侵犯了您的权益,请联系站长,我们会及时删除侵权内容,谢谢合作!更多信息从访问主页:qidao123.com:ToB企服之家,中国第一个企服评测及商务社交产业平台。 |