Redis数据迁移-RedisShake

redis-shake是阿里云Redis团队开源的用于Redis数据迁移和数据过滤的工具。

  一、根本功能

      

redis-shake它支持解析、规复、备份、同步四个功能

   

规复restore：将RDB文件规复到目的redis数据库。

   

备份dump：将源redis的全量数据通过RDB文件备份起来。

   

解析decode：对RDB文件进行读取，并以json格式解析存储。

   

同步sync：支持源redis和目的redis的数据同步，支持全量和增量数据的迁移，支持单节点、主从版、集群版之间的互相同步。

   

同步rump：支持源redis和目的redis的数据同步，仅支持全量的迁移,采用scan和restore下令进行迁移，支持不同云厂商不同redis版本的迁移。

   

   二、根本原理

   

   
   三、RedisShake同步原理

   
         

1.源Redis服务实例相当于主库，Redis-shake相当于从库，它会发送psync指令给源Redis服务实例。

   

2.源Redis实例先把RDB文件传输给 Redis-shake ，Redis-shake 会把RDB文件发送给目的实例。

   

3.源实例会再把增量下令发送给 Redis-shake ，Redis-shake负责把这些增量下令再同步给目的实例。

   

    四、RedisShake安装

    确保您在当地呆板上设置了 Golang 环境。
    4.1、release包下载

    Releases · tair-opensource/RedisShake · GitHub
    4.2、解压

   

1. tar -zxvf redis-shake-linux-amd64.tar.gz -C /home/redisshake/

复制代码

                  

解压完了之后有两个文件：

            

    4.3、修改shake.toml配置文件

   

1. function = ""
4. [sync_reader]
5. cluster = false            # set to true if source is a redis cluster
6. address = "127.0.0.1:6379" # when cluster is true, set address to one of the cluster node
7. username = ""              # keep empty if not using ACL
8. password = ""              # keep empty if no authentication is required
9. tls = false
10. sync_rdb = true # set to false if you don't want to sync rdb
11. sync_aof = true # set to false if you don't want to sync aof
12. prefer_replica = true # set to true if you want to sync from replica node
14. #[scan_reader]
15. #cluster = false            # set to true if source is a redis cluster
16. #address = "127.0.0.1:6379" # when cluster is true, set address to one of the cluster node
17. #username = ""              # keep empty if not using ACL
18. #password = ""              # keep empty if no authentication is required
19. #tls = false
20. #dbs = []                   # set you want to scan dbs such as [1,5,7], if you don't want to scan all
21. #scan = true                # set to false if you don't want to scan keys
22. #ksn = false                # set to true to enabled Redis keyspace notifications (KSN) subscription
23. #count = 1                  # number of keys to scan per iteration
25. # [rdb_reader]
26. # filepath = "/tmp/dump.rdb"
28. # [aof_reader]
29. # filepath = "/tmp/.aof"
30. # timestamp = 0              # subsecond
32. [redis_writer]
33. cluster = false            # set to true if target is a redis cluster
34. sentinel = false           # set to true if target is a redis sentinel
35. master = ""                # set to master name if target is a redis sentinel
36. address = "192.168.72.129:6379" # when cluster is true, set address to one of the cluster node
37. username = ""              # keep empty if not using ACL
38. password = ""              # keep empty if no authentication is required
39. tls = false
40. off_reply = false       # ture off the server reply
42. [advanced]
43. dir = "data"
44. ncpu = 0        # runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
45. pprof_port = 0  # pprof port, 0 means disable
46. status_port = 0 # status port, 0 means disable
48. # log
49. log_file = "shake.log"
50. log_level = "info"     # debug, info or warn
51. log_interval = 5       # in seconds
53. # redis-shake gets key and value from rdb file, and uses RESTORE command to
54. # create the key in target redis. Redis RESTORE will return a "Target key name
55. # is busy" error when key already exists. You can use this configuration item
56. # to change the default behavior of restore:
57. # panic:   redis-shake will stop when meet "Target key name is busy" error.
58. # rewrite: redis-shake will replace the key with new value.
59. # ignore:  redis-shake will skip restore the key when meet "Target key name is busy" error.
60. rdb_restore_command_behavior = "panic" # panic, rewrite or skip
62. # redis-shake uses pipeline to improve sending performance.
63. # This item limits the maximum number of commands in a pipeline.
64. pipeline_count_limit = 1024
66. # Client query buffers accumulate new commands. They are limited to a fixed
67. # amount by default. This amount is normally 1gb.
68. target_redis_client_max_querybuf_len = 1024_000_000
70. # In the Redis protocol, bulk requests, that are, elements representing single
71. # strings, are normally limited to 512 mb.
72. target_redis_proto_max_bulk_len = 512_000_000
74. # If the source is Elasticache or MemoryDB, you can set this item.
75. aws_psync = "" # example: aws_psync = "10.0.0.1:6379@nmfu2sl5osync,10.0.0.1:6379@xhma21xfkssync"
77. # destination will delete itself entire database before fetching files
78. # from source during full synchronization.
79. # This option is similar redis replicas RDB diskless load option:
80. #   repl-diskless-load on-empty-db
81. empty_db_before_sync = false
83. [module]
84. # The data format for BF.LOADCHUNK is not compatible in different versions. v2.6.3 <=> 20603
85. target_mbbloom_version = 20603

复制代码

        官方使用指导手册：     
     
Sync Reader | RedisShake            4.4、启动RediShake

   

1. ./redis-shake shake.toml

复制代码

   

    4.5、测试数据迁移
                  

在192.168.72.128这台呆板上插入几个key

      

                           

可以看到RedisShake工具在实施监听key，有新增的就会把新增的key迁移到另外一呆板的redis中；看打印的日记是写了4条数据

        

                                   

在192.168.72.129这台呆板上查看迁移的数据

         

         

          4.5、数据校验

                                          

通过info keyspace下令，查看所有的key和过期key的数量。

            

                              

         

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。