repmgr搭建一主+一从+一witness的PostgreSQL高可用集群

祗疼妳一个  金牌会员 | 2024-11-7 19:31:24 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 846|帖子 846|积分 2540

测试环境说明


环境如下：
OS：　　　　　　ubuntu 20
PostgreSQL：　　PostgreSQL 16.4
repmgr:　　　　　repmgr 5.3.3
三台测试环境，均已安装好PostgreSQL 16.4，数据库实例信息如下
IP          　　postgresql实例名   身份
192.168.152.100：　　postgresql9000：  主节点
192.168.152.101：　　postgresql9000：  从节点
192.168.152.102：　　postgresql9000：  witness节点
 

主节点上的操作步骤

###1 repmgr源码编译安装
  1. drwxr-xr-x  4 root root     4096 Nov  5 12:45 ./
  2. drwxr-xr-x 14 root root     4096 Sep 29 16:10 ../
  3. -rw-r--r--  1 root root     2784 Nov  6 11:28 pg_install.sh
  4. drwxrwxrwx  6 1107 1107     4096 Nov  6 11:31 postgresql-16.4/
  5. -rw-r--r--  1 root root 32660355 Aug  6 04:19 postgresql-16.4.tar.gz
  6. drwxrwxr-x  6 root root     4096 Nov  7 09:20 repmgr-5.3.3/
  7. -rw-r--r--  1 root root   456931 Oct 21 17:00 repmgr-5.3.3.tar.gz
  8. root@ubuntu01:/usr/local/pg16#
  9. root@ubuntu01:/usr/local/pg16# tar -xzvf repmgr-5.3.3.tar.gz
  10. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3# cd repmgr-5.3.3/
  11. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3# whereis pg_config
  12. pg_config: /usr/local/pgsql16/server/bin/pg_config
  13. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3#
  14. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3#
  15. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3#  ./configure --prefix=/usr/local/pgsql16/server
  16. checking for a sed that does not truncate output... /usr/bin/sed
  17. checking for pg_config... /usr/local/pgsql16/server/bin/pg_config
  18. configure: building against PostgreSQL 16.4
  19. checking for gnused... no
  20. checking for gsed... no
  21. checking for sed... yes
  22. configure: creating ./config.status
  23. config.status: creating Makefile
  24. config.status: creating Makefile.global
  25. config.status: creating config.h
  26. config.status: config.h is unchanged
  27. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3#
  28. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3# make && make install
  29. Building against PostgreSQL 16
  30. make: Nothing to be done for 'all'.
  31. Building against PostgreSQL 16
  32. /usr/bin/mkdir -p '/usr/local/pgsql16/server/lib'
  33. /usr/bin/mkdir -p '/usr/local/pgsql16/server/share/extension'
  34. /usr/bin/mkdir -p '/usr/local/pgsql16/server/share/extension'
  35. /usr/bin/mkdir -p '/usr/local/pgsql16/server/bin'
  36. /usr/bin/install -c -m 755  repmgr.so '/usr/local/pgsql16/server/lib/repmgr.so'
  37. /usr/bin/install -c -m 644 .//repmgr.control '/usr/local/pgsql16/server/share/extension/'
  38. /usr/bin/install -c -m 644 .//repmgr--unpackaged--4.0.sql .//repmgr--unpackaged--5.1.sql .//repmgr--unpackaged--5.2.sql .//repmgr--unpackaged--5.3.sql .//repmgr--4.0.sql .//repmgr--4.0--4.1.sql .//repmgr--4.1.sql .//repmgr--4.1--4.2.sql .//repmgr--4.2.sql .//repmgr--4.2--4.3.sql .//repmgr--4.3.sql .//repmgr--4.3--4.4.sql .//repmgr--4.4.sql .//repmgr--4.4--5.0.sql .//repmgr--5.0.sql .//repmgr--5.0--5.1.sql .//repmgr--5.1.sql .//repmgr--5.1--5.2.sql .//repmgr--5.2.sql .//repmgr--5.2--5.3.sql .//repmgr--5.3.sql  '/usr/local/pgsql16/server/share/extension/'
  39. /usr/bin/install -c -m 755 repmgr repmgrd '/usr/local/pgsql16/server/bin/'
  40. root@ubuntu01:/usr/local/pg16/repmgr-5.3.3#
复制代码
注意：
1，编译后会在/usr/local/pgsql16/server/bin路径下生成repmgrd和repmgr两个文件
2，尝试偷懒，将当地编译的repmgrd和repmgr两个文件复制到其他机器上，绕过编译步骤，但是在启动repmgr的时间会不绝提示找不到可执行文件，所以照旧老老实实逐台机器编译。
3，提示找不到libpq.so.5，尝试建一个软链接后办理
  1. root@ubuntu01:~# repmgr -V
  2. repmgr: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory
  3. root@ubuntu01:~# psql --version
  4. psql (PostgreSQL) 16.4
  5. root@ubuntu01:~# sudo find / -name libpq.so.5
  6. /usr/local/pgsql16/server/lib/libpq.so.5
  7. /usr/local/pg16/postgresql-16.4/src/interfaces/libpq/libpq.so.5
  8. root@ubuntu01:~#
  9. root@ubuntu01:~# ln -s /usr/local/pgsql16/server/lib/libpq.so.5 /usr/lib/x86_64-linux-gnu/libpq.so.5
  10. root@ubuntu01:~# repmgr -V
  11. repmgr 5.3.3
  12. root@ubuntu01:~#
复制代码
 
###2 主节点配置文件修改
  1. wal_level = 'replica'或者'logic'
  2. hot_standby = on
  3. archive_mode = on
  4. archive_command = 'test ! -f /postgres/product/archivedir/%f && cp %p /postgres/product/archivedir/%f'
  5. shared_preload_libraries = 'repmgr'
复制代码
 
###3 修改属主

由于是root用户编译的repmgr，所以重新授权
  1. chown -R postgres:postgres /usr/local/pgsql16/
  2. chmod 700 -R /usr/local/pgsql16/
复制代码
 
###4 重启数据库
  1. systemdctl restart postgresql9000
复制代码
 
###5 创建repmgr数据库
  1. --创建repmgr用户
  2. create user repmgr replication password 'password123456';
  3. --alter  user repmgr replication password 'password123456';
  4. alter user repmgr superuser;
  5. create database repmgr owner repmgr;
  6. ALTER USER repmgr SET search_path TO repmgr, "$user", public;
  7. alter user repmgr superuser ;
  8. --进入repmgr数据库
  9. create schema repmgr ;
  10. ALTER USER repmgr SET search_path TO repmgr, "$user", public;
  11. --创建repmgr扩展
  12. create extension repmgr;
  13. --检查扩展
  14. SELECT * FROM pg_extension;
复制代码
 
###6 pg_hba.conf新增配置
  1. #TYPE DATABASE USER ADDRESS METHOD [OPTIONS]
  2. host repmgr repmgr 192.168.152.0/24         md5
  3. host repmgr repmgr 192.168.152.0/24         md5
  4. host replication repmgr 192.168.152.0/24    md5
  5. host replication repmgr 192.168.152.0/24    md5
复制代码
 
###7 编辑repmgr配置文件
  1. ###/usr/local/pgsql16/repmgr/repmgr.conf
  2. node_id=1
  3. node_name='ubuntu01'
  4. conninfo='host=192.168.150.100 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100'
  5. data_directory='/usr/local/pgsql16/pg9000/data'
  6. pg_bindir='/usr/local/pgsql16/server/bin'
  7. failover=automatic
  8. promote_command='/usr/local/pgsql16/server/bin/repmgr standby promote -f /usr/local/pgsql16/repmgr/repmgr.conf --log-to-file'
  9. follow_command='/usr/local/pgsql16/server/bin/repmgr standby follow -f /usr/local/pgsql16/repmgr/repmgr.conf --log-to-file --upstream-node-id=%n'
  10. log_file='/usr/local/pgsql16/repmgr/repmgr.log'
复制代码
 
###8 编辑密码文件
  1. root@ubuntu01:~# su postgres
  2. postgres@ubuntu01:/root$
  3. postgres@ubuntu01:/root$
  4. postgres@ubuntu01:/root$ cd /home/postgres/
  5. postgres@ubuntu01:~$
  6. postgres@ubuntu01:~$ vi ~/.pgpass
  7. 192.168.152.100:9000:repmgr:repmgr:password123456
  8. 192.168.152.100:9000:repmgr:repmgr:password123456
  9. 192.168.152.100:9000:replication:repmgr:password123456
  10. 192.168.152.100:9000:replication:repmgr:password123456
  11. 192.168.152.101:9000:repmgr:repmgr:password123456
  12. 192.168.152.101:9000:repmgr:repmgr:password123456
  13. 192.168.152.101:9000:replication:repmgr:password123456
  14. 192.168.152.101:9000:replication:repmgr:password123456
  15. 192.168.152.102:9000:repmgr:repmgr:password123456
  16. 192.168.152.102:9000:repmgr:repmgr:password123456
  17. 192.168.152.102:9000:replication:repmgr:password123456
  18. 192.168.152.102:9000:replication:repmgr:password12345
复制代码
对密码文件重新授权，一定一定一定要授权，否则后续节点注册的时间会提示connection to server at "192.168.152.100", port 9000 failed: fe_sendauth: no password supplied
chmod 0600 ~/.pgpass
 
###9 注册主节点
  1. postgres@ubuntu01:~$ /usr/local/pgsql16/server/bin/repmgr -f /usr/local/pgsql16/repmgr/repmgr.conf primary register
  2. INFO: connecting to primary database...
  3. INFO: "repmgr" extension is already installed
  4. NOTICE: primary node record (ID: 100) registered
  5. postgres@ubuntu01:~$
复制代码
此时可以从repmgr数据库的表中查询到已注册的节点
  1. select * from repmgr.repmgr.nodes;
  2. node_id|upstream_node_id|active|node_name|type   |location|priority|conninfo                                                                                            |repluser|slot_name|config_file                          |
  3. -------+----------------+------+---------+-------+--------+--------+----------------------------------------------------------------------------------------------------+--------+---------+-------------------------------------+
  4.     100|                |true  |ubuntu01 |primary|default |     100|host=192.168.152.100 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100|repmgr  |         |/usr/local/pgsql16/repmgr/repmgr.conf|
复制代码
 

 

从节点上的操作步骤

###1 编译安装repmgr

与主节点不绝，略
 
###2 编辑配置文件
  1. node_id=101
  2. node_name='192.168.152.101'
  3. conninfo='host=192.168.152.101 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100'
  4. data_directory='/usr/local/pgsql16/pg9000/data'
  5. pg_bindir='/usr/local/pgsql16/server/bin'
  6. failover=automatic
  7. promote_command='/usr/local/pgsql16/server/bin/repmgr standby promote -f /usr/local/pgsql16/repmgr/repmgr.conf --log-to-file'
  8. follow_command='/usr/local/pgsql16/server/bin/repmgr standby follow -f /usr/local/pgsql16/repmgr/repmgr.conf --log-to-file --upstream-node-id=%n'
  9. log_file='/usr/local/pgsql16/repmgr/repmgr.log'
复制代码
 

###3 起首停止从节点上的服务
  1. systemctl stop postgresql9000
复制代码
备份好当前节点的数据文件，清空数据文件目录内容
 

###4 编辑密码文件
  1. root@ubuntu01:~# su postgres
  2. postgres@ubuntu01:/root$
  3. postgres@ubuntu01:/root$
  4. postgres@ubuntu01:/root$ cd /home/postgres/
  5. postgres@ubuntu01:~$
  6. postgres@ubuntu01:~$ vi ~/.pgpass
  7. 192.168.152.100:9000:repmgr:repmgr:password123456
  8. 192.168.152.100:9000:repmgr:repmgr:password123456
  9. 192.168.152.100:9000:replication:repmgr:password123456
  10. 192.168.152.100:9000:replication:repmgr:password123456
  11. 192.168.152.101:9000:repmgr:repmgr:password123456
  12. 192.168.152.101:9000:repmgr:repmgr:password123456
  13. 192.168.152.101:9000:replication:repmgr:password123456
  14. 192.168.152.101:9000:replication:repmgr:password123456
  15. 192.168.152.102:9000:repmgr:repmgr:password123456
  16. 192.168.152.102:9000:repmgr:repmgr:password123456
  17. 192.168.152.102:9000:replication:repmgr:password123456
  18. 192.168.152.102:9000:replication:repmgr:password12345
复制代码
对密码文件重新授权，一定一定一定要授权，否则会提示connection to server at "192.168.152.100", port 9000 failed: fe_sendauth: no password supplied
chmod 0600 ~/.pgpass
 

###5 从主节点clone测试
  1. /usr/local/pgsql16/server/bin/repmgr -h 192.168.152.100 -p 9000 -U repmgr -d repmgr -f /usr/local/pgsql16/repmgr/repmgr.conf standby clone --dry-run
  3. postgres@ubuntu02:/usr/local/pgsql16$ /usr/local/pgsql16/server/bin/repmgr -h 192.168.152.100 -p 9000 -U repmgr -d repmgr -f /usr/local/pgsql16/repmgr/repmgr.conf standby clone --dry-run
  4. NOTICE: destination directory "/usr/local/pgsql16/pg9000/data" provided
  5. INFO: connecting to source node
  6. DETAIL: connection string is: host=192.168.152.100 port=9000 user=repmgr dbname=repmgr
  7. DETAIL: current installation size is 28 MB
  8. INFO: "repmgr" extension is installed in database "repmgr"
  9. INFO: replication slot usage not requested;  no replication slot will be set up for this standby
  10. INFO: parameter "max_wal_senders" set to 10
  11. NOTICE: checking for available walsenders on the source node (2 required)
  12. INFO: sufficient walsenders available on the source node
  13. DETAIL: 2 required, 10 available
  14. NOTICE: checking replication connections can be made to the source server (2 required)
  15. INFO: required number of replication connections could be made to the source server
  16. DETAIL: 2 replication connections required
  17. WARNING: data checksums are not enabled and "wal_log_hints" is "off"
  18. DETAIL: pg_rewind requires "wal_log_hints" to be enabled
  19. NOTICE: standby will attach to upstream node 100
  20. HINT: consider using the -c/--fast-checkpoint option
  21. INFO: would execute:
  22.   /usr/local/pgsql16/server/bin/pg_basebackup -l "repmgr base backup"  -D /usr/local/pgsql16/pg9000/data -h 192.168.152.100 -p 9000 -U repmgr -X stream
  23. INFO: all prerequisites for "standby clone" are met
  24. postgres@ubuntu02:/usr/local/pgsql16$
复制代码
 
###6 从主节点clone
  1. postgres@ubuntu02:/usr/local/pgsql16$ /usr/local/pgsql16/server/bin/repmgr -h 192.168.152.100 -p 9000 -U repmgr -d repmgr -f /usr/local/pgsql16/repmgr/repmgr.conf standby clone
  2. NOTICE: destination directory "/usr/local/pgsql16/pg9000/data" provided
  3. INFO: connecting to source node
  4. DETAIL: connection string is: host=192.168.152.100 port=9000 user=repmgr dbname=repmgr
  5. DETAIL: current installation size is 28 MB
  6. INFO: replication slot usage not requested;  no replication slot will be set up for this standby
  7. NOTICE: checking for available walsenders on the source node (2 required)
  8. NOTICE: checking replication connections can be made to the source server (2 required)
  9. WARNING: data checksums are not enabled and "wal_log_hints" is "off"
  10. DETAIL: pg_rewind requires "wal_log_hints" to be enabled
  11. INFO: checking and correcting permissions on existing directory "/usr/local/pgsql16/pg9000/data"
  12. NOTICE: starting backup (using pg_basebackup)...
  13. HINT: this may take some time; consider using the -c/--fast-checkpoint option
  14. INFO: executing:
  15.   /usr/local/pgsql16/server/bin/pg_basebackup -l "repmgr base backup"  -D /usr/local/pgsql16/pg9000/data -h 192.168.152.100 -p 9000 -U repmgr -X stream
  16. NOTICE: standby clone (using pg_basebackup) complete
  17. NOTICE: you can now start your PostgreSQL server
  18. HINT: for example: pg_ctl -D /usr/local/pgsql16/pg9000/data start
  19. HINT: after starting the server, you need to register this standby with "repmgr standby register"
  20. postgres@ubuntu02:/usr/local/pgsql16$
复制代码
 

###7 从节点注册
  1. postgres@ubuntu02:/usr/local/pgsql16$ /usr/local/pgsql16/server/bin/repmgr -f /usr/local/pgsql16/repmgr/repmgr.conf standby register
  2. INFO: connecting to local node "192.168.152.101" (ID: 101)
  3. INFO: connecting to primary database
  4. WARNING: --upstream-node-id not supplied, assuming upstream node is primary (node ID: 100)
  5. INFO: standby registration complete
  6. NOTICE: standby node "192.168.152.101" (ID: 101) successfully registered
  7. postgres@ubuntu02:/usr/local/pgsql16$
复制代码
 

###8 主节点上：检查节点以及信息
  1. ###主节点上：检查主从复制状态
  2. select * from pg_stat_replication ;
  3. pid |usesysid|usename|application_name|client_addr    |client_hostname|client_port|backend_start                |backend_xmin|state    |sent_lsn |write_lsn|flush_lsn|replay_lsn|write_lag|flush_lag|replay_lag|sync_priority|sync_state
  4. ----+--------+-------+----------------+---------------+---------------+-----------+-----------------------------+------------+---------+---------+---------+---------+----------+---------+---------+----------+-------------+----------
  5. 2898|   16388|repmgr |192.168.152.101 |192.168.152.101|               |      53602|2024-11-07 10:09:48.526447+08|            |streaming|0/50007F0|0/50007F0|0/50007F0|0/50007F0 |         |         |          |            0|async     
  6. ###任何一个节点上：检查集群节点信息
  7. select * from repmgr.nodes;
  8. node_id|upstream_node_id|active|node_name      |type   |location|priority|conninfo                                                                                            |repluser|slot_name|config_file                          |
  9. -------+----------------+------+---------------+-------+--------+--------+----------------------------------------------------------------------------------------------------+--------+---------+-------------------------------------+
  10.     100|                |true  |ubuntu01       |primary|default |     100|host=192.168.152.100 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100|repmgr  |         |/usr/local/pgsql16/repmgr/repmgr.conf|
  11.     101|             100|true  |192.168.152.101|standby|default |     100|host=192.168.152.101 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100|repmgr  |         |/usr/local/pgsql16/repmgr/repmgr.conf|
复制代码
 
 

witness 节点的操作步骤

 
###1 编译安装repmgr

与主节点不绝，略
 

###2 编辑repmgr配置文件
  1. node_id=102
  2. node_name='192.168.152.102'
  3. conninfo='host=192.168.152.102 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100'
  4. data_directory='/usr/local/pgsql16/pg9000/data'
  5. pg_bindir='/usr/local/pgsql16/server/bin'
  6. failover=automatic
  7. promote_command='/usr/local/pgsql16/server/bin/repmgr standby promote -f /usr/local/pgsql16/repmgr/repmgr.conf --log-to-file'
  8. follow_command='/usr/local/pgsql16/server/bin/repmgr standby follow -f /usr/local/pgsql16/repmgr/repmgr.conf --log-to-file --upstream-node-id=%n'
  9. log_file='/usr/local/pgsql16/repmgr/repmgr.log'
复制代码
 

###3 编辑密码文件
  1. ###编辑密码文件
  2. postgres@ubuntu03:/usr/local/pg_install_package$ vi ~/.pgpass
  3. 192.168.152.100:9000:repmgr:repmgr:password123456
  4. 192.168.152.100:9000:repmgr:repmgr:password123456
  5. 192.168.152.100:9000:replication:repmgr:password123456
  6. 192.168.152.100:9000:replication:repmgr:password123456
  7. 192.168.152.101:9000:repmgr:repmgr:password123456
  8. 192.168.152.101:9000:repmgr:repmgr:password123456
  9. 192.168.152.101:9000:replication:repmgr:password123456
  10. 192.168.152.101:9000:replication:repmgr:password123456
  11. 192.168.152.102:9000:repmgr:repmgr:password123456
  12. 192.168.152.102:9000:repmgr:repmgr:password123456
  13. 192.168.152.102:9000:replication:repmgr:password123456
  14. 192.168.152.102:9000:replication:repmgr:password123456
复制代码
 

###4 编辑配置文件

pg_hba.conf postgresql.conf 参考主节点即可


 
###5 重启数据库
  1. systemdctl restart postgresql9000
复制代码


###6 创建repmgr用户
  1. create user repmgr replication password 'password123456';
  2. --alter  user repmgr replication password 'password123456';
  3. alter user repmgr superuser;
  4. create database repmgr owner repmgr;
  5. ALTER USER repmgr SET search_path TO repmgr, "$user", public;
  6. alter user repmgr superuser ;
复制代码
 

###7 注册witness
  1. postgres@ubuntu03:/usr/local/pg_install_package$ /usr/local/pgsql16/server/bin/repmgr -h 192.168.152.100 -U repmgr -d repmgr -p9000 -f /usr/local/pgsql16/repmgr/repmgr.conf witness register
  2. INFO: connecting to witness node "192.168.152.102" (ID: 102)
  3. INFO: connecting to primary node
  4. NOTICE: attempting to install extension "repmgr"
  5. NOTICE: "repmgr" extension successfully installed
  6. INFO: witness registration complete
  7. NOTICE: witness node "192.168.152.102" (ID: 102) successfully registered
  8. postgres@ubuntu03:/usr/local/pg_install_package$
复制代码
 

###8 检查节点信息以及复制状态
  1. ###主节点上：检查主从复制状态
  2. pid |usesysid|usename|application_name|client_addr    |client_hostname|client_port|backend_start                |backend_xmin|state    |sent_lsn |write_lsn|flush_lsn|replay_lsn|write_lag|flush_lag|replay_lag|sync_priority|sync_state
  3. ----+--------+-------+----------------+---------------+---------------+-----------+-----------------------------+------------+---------+---------+---------+---------+----------+---------+---------+----------+-------------+----------
  4. 2898|   16388|repmgr |192.168.152.101 |192.168.152.101|               |      53602|2024-11-07 10:09:48.526447+08|            |streaming|0/5000FA8|0/5000FA8|0/5000FA8|0/5000FA8 |         |         |          |            0|async     
  5. ###任何一个节点上：检查集群节点信息
  6. select * from repmgr.nodes;
  7. node_id|upstream_node_id|active|node_name      |type   |location|priority|conninfo                                                                                            |repluser|slot_name|config_file                          |
  8. -------+----------------+------+---------------+-------+--------+--------+----------------------------------------------------------------------------------------------------+--------+---------+-------------------------------------+
  9.     100|                |true  |ubuntu01       |primary|default |     100|host=192.168.152.100 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100|repmgr  |         |/usr/local/pgsql16/repmgr/repmgr.conf|
  10.     101|             100|true  |192.168.152.101|standby|default |     100|host=192.168.152.101 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100|repmgr  |         |/usr/local/pgsql16/repmgr/repmgr.conf|
  11.     102|             100|true  |192.168.152.102|witness|default |       0|host=192.168.152.102 user=repmgr dbname=repmgr password=password123456 port=9000 connect_timeout=100|repmgr  |         |/usr/local/pgsql16/repmgr/repmgr.conf|
复制代码
 
 
至此，一个最简朴的基于repmgr的PostgreSQL流复制高可用环境搭建完成，整个过程虽然不复杂，但是比较繁琐。
另外，本文是1主1从1witness的集群结构，如果是1主多从，可以重复步骤2中的从节点上操作步骤即可。

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

祗疼妳一个

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

存储 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表