一、什么是分片
分片是一种跨多台机器分布数据的方法。MongoDB 利用分片来支持超大数据集和高吞吐量操作的部署。
存在大型数据集或高吞吐量应用程序的数据库系统可能对单个服务器的容量构成挑战。例如,较高的查询速率可能会耗尽服务器的 CPU 容量。大于系统 RAM 的工作集巨细会对磁盘驱动器的 I/O 容量造成压力。
有两种方法可解决系统增长问题:垂直扩展和水平扩展。
Vertical Scaling(垂直扩展) 涉及增大单个服务器的容量,例如利用更强大的 CPU、添加更多 RAM 或增长存储空间量。可用技能所存在的限定可能会导致单个机器对给定工作负载来说不够强大。此外,基于云的提供商存在基于可用硬件配置的硬上限。因此,垂直扩展存在现实的最大值。
Horizontal Scaling(横向扩展) 涉及将系统数据集和负载划分到多个服务器,以及按需增长服务器以进步容量。固然单个机器的总体速度或容量可能不高,但每个机器均可处理总体工作负载的一部分,因此可能会比单个高速、高容量服务器提供更高的效率。扩展部署的容量只需按需添加额外的服务器,而这可能会比单个机器的高端硬件的整体成本更低。但代价在于它会增大部署的底子设施与维护的复杂性。
MongoDB 支持通过水平扩展进行分片
二、分片集群
1、组件构成
- 分片:每个分片都包含分片数据的一个子集。每个分片都必须作为一个副本集
- mongos : mongos充当查询路由器,在客户端应用程序和分片集群之间提供接口。 mongos可以支持对冲读,以最大限度地淘汰延迟。
- 配置服务器:配置服务器会存储集群的元数据和配置设置。 从 MongoDB 3.4 开始,配置服务器必须作为副本集 (CSRS) 部署。
2、分片集群内各组件间交互
三、数据怎样切分
基于分片切分后的数据块称为 chunk,一个分片后的集合会包含多个 chunk,每个 chunk 位于哪个分片(Shard) 则记载在 Config Server(配置服务器)上。
Mongos 在操作分片集适时,会自动根据分片键找到对应的 chunk,并向该 chunk 所在的分片发起操作请求。
数据是根据分片策略来进行切分的,而分片策略则由 分片键(ShardKey)+分片算法(ShardStrategy)组成。
四、分片策略
1、哈希分片
哈希分片涉及计算分片键字段值的哈希值。然后,根据哈希分片键值为每个数据段分配一个范围。
在利用哈希索引解析查询时,MongoDB 会自动计算哈希值。应用程序无需计算哈希值
固然分片键的范围可能“相近”,但它们的哈希值却不太可能位于同一数据段。基于哈希值的数据分配可促进更均匀的数据分布,尤其是在分片键单调
然而,哈希分布意味着对分片键进行基于范围的查询时,不太可能以单个分片为目标,从而导致更多的集群范围的广播操作
2、范围分片
范围分片涉及根据分片键值将数据划分为多个范围。然后,根据分片键值为每个数据段分配一个范围。
具有“相近”数值的一系列分片键更有可能位于同一个数据段上。这允许进行有针对性的操作,因为 mongos 只能将操作路由到包含所需数据的分片。
范围分片的效率取决于所选分片键。思量不周的分片键可能会导致数据分布不均,从而抵消分片的某些好处甚或导致性能瓶颈。请参阅针对基于范围的分片的分片键选择
五、分片集群架构
两个分片节点副本集(3+3)+ 一个配置节点副本集(3)+两个路由节点(2)
共11个服务节点。
六、搭建分片集群
1、涉及主机
角色主机名IP地址分片节点1shard1192.168.112.10分片节点2shard2192.168.112.20配置节点config-server192.168.112.30路由节点router192.168.112.40 2、全部主机安装MongoDB
- wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-4.4.6.tgz
- tar xzvf mongodb-linux-x86_64-rhel70-4.4.6.tgz -C /usr/local
- cd /usr/local/
- ln -s /usr/local/mongodb-linux-x86_64-rhel70-4.4.6 /usr/local/mongodb
- vim /etc/profile
- export PATH=/usr/local/mongodb/bin/:$PATH
- source /etc/profile
复制代码 3、分片节点副本集的创建
3.1、第一套副本集shard1
shard1主机操作
3.1.1、准备存放数据和日志的目录
- mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27018/log \
- /usr/local/mongodb/sharded_cluster/myshardrs01_27018/data/db \
- /usr/local/mongodb/sharded_cluster/myshardrs01_27118/log \
- /usr/local/mongodb/sharded_cluster/myshardrs01_27118/data/db \
- /usr/local/mongodb/sharded_cluster/myshardrs01_27218/log \
- /usr/local/mongodb/sharded_cluster/myshardrs01_27218/data/db
复制代码 3.1.2、创建配置文件
myshardrs01_27018
- vim /usr/local/mongodb/sharded_cluster/myshardrs01_27018/mongod.conf
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myshardrs01_27018/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27018/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27018/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.10
- port: 27018
- replication:
- replSetName: myshardrs01
- sharding:
- clusterRole: shardsvr
复制代码 myshardrs01_27118
- vim /usr/local/mongodb/sharded_cluster/myshardrs01_27118/mongod.conf
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myshardrs01_27118/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27118/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27118/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.10
- port: 27118
- replication:
- replSetName: myshardrs01
- sharding:
- clusterRole: shardsvr
复制代码 myshardrs01_27218
- vim /usr/local/mongodb/sharded_cluster/myshardrs01_27218/mongod.conf
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myshardrs01_27218/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27218/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27218/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.10
- port: 27218
- replication:
- replSetName: myshardrs01
- sharding:
- clusterRole: shardsvr
复制代码 3.1.3、启动第一套副本集:一主一副本一仲裁
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27018/mongod.conf
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27118/mongod.conf
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27218/mongod.conf
复制代码
3.1.4、初始化副本集、添加副本,仲裁节点
毗连主节点
- mongo --host 192.168.112.10 --port 27018
复制代码
- rs.initiate()
- {
- "info2" : "no configuration specified. Using a default configuration for the set",
- "me" : "192.168.112.10:27018",
- "ok" : 1
- }
- myshardrs01:SECONDARY>
- myshardrs01:PRIMARY>
复制代码
- rs.add("192.168.112.10:27118")
- {
- "ok" : 1,
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714658449, 1),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- },
- "operationTime" : Timestamp(1714658449, 1)
- }
复制代码
- rs.addArb("192.168.112.10:27218")
- {
- "ok" : 1,
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714658509, 1),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- },
- "operationTime" : Timestamp(1714658509, 1)
- }
复制代码
- 查看副本集(关注各成员角色信息、健康状态即可)
- rs.conf()
- 作用:此命令用于表现当前副本集的配置信息。它返回一个文档,包含了副本集的名称、版本、协议版本、成员列表(包罗每个成员的ID、主机地址、优先级、角色等信息)、设置选项等详细配置细节。
- 应用场景:当你需要查看或修改副本集的配置时(比如添加新成员、调整成员优先级、查看仲裁节点配置等),可以利用此命令。获取到的配置信息可以进一步用于重新配置副本集(rs.reconfig())。
- rs.status()
- 作用:此命令提供了副本集当前状态的快照,包罗各成员的运行状态、主从角色、健康状态、同步进度、心跳信息、选举信息等。它帮助你了解副本集是否运行正常,成员之间是否保持同步,以及是否有主节点故障等情况。
- 应用场景:当监控副本集健康状态、排查毗连问题、查抄数据同步情况或确认主节点变更时,利用此命令非常有用。通过查抄rs.status()的输出,管理员可以快速识别并解决集群中的问题。
3.2、第二套副本集shard2
shard2主机操作
3.2.1、准备存放数据和日志的目录
- mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs02_27318/log \
- /usr/local/mongodb/sharded_cluster/myshardrs02_27318/data/db \
- /usr/local/mongodb/sharded_cluster/myshardrs02_27418/log \
- /usr/local/mongodb/sharded_cluster/myshardrs02_27418/data/db \
- /usr/local/mongodb/sharded_cluster/myshardrs02_27518/log \
- /usr/local/mongodb/sharded_cluster/myshardrs02_27518/data/db
复制代码 3.1.2、创建配置文件
myshardrs02_27318
- vim /usr/local/mongodb/sharded_cluster/myshardrs02_27318/mongod.conf
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myshardrs02_27318/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs02_27318/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs02_27318/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.20
- port: 27318
- replication:
- replSetName: myshardrs02
- sharding:
- clusterRole: shardsvr
复制代码 myshardrs02_27418
- vim /usr/local/mongodb/sharded_cluster/myshardrs02_27418/mongod.conf
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myshardrs02_27418/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs02_27418/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs02_27418/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.20
- port: 27418
- replication:
- replSetName: myshardrs02
- sharding:
- clusterRole: shardsvr
复制代码 myshardrs02_27518
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myshardrs02_27518/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs02_27518/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs02_27518/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.20
- port: 27518
- replication:
- replSetName: myshardrs02
- sharding:
- clusterRole: shardsvr
复制代码 3.2.3、启动第二套副本集:一主一副本一仲裁
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs02_27318/mongod.conf
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs02_27418/mongod.conf
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs02_27518/mongod.conf
复制代码
3.2.4、初始化副本集、添加副本,仲裁节点
毗连主节点
- mongo --host 192.168.112.20 --port 27318
复制代码
- rs.initiate()
- {
- "info2" : "no configuration specified. Using a default configuration for the set",
- "me" : "192.168.112.20:27318",
- "ok" : 1
- }
- myshardrs02:SECONDARY>
- myshardrs02:PRIMARY>
复制代码
- rs.add("192.168.112.20:27418")
- {
- "ok" : 1,
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714660065, 1),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- },
- "operationTime" : Timestamp(1714660065, 1)
- }
复制代码
- rs.addArb("192.168.112.20:27518")
- {
- "ok" : 1,
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714660094, 1),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- },
- "operationTime" : Timestamp(1714660094, 1)
- }
复制代码
- rs.status()
- {
- "set" : "myshardrs02",
- "date" : ISODate("2024-05-02T14:29:57.930Z"),
- "myState" : 1,
- "term" : NumberLong(1),
- "syncSourceHost" : "",
- "syncSourceId" : -1,
- "heartbeatIntervalMillis" : NumberLong(2000),
- "majorityVoteCount" : 2,
- "writeMajorityCount" : 2,
- "votingMembersCount" : 3,
- "writableVotingMembersCount" : 2,
- "optimes" : {
- "lastCommittedOpTime" : {
- "ts" : Timestamp(1714660197, 1),
- "t" : NumberLong(1)
- },
- "lastCommittedWallTime" : ISODate("2024-05-02T14:29:57.868Z"),
- "readConcernMajorityOpTime" : {
- "ts" : Timestamp(1714660197, 1),
- "t" : NumberLong(1)
- },
- "readConcernMajorityWallTime" : ISODate("2024-05-02T14:29:57.868Z"),
- "appliedOpTime" : {
- "ts" : Timestamp(1714660197, 1),
- "t" : NumberLong(1)
- },
- "durableOpTime" : {
- "ts" : Timestamp(1714660197, 1),
- "t" : NumberLong(1)
- },
- "lastAppliedWallTime" : ISODate("2024-05-02T14:29:57.868Z"),
- "lastDurableWallTime" : ISODate("2024-05-02T14:29:57.868Z")
- },
- "lastStableRecoveryTimestamp" : Timestamp(1714660147, 1),
- "electionCandidateMetrics" : {
- "lastElectionReason" : "electionTimeout",
- "lastElectionDate" : ISODate("2024-05-02T14:27:07.797Z"),
- "electionTerm" : NumberLong(1),
- "lastCommittedOpTimeAtElection" : {
- "ts" : Timestamp(0, 0),
- "t" : NumberLong(-1)
- },
- "lastSeenOpTimeAtElection" : {
- "ts" : Timestamp(1714660027, 1),
- "t" : NumberLong(-1)
- },
- "numVotesNeeded" : 1,
- "priorityAtElection" : 1,
- "electionTimeoutMillis" : NumberLong(10000),
- "newTermStartDate" : ISODate("2024-05-02T14:27:07.802Z"),
- "wMajorityWriteAvailabilityDate" : ISODate("2024-05-02T14:27:07.899Z")
- },
- "members" : [
- {
- "_id" : 0,
- "name" : "192.168.112.20:27318",
- "health" : 1,
- "state" : 1,
- "stateStr" : "PRIMARY",
- "uptime" : 403,
- "optime" : {
- "ts" : Timestamp(1714660197, 1),
- "t" : NumberLong(1)
- },
- "optimeDate" : ISODate("2024-05-02T14:29:57Z"),
- "syncSourceHost" : "",
- "syncSourceId" : -1,
- "infoMessage" : "",
- "electionTime" : Timestamp(1714660027, 2),
- "electionDate" : ISODate("2024-05-02T14:27:07Z"),
- "configVersion" : 3,
- "configTerm" : -1,
- "self" : true,
- "lastHeartbeatMessage" : ""
- },
- {
- "_id" : 1,
- "name" : "192.168.112.20:27418",
- "health" : 1,
- "state" : 2,
- "stateStr" : "SECONDARY",
- "uptime" : 132,
- "optime" : {
- "ts" : Timestamp(1714660187, 1),
- "t" : NumberLong(1)
- },
- "optimeDurable" : {
- "ts" : Timestamp(1714660187, 1),
- "t" : NumberLong(1)
- },
- "optimeDate" : ISODate("2024-05-02T14:29:47Z"),
- "optimeDurableDate" : ISODate("2024-05-02T14:29:47Z"),
- "lastHeartbeat" : ISODate("2024-05-02T14:29:57.035Z"),
- "lastHeartbeatRecv" : ISODate("2024-05-02T14:29:57.113Z"),
- "pingMs" : NumberLong(0),
- "lastHeartbeatMessage" : "",
- "syncSourceHost" : "192.168.112.20:27318",
- "syncSourceId" : 0,
- "infoMessage" : "",
- "configVersion" : 3,
- "configTerm" : -1
- },
- {
- "_id" : 2,
- "name" : "192.168.112.20:27518",
- "health" : 1,
- "state" : 7,
- "stateStr" : "ARBITER",
- "uptime" : 102,
- "lastHeartbeat" : ISODate("2024-05-02T14:29:57.035Z"),
- "lastHeartbeatRecv" : ISODate("2024-05-02T14:29:57.068Z"),
- "pingMs" : NumberLong(0),
- "lastHeartbeatMessage" : "",
- "syncSourceHost" : "",
- "syncSourceId" : -1,
- "infoMessage" : "",
- "configVersion" : 3,
- "configTerm" : -1
- }
- ],
- "ok" : 1,
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714660197, 1),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- },
- "operationTime" : Timestamp(1714660197, 1)
- }
复制代码 4、配置节点副本集的创建
config-server主机操作
4.1、准备存放数据和日志的目录
- mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27019/log \
- /usr/local/mongodb/sharded_cluster/myconfigrs_27019/data/db \
- /usr/local/mongodb/sharded_cluster/myconfigrs_27119/log \
- /usr/local/mongodb/sharded_cluster/myconfigrs_27119/data/db \
- /usr/local/mongodb/sharded_cluster/myconfigrs_27219/log \
- /usr/local/mongodb/sharded_cluster/myconfigrs_27219/data/db
复制代码 4.2、创建配置文件
myconfigrs_27019
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myconfigrs_27019/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27019/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27019/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.30
- port: 27019
- replication:
- replSetName: myconfigr
- sharding:
- clusterRole: configsvr
复制代码 注意:这里的配置文件与分片节点的配置文件不一样
分片角色:shardsvr为分片节点,configsvr配置节点
myconfigrs_27119
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myconfigrs_27119/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27119/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27119/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.30
- port: 27119
- replication:
- replSetName: myconfigr
- sharding:
- clusterRole: configsvr
复制代码 myconfigrs_27219
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/myconfigrs_27219/log/mongod.log"
- logAppend: true
- storage:
- dbPath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27219/data/db"
- journal:
- enabled: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27219/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.30
- port: 27219
- replication:
- replSetName: myconfigr
- sharding:
- clusterRole: configsvr
复制代码 4.3、启动副本集
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myconfigrs_27019/mongod.conf
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myconfigrs_27119/mongod.conf
- /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myconfigrs_27219/mongod.conf
复制代码
4.4、初始化副本集、添加副本节点
毗连主节点
- mongo --host 192.168.112.30 --port 27019
复制代码
- rs.initiate()
- {
- "info2" : "no configuration specified. Using a default configuration for the set",
- "me" : "192.168.112.30:27019",
- "ok" : 1,
- "$gleStats" : {
- "lastOpTime" : Timestamp(1714702204, 1),
- "electionId" : ObjectId("000000000000000000000000")
- },
- "lastCommittedOpTime" : Timestamp(0, 0)
- }
复制代码
- rs.add("192.168.112.30:27119")
- {
- "ok" : 1,
- "$gleStats" : {
- "lastOpTime" : {
- "ts" : Timestamp(1714702333, 2),
- "t" : NumberLong(1)
- },
- "electionId" : ObjectId("7fffffff0000000000000001")
- },
- "lastCommittedOpTime" : Timestamp(1714702333, 2),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714702335, 1),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- },
- "operationTime" : Timestamp(1714702333, 2)
- }
复制代码- rs.add("192.168.112.30:27219")
- {
- "ok" : 1,
- "$gleStats" : {
- "lastOpTime" : {
- "ts" : Timestamp(1714702365, 2),
- "t" : NumberLong(1)
- },
- "electionId" : ObjectId("7fffffff0000000000000001")
- },
- "lastCommittedOpTime" : Timestamp(1714702367, 1),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714702367, 1),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- },
- "operationTime" : Timestamp(1714702365, 2)
- }
复制代码
- rs.conf()
- {
- "_id" : "myconfigr",
- "version" : 3,
- "term" : 1,
- "configsvr" : true,
- "protocolVersion" : NumberLong(1),
- "writeConcernMajorityJournalDefault" : true,
- "members" : [
- {
- "_id" : 0,
- "host" : "192.168.112.30:27019",
- "arbiterOnly" : false,
- "buildIndexes" : true,
- "hidden" : false,
- "priority" : 1,
- "tags" : {
- },
- "slaveDelay" : NumberLong(0),
- "votes" : 1
- },
- {
- "_id" : 1,
- "host" : "192.168.112.30:27119",
- "arbiterOnly" : false,
- "buildIndexes" : true,
- "hidden" : false,
- "priority" : 1,
- "tags" : {
- },
- "slaveDelay" : NumberLong(0),
- "votes" : 1
- },
- {
- "_id" : 2,
- "host" : "192.168.112.30:27219",
- "arbiterOnly" : false,
- "buildIndexes" : true,
- "hidden" : false,
- "priority" : 1,
- "tags" : {
- },
- "slaveDelay" : NumberLong(0),
- "votes" : 1
- }
- ],
- "settings" : {
- "chainingAllowed" : true,
- "heartbeatIntervalMillis" : 2000,
- "heartbeatTimeoutSecs" : 10,
- "electionTimeoutMillis" : 10000,
- "catchUpTimeoutMillis" : -1,
- "catchUpTakeoverDelayMillis" : 30000,
- "getLastErrorModes" : {
- },
- "getLastErrorDefaults" : {
- "w" : 1,
- "wtimeout" : 0
- },
- "replicaSetId" : ObjectId("6634477c7677719ed2388701")
- }
- }
复制代码 5、路由节点的创建和操作
router主机操作
5.1、创建毗连第一个路由节点
5.1.1、准备存放日志的目录
- mkdir -p /usr/local/mongodb/sharded_cluster/mymongos_27017/log
复制代码 5.1.2、创建配置文件
- vim /usr/local/mongodb/sharded_cluster/mymongos_27017/mongos.conf
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/mymongos_27017/log/mongod.log"
- logAppend: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/mymongos_27017/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.40
- port: 27017
- sharding:
- configDB: myconfigrs/192.168.112.30:27019,192.168.112.30:27119,192.168.112.30:27219
复制代码 5.1.3、启动mongos
- /usr/local/mongodb/bin/mongos -f /usr/local/mongodb/sharded_cluster/mymongos_27017/mongos.conf
复制代码 5.1.4、客户端登录mongos
- mongos> use test1
- switched to db test1
- mongos> db.q1.insert({a:"a1"})
- WriteCommandError({
- "ok" : 0,
- "errmsg" : "unable to initialize targeter for write op for collection test1.q1 :: caused by :: Database test1 could not be created :: caused by :: No shards found",
- "code" : 70,
- "codeName" : "ShardNotFound",
- "operationTime" : Timestamp(1714704496, 3),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714704496, 3),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- }
- })
复制代码 此时写入不了数据
通过路由节点操作,如今只是毗连了配置节点
还没有毗连分片数据节点,因此无法写入业务数据
5.2、添加分片副本集
5.2.1、将第一套分片副本集添加进来
- sh.addShard("myshardrs01/192.168.112.10:27018,192.168.112.10:27118,192.168.112.10:27218")
- {
- "shardAdded" : "myshardrs01",
- "ok" : 1,
- "operationTime" : Timestamp(1714705107, 3),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714705107, 3),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- }
- }
复制代码 5.2.2、查看分片状态情况
- sh.status()
- --- Sharding Status ---
- sharding version: {
- "_id" : 1,
- "minCompatibleVersion" : 5,
- "currentVersion" : 6,
- "clusterId" : ObjectId("6634477c7677719ed2388706")
- }
- shards:
- { "_id" : "myshardrs01", "host" : "myshardrs01/192.168.112.10:27018,192.168.112.10:27118", "state" : 1 }
- most recently active mongoses:
- "4.4.6" : 1
- autosplit:
- Currently enabled: yes
- balancer:
- Currently enabled: yes
- Currently running: no
- Failed balancer rounds in last 5 attempts: 0
- Migration Results for the last 24 hours:
- No recent migrations
- databases:
- { "_id" : "config", "primary" : "config", "partitioned" : true }
复制代码 5.2.3、将第二套分片副本集添加进来
- sh.addShard("myshardrs02/192.168.112.20:27318,192.168.112.20:27418,192.168.112.20:27518")
- {
- "shardAdded" : "myshardrs02",
- "ok" : 1,
- "operationTime" : Timestamp(1714705261, 2),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714705261, 2),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- }
- }
复制代码 5.2.4、查看分片状态情况
- sh.status()
- --- Sharding Status ---
- sharding version: {
- "_id" : 1,
- "minCompatibleVersion" : 5,
- "currentVersion" : 6,
- "clusterId" : ObjectId("6634477c7677719ed2388706")
- }
- shards:
- { "_id" : "myshardrs01", "host" : "myshardrs01/192.168.112.10:27018,192.168.112.10:27118", "state" : 1 }
- { "_id" : "myshardrs02", "host" : "myshardrs02/192.168.112.20:27318,192.168.112.20:27418", "state" : 1 }
- most recently active mongoses:
- "4.4.6" : 1
- autosplit:
- Currently enabled: yes
- balancer:
- Currently enabled: yes
- Currently running: no
- Failed balancer rounds in last 5 attempts: 0
- Migration Results for the last 24 hours:
- 34 : Success
- databases:
- { "_id" : "config", "primary" : "config", "partitioned" : true }
- config.system.sessions
- shard key: { "_id" : 1 }
- unique: false
- balancing: true
- chunks:
- myshardrs01 990
- myshardrs02 34
- too many chunks to print, use verbose if you want to force print
复制代码 5.3、移除分片
假如只剩下末了一个shard,是无法删除的。
移除时会自动转移分片数据,需要一个时间过程。
完成后,再次执行删除分片命令才气真正删除。
- #示例:移除分片myshardrs02
- use db
- db.runCommand({removeShard: "myshardrs02"})
复制代码 5.4、开启分片功能
- sh.enableSharding(“库名”)
- sh.shardCollection(“库名.集合名”,{“key”:1})
- mongos> sh.enableSharding("test")
- {
- "ok" : 1,
- "operationTime" : Timestamp(1714706298, 6),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714706298, 6),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- }
- }
复制代码 5.5、针对分片策略–哈希分片
router主机操作
对comment这个集合利用hash方式分片
- sh.shardCollection("test.comment",{"nickname":"hashed"})
- {
- "collectionsharded" : "test.comment",
- "collectionUUID" : UUID("61ea1ec6-500b-4d00-86ed-3b2cce81a021"),
- "ok" : 1,
- "operationTime" : Timestamp(1714706573, 25),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714706573, 25),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- }
- }
复制代码
- sh.status()
- --- Sharding Status ---
- sharding version: {
- "_id" : 1,
- "minCompatibleVersion" : 5,
- "currentVersion" : 6,
- "clusterId" : ObjectId("6634477c7677719ed2388706")
- }
- shards:
- { "_id" : "myshardrs01", "host" : "myshardrs01/192.168.112.10:27018,192.168.112.10:27118", "state" : 1 }
- { "_id" : "myshardrs02", "host" : "myshardrs02/192.168.112.20:27318,192.168.112.20:27418", "state" : 1 }
- most recently active mongoses:
- "4.4.6" : 1
- autosplit:
- Currently enabled: yes
- balancer:
- Currently enabled: yes
- Currently running: no
- Failed balancer rounds in last 5 attempts: 0
- Migration Results for the last 24 hours:
- 512 : Success
- databases:
- { "_id" : "config", "primary" : "config", "partitioned" : true }
- config.system.sessions
- shard key: { "_id" : 1 }
- unique: false
- balancing: true
- chunks:
- myshardrs01 512
- myshardrs02 512
- too many chunks to print, use verbose if you want to force print
- { "_id" : "test", "primary" : "myshardrs02", "partitioned" : true, "version" : { "uuid" : UUID("7f240c27-83dd-48eb-be8c-2e1336ec9190"), "lastMod" : 1 } }
- test.comment
- shard key: { "nickname" : "hashed" }
- unique: false
- balancing: true
- chunks:
- myshardrs01 2
- myshardrs02 2
- { "nickname" : { "$minKey" : 1 } } -->> { "nickname" : NumberLong("-4611686018427387902") } on : myshardrs01 Timestamp(1, 0)
- { "nickname" : NumberLong("-4611686018427387902") } -->> { "nickname" : NumberLong(0) } on : myshardrs01 Timestamp(1, 1)
- { "nickname" : NumberLong(0) } -->> { "nickname" : NumberLong("4611686018427387902") } on : myshardrs02 Timestamp(1, 2)
- { "nickname" : NumberLong("4611686018427387902") } -->> { "nickname" : { "$maxKey" : 1 } } on : myshardrs02 Timestamp(1, 3)
复制代码 5.5.1、插入数据测试
哈希分片
向comment集合循环插入1000条数据做测试
- mongos> use test
- switched to db test
- mongos> for(var i=1;i<=1000;i++){db.comment.insert({_id:i+"",nickname:"Test"+i})}
- WriteResult({ "nInserted" : 1 })
- mongos> db.comment.count()
- 1000
复制代码 5.5.2、分别登录两个分片节点主节点,统计文档数量
- myshardrs01:
- mongo --host 192.168.112.10 --port 27018
- myshardrs01:PRIMARY> show dbsadmin 0.000GBconfig 0.001GBlocal 0.001GBtest 0.000GBmyshardrs01:PRIMARY> use testswitched to db testmyshardrs01:PRIMARY> db.comment.count()505
复制代码- myshardrs01:PRIMARY> db.comment.find()
- { "_id" : "2", "nickname" : "Test2" }
- { "_id" : "4", "nickname" : "Test4" }
- { "_id" : "8", "nickname" : "Test8" }
- { "_id" : "9", "nickname" : "Test9" }
- { "_id" : "13", "nickname" : "Test13" }
- { "_id" : "15", "nickname" : "Test15" }
- { "_id" : "16", "nickname" : "Test16" }
- { "_id" : "18", "nickname" : "Test18" }
- { "_id" : "19", "nickname" : "Test19" }
- { "_id" : "20", "nickname" : "Test20" }
- { "_id" : "21", "nickname" : "Test21" }
- { "_id" : "25", "nickname" : "Test25" }
- { "_id" : "26", "nickname" : "Test26" }
- { "_id" : "27", "nickname" : "Test27" }
- { "_id" : "31", "nickname" : "Test31" }
- { "_id" : "32", "nickname" : "Test32" }
- { "_id" : "33", "nickname" : "Test33" }
- { "_id" : "35", "nickname" : "Test35" }
- { "_id" : "36", "nickname" : "Test36" }
- { "_id" : "38", "nickname" : "Test38" }
- Type "it" for more
复制代码 基于哈希值的数据分配
- myshardrs02:
- mongo --port 27318
- myshardrs02:PRIMARY> show dbs
- admin 0.000GB
- config 0.001GB
- local 0.001GB
- test 0.000GB
- myshardrs02:PRIMARY> use test
- switched to db test
- myshardrs02:PRIMARY> db.comment.count()
- 495
复制代码- myshardrs02:PRIMARY> db.comment.find()
- { "_id" : "1", "nickname" : "Test1" }
- { "_id" : "3", "nickname" : "Test3" }
- { "_id" : "5", "nickname" : "Test5" }
- { "_id" : "6", "nickname" : "Test6" }
- { "_id" : "7", "nickname" : "Test7" }
- { "_id" : "10", "nickname" : "Test10" }
- { "_id" : "11", "nickname" : "Test11" }
- { "_id" : "12", "nickname" : "Test12" }
- { "_id" : "14", "nickname" : "Test14" }
- { "_id" : "17", "nickname" : "Test17" }
- { "_id" : "22", "nickname" : "Test22" }
- { "_id" : "23", "nickname" : "Test23" }
- { "_id" : "24", "nickname" : "Test24" }
- { "_id" : "28", "nickname" : "Test28" }
- { "_id" : "29", "nickname" : "Test29" }
- { "_id" : "30", "nickname" : "Test30" }
- { "_id" : "34", "nickname" : "Test34" }
- { "_id" : "37", "nickname" : "Test37" }
- { "_id" : "39", "nickname" : "Test39" }
- { "_id" : "44", "nickname" : "Test44" }
- Type "it" for more
复制代码 基于哈希值的数据分配
5.6、针对分片策略–范围分片
router主机操作
利用作者年龄字段作为片键,按照年龄的值进行分片
- mongos> sh.shardCollection("test.author",{"age":1})
- {
- "collectionsharded" : "test.author",
- "collectionUUID" : UUID("9b8055c8-7a35-49d5-9e8f-07d6004b19b3"),
- "ok" : 1,
- "operationTime" : Timestamp(1714709641, 12),
- "$clusterTime" : {
- "clusterTime" : Timestamp(1714709641, 12),
- "signature" : {
- "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
- "keyId" : NumberLong(0)
- }
- }
- }
复制代码 5.6.1、插入数据测试
范围分片
向author集合循环插入20000条测试数据
- mongos> use test
- switched to db test
- mongos> for (var i=1;i<=2000;i++){db.author.save({"name":"test"+i,"age":NumberInt(i%120)})}
- WriteResult({ "nInserted" : 1 })
- mongos> db.author.count()
- 2000
复制代码 5.6.2、分别登录两个分片节点主节点,统计文档数量
- myshardrs02
- myshardrs02:PRIMARY> show dbs
- admin 0.000GB
- config 0.001GB
- local 0.001GB
- test 0.000GB
- myshardrs02:PRIMARY> use test
- switched to db test
- myshardrs02:PRIMARY> show collections
- author
- comment
- myshardrs02:PRIMARY> db.author.count()
- 2000
复制代码
- myshardrs01
- myshardrs01:PRIMARY> db.author.count()
- 0
复制代码
- 发现全部的数据都集中在了一个分片副本上
- 假如发现没有分片:
- 系统繁忙,正在分片中
- 数据块(chunk)没有填满,默认的数据块尺寸是64M,填满后才会向其他片的数据库填充数据,为了测试可以改小,但是生产环境请勿改动
- use config
- db.settings.save({_id:"chunksize",value:1})
- # 改成1M
- db.settings.save({_id:"chunksize",value:64})
复制代码 6、再添加一个路由节点
router主机操作
6.1、准备存放日志的目录
- mkdir -p /usr/local/mongodb/sharded_cluster/mymongos_27117/log
复制代码 6.2、创建配置文件
- vim /usr/local/mongodb/sharded_cluster/mymongos_27117/mongos.conf
- systemLog:
- destination: file
- path: "/usr/local/mongodb/sharded_cluster/mymongos_27117/log/mongod.log"
- logAppend: true
- processManagement:
- fork: true
- pidFilePath: "/usr/local/mongodb/sharded_cluster/mymongos_27117/log/mongod.pid"
- net:
- bindIp: localhost,192.168.112.40
- port: 27117
- sharding:
- configDB: myconfigrs/192.168.112.30:27019,192.168.112.30:27119,192.168.112.30:27219
复制代码 6.3、启动mongos
- /usr/local/mongodb/bin/mongos -f /usr/local/mongodb/sharded_cluster/mymongos_27117/mongos.conf
复制代码 6.4、客户端登录mongos
利用mongo客户端登录27117
发现第二个路由无需配置,因为分片配置都保存到了配置服务器中了
- mongo --port 27117
- mongos> sh.status()
- --- Sharding Status ---
- sharding version: {
- "_id" : 1,
- "minCompatibleVersion" : 5,
- "currentVersion" : 6,
- "clusterId" : ObjectId("6634477c7677719ed2388706")
- }
- shards:
- { "_id" : "myshardrs01", "host" : "myshardrs01/192.168.112.10:27018,192.168.112.10:27118", "state" : 1 }
- { "_id" : "myshardrs02", "host" : "myshardrs02/192.168.112.20:27318,192.168.112.20:27418", "state" : 1 }
- active mongoses:
- "4.4.6" : 2
- autosplit:
- Currently enabled: yes
- balancer:
- Currently enabled: yes
- Currently running: no
- Failed balancer rounds in last 5 attempts: 0
- Migration Results for the last 24 hours:
- 512 : Success
- databases:
- { "_id" : "config", "primary" : "config", "partitioned" : true }
- config.system.sessions
- shard key: { "_id" : 1 }
- unique: false
- balancing: true
- chunks:
- myshardrs01 512
- myshardrs02 512
- too many chunks to print, use verbose if you want to force print
- { "_id" : "test", "primary" : "myshardrs02", "partitioned" : true, "version" : { "uuid" : UUID("7f240c27-83dd-48eb-be8c-2e1336ec9190"), "lastMod" : 1 } }
- test.author
- shard key: { "age" : 1 }
- unique: false
- balancing: true
- chunks:
- myshardrs02 1
- { "age" : { "$minKey" : 1 } } -->> { "age" : { "$maxKey" : 1 } } on : myshardrs02 Timestamp(1, 0)
- test.comment
- shard key: { "nickname" : "hashed" }
- unique: false
- balancing: true
- chunks:
- myshardrs01 2
- myshardrs02 2
- { "nickname" : { "$minKey" : 1 } } -->> { "nickname" : NumberLong("-4611686018427387902") } on : myshardrs01 Timestamp(1, 0)
- { "nickname" : NumberLong("-4611686018427387902") } -->> { "nickname" : NumberLong(0) } on : myshardrs01 Timestamp(1, 1)
- { "nickname" : NumberLong(0) } -->> { "nickname" : NumberLong("4611686018427387902") } on : myshardrs02 Timestamp(1, 2)
- { "nickname" : NumberLong("4611686018427387902") } -->> { "nickname" : { "$maxKey" : 1 } } on : myshardrs02 Timestamp(1, 3)
复制代码 6.5、查看之前测试的数据
- mongos> show dbs
- admin 0.000GB
- config 0.003GB
- test 0.000GB
- mongos> use test
- switched to db test
- mongos> db.comment.count()
- 1000
- mongos> db.author.count()
- 2000
复制代码 至此两个分片节点副本集(3+3)+ 一个配置节点副本集(3)+两个路由节点(2)的分片集群搭建完成
免责声明:如果侵犯了您的权益,请联系站长,我们会及时删除侵权内容,谢谢合作!更多信息从访问主页:qidao123.com:ToB企服之家,中国第一个企服评测及商务社交产业平台。 |