MongoDB数据迁移到MySQL
1、安装MongoDB参考链接
https://blog.csdn.net/qq_45664055/article/details/119794071
http://bbs.itying.com/topic/5bd675d80e525017c449479d
1.1 创建repo文件，MongoDB数据库不在yum的官方仓库，通过配置如下文件，可以通过yum下载MongoDB
vi /etc/yum.repos.d/mongodb-org-4.4.repo
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.4/x86_64/
gpgcheck=0
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.4/x86_64/ gpgcheck=0 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc
1.2 安装MongoDB
yum install -y mongodb-org 1.3 卸载MongoDB
systemctl stop mongod 关闭
yum erase $(rpm -qa | grep mongodb-org) 卸载
#删除数据及日志
rm -rf /var/log/mongodb
rm -rf /var/lib/mongo
1.4 修改MongoDB配置文件，答应长途毗连
vi /etc/mongod.conf
net: port: 27017 #数据库端口，如果要远程连接，服务器记得开放端口
bindIp: 0.0.0.0 # 默认是 127.0.0.1 只限于本机连接，如果要远程连接修改为：0.0.0.0
1.5 启动MongoDB
service mongod start centos6的写法
systemctl start mongod centos7的写法 1.6 MongoDB的基本操纵
客户端进入MongoDB：mongo
创建数据库 ：use demo
创建表，并且数据 : db.w_demo.insert({"name":"王明","age":"10","grade":"小学"})
2、MongoDB可视化工具
https://i-blog.csdnimg.cn/direct/1db3fad1a580485d9b3a4053860b2702.png
3、创建表、添加测试数据
685758046e0fb0001dad8e8,340030000B47363438383733,8C780D32F900260383493808CC96,2024-07-04 00:00:00 055 6685758046e0fb0001dad8e9,340030000B47363438383733,8C79A06C39EE65FC81D828307124,2024-07-04 00:00:00 055 6685758046e0fb0001dad8ea,340030000B47363438383733,8C79A06C39EE632C2C12766ABC7D,2024-07-04 00:00:00 055 6685758046e0fb0001dad8eb,340030000B47363438383733,8C780D32381A65EEB9D6ACD107E7,2024-07-04 00:00:00 055 6685758046e0fb0001dad8ec,340030000B47363438383733,8C79A06C39EE65FC83D8242B91FC,2024-07-04 00:00:00 055 6685758046e0fb0001dadb53,180025000847363438383733,02818334223D7A,2024-07-04 00:00:00 125 6685758046e0fb0001dadb54,180025000847363438383733,8C7813B93818F058371851BB46ED,2024-07-04 00:00:00 125 6685758046e0fb0001dadb55,180025000847363438383733,A8001BAF809CEF25E00492C097AD,2024-07-04 00:00:00 125 6685758046e0fb0001dadb56,180025000847363438383733,8D78046A990C8E9DF09019F5FFD9,2024-07-04 00:00:00 125 6685758046e0fb0001dadb57,180025000847363438383733,02C18CB2F5ACA1,2024-07-04 00:00:00 125 6685758046e0fb0001dadb58,180025000847363438383733,200016303DA8AC,2024-07-04 00:00:00 125 6685758046e0fb0001dadb59,180025000847363438383733,02C18CB2F5ACA1,2024-07-04 00:00:00 125 6685758046e0fb0001dadb5a,180025000847363438383733,02C189B8C3FFB4,2024-07-04 00:00:00 125 6685758046e0fb0001dadb5b,180025000847363438383733,8D89805E584FE2AC38F4F65130D7,2024-07-04 00:00:00 125 6685758046e0fb0001dadb5c,180025000847363438383733,02A185BA442656,2024-07-04 00:00:00 125 6685758046e0fb0001dadb5d,180025000847363438383733,8D7805AF9909180C18041613AFAB,2024-07-04 00:00:00 125 6685758046e0fb0001dadb5e,180025000847363438383733,02E18D1AB8F754,2024-07-04 00:00:00 125 6685758046e0fb0001dadb5f,180025000847363438383733,02A184B1B5AC11,2024-07-04 00:00:00 125 6685758046e0fb0001dadb60,180025000847363438383733,80618193580D32DD1EC5D965CAAF,2024-07-04 00:00:00 125 6685758046e0fb0001dadb61,180025000847363438383733,A000019389C80030A40000B08473,2024-07-04 00:00:00 125 6685758046e0fb0001dadb62,180025000847363438383733,A8001235FF731F13FFF453FB3E9D,2024-07-04 00:00:00 125 6685758046e0fb0001dadb63,180025000847363438383733,A00015BDC2980030A400000C9499,2024-07-04 00:00:00 125 6685758046e0fb0001dadb64,180025000847363438383733,02A18639AEDAAD,2024-07-04 00:00:00 125 6685758046e0fb0001dadb65,180025000847363438383733,8D780E409908D120F0482094F4EF,2024-07-04 00:00:00 125 6685758046e0fb0001dadb66,180025000847363438383733,5D75021BAFC19A,2024-07-04 00:00:00 125 6685758046e0fb0001dadb67,180025000847363438383733,02C18930C484A8,2024-07-04 00:00:00 125 6685758046e0fb0001dadb68,180025000847363438383733,A00015BDFFD9F93B2004E186573A,2024-07-04 00:00:00 125
pip install pymongo==4.4 加载数据到mysql数据库
4、Datax - mongodb reader
参考文章
https://www.cnblogs.com/LIAOBO/p/13666845.html
4.1源码修改
https://i-blog.csdnimg.cn/direct/8c067e26a8604748abf456d96e4150d8.png
https://i-blog.csdnimg.cn/direct/f0ee9be23c674e7d8f4d73814b32ac04.png
https://i-blog.csdnimg.cn/direct/c55043415c6a496ebaee16bfd85f916e.png
https://i-blog.csdnimg.cn/direct/efdd9b7a92184b528649543a7acc7d64.png
5、修改datax源码打包
对datax的所有模块进行打包，时间比较长
mvn -U clean package assembly:assembly '-Dmaven.test.skip=true' https://i-blog.csdnimg.cn/direct/87cf164a3bde491c8fe21208c4f6b390.png
指定mongodbreader模块 以及 它所依赖的模块进行打包
mvn -U clean package -pl mongodbreader -amassembly:assembly '-Dmaven.test.skip=true' https://i-blog.csdnimg.cn/direct/e94203290b8141f59025b00a9f8b04dd.png
https://i-blog.csdnimg.cn/direct/0eb1aa4904b74b389460b67d4a049d8f.png
将idea中打的jar包上传到datax的mongodbreader下，更换原本的插件jar包
https://i-blog.csdnimg.cn/direct/508fd4a9b27c49e2bb7c4c267f820b2c.png
hutool工具类jar包上传到datax的mongodbreader的libs目录下
https://i-blog.csdnimg.cn/direct/e77a9c5dbb5e4db4addd55176c2ded90.png
6、自己写的项目 已经打jar包
6.1添加依赖
https://i-blog.csdnimg.cn/direct/ceafffb7564f487a94d315f19b4d2f98.png
6.2自定义函数，解析字段，生成系统时间（自己的需求）
https://i-blog.csdnimg.cn/direct/9ab647223d1d40f7ab9b14fb81ceb791.png
https://i-blog.csdnimg.cn/direct/7314fce5d01e409c902b9861d56198d0.png
6.3项目打jar包上传到/opt/installs/datax/plugin/reader/mongodbreader/libs/目录下
https://i-blog.csdnimg.cn/direct/78ca8494b5434948b2af63e881659177.png
7、mysql建表
create table y_demo(
device_code varchar(100),
rev_timevarchar(100),
msg_contentvarchar(100),
icaovarchar(100),
msg_typevarchar(100),
handle_timevarchar(100)
) https://i-blog.csdnimg.cn/direct/e84bd41c8a2e4accb53e78c3fb35a818.png
8.编写datax的json文件,并且测试运行
{
"job": {
"content": [
{
"reader": {
"name": "mongodbreader",
"parameter": {
"address": ["hadoop13:27017"],
"collectionName": "y_demo",
"column": [
{
"name":"deviceCode",
"type":"string"
},
{
"name":"revTime",
"type":"string"
},
{
"name":"rawDataContent",
"type":"string"
},
{
"name":"rawDataContent",
"type":"string",
"functionClassName":"com.demo.ParseContent",
"functionMethodName":"m1"
},
{
"name":"rawDataContent",
"type":"string",
"functionClassName":"com.demo.ParseContent",
"functionMethodName":"m2"
},
{
"type":"string",
"functionClassName":"com.demo.ParseContent",
"functionMethodName":"m3"
}
],
"dbName": "demo",
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": ["device_code","rev_time","msg_content","icao","msg_type","handle_time"],
"connection": [
{
"jdbcUrl": "jdbc:mysql://hadoop11:3306/test1",
"table": ["y_demo"]
}
],
"password": "123456",
"username": "root",
"writeMode": "insert"
}
}
}
],
"setting": {
"speed": {
"channel": "1"
}
}
}
} 运行datax文件：
pythonbin/datax.py job/xxx.json 9、资料
资料下载 夸克网盘分享
Datax mongodbreader源码jar包 ，更换/opt/installs/datax/plugin/reader/mongodbreader/
自定义函数的jar包 /opt/installs/datax/plugin/reader/mongodbreader/libs
hutool工具类 /opt/installs/datax/plugin/reader/mongodbreader/libs
https://i-blog.csdnimg.cn/direct/e72bf40e22b24d00a5870c93163e6038.png
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
页:
[1]