DataX vs 腾讯云COS对象存储 -> StarRocks集群

飞不高  金牌会员 | 2024-2-13 15:03:37 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 662|帖子 662|积分 1986

本文将介绍使用DataX读出Cos的Orc文件往StarRocks里面写。
 
需求: 需要将腾讯云cos上84TB的数据, 同步到StarRocks某个大表。正常每个分区数据量20~30亿，600GB。
工具：DataX
插件：hdfsreader、starrockswriter
对象存储COS：非融合
 
DataX

这里我使用的datax版本是 DataX (DATAX-OPENSOURCE-3.0)
  1. [svccnetlhs@HOST datax]<231211 17:17:11>$ tree bin/ conf/
  2. bin/
  3. ├── datax.py
  4. ├── dxprof.py
  5. └── perftrace.py
  6. conf/
  7. ├── core.json
  8. └── logback.xml
  10. 0 directories, 5 files
  11. [svccnetlhs@HOST datax]<231211 17:18:52>$ /bin/python3
  12. python3     python3.6   python3.6m  
  13. [svccnetlhs@HOST datax]<231211 17:18:52>$ /bin/python3 bin/datax.py
  15. DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
  16. Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
  19. Usage: datax.py [options] job-url-or-path
  21. Options:
  22.   -h, --help            show this help message and exit
  24.   Product Env Options:
  25.     Normal user use these options to set jvm parameters, job runtime mode
  26.     etc. Make sure these options can be used in Product Env.
  28.     -j <jvm parameters>, --jvm=<jvm parameters>
  29.                         Set jvm parameters if necessary.
  30.     --jobid=<job unique id>
  31.                         Set job unique id when running by Distribute/Local
  32.                         Mode.
  33.     -m <job runtime mode>, --mode=<job runtime mode>
  34.                         Set job runtime mode such as: standalone, local,
  35.                         distribute. Default mode is standalone.
  36.     -p <parameter used in job config>, --params=<parameter used in job config>
  37.                         Set job parameter, eg: the source tableName you want
  38.                         to set it by command, then you can use like this:
  39.                         -p"-DtableName=your-table-name", if you have mutiple
  40.                         parameters: -p"-DtableName=your-table-name
  41.                         -DcolumnName=your-column-name".Note: you should config
  42.                         in you job tableName with ${tableName}.
  43.     -r <parameter used in view job config[reader] template>, --reader=<parameter used in view job config[reader] template>
  44.                         View job config[reader] template, eg:
  45.                         mysqlreader,streamreader
  46.     -w <parameter used in view job config[writer] template>, --writer=<parameter used in view job config[writer] template>
  47.                         View job config[writer] template, eg:
  48.                         mysqlwriter,streamwriter
  50.   Develop/Debug Options:
  51.     Developer use these options to trace more details of DataX.
  53.     -d, --debug         Set to remote debug mode.
  54.     --loglevel=<log level>
  55.                         Set log level such as: debug, info, all etc.
  56. [svccnetlhs@HOST datax]<231211 17:19:06>$
复制代码
 
DataX (HdfsReader) 插件
  1. [svccnetlhs@HOST datax]<231211 17:23:29>$ ls
  2. bin  conf  job  lib  log  log_perf  plugin  script  tmp
  3. [svccnetlhs@HOST datax]<231211 17:23:29>$
  4. [svccnetlhs@HOST datax]<231211 17:23:30>$ cd plugin/
  5. [svccnetlhs@HOST plugin]<231211 17:23:32>$ ls
  6. reader  writer
  7. [svccnetlhs@HOST plugin]<231211 17:23:32>$ cd reader/
  8. [svccnetlhs@HOST reader]<231211 17:23:36>$ ls
  9. cassandrareader   datahubreader  ftpreader  hbase094xreader  hbase11xsqlreader  hdfsreader        loghubreader   mysqlreader         odpsreader      oraclereader  otsreader        postgresqlreader  sqlserverreader  streamreader    tsdbreader
  10. clickhousereader  drdsreader     gdbreader  hbase11xreader   hbase20xsqlreader  kingbaseesreader  mongodbreader  oceanbasev10reader  opentsdbreader  ossreader     otsstreamreader  rdbmsreader       starrocksreader  tdenginereader  txtfilereader
  11. [svccnetlhs@HOST reader]<231211 17:23:37>$ cd hdfsreader/
  12. [svccnetlhs@HOST hdfsreader]<231211 17:23:39>$ ls
  13. hdfsreader-0.0.1-SNAPSHOT.jar  libs  plugin_job_template.json  plugin.json
  14. [svccnetlhs@HOST hdfsreader]<231211 17:23:40>$
  15. [svccnetlhs@HOST hdfsreader]<231211 17:23:42>$ pwd
  16. /home/svccnetlhs/chengken/starrocks/datax/plugin/reader/hdfsreader
  17. [svccnetlhs@HOST hdfsreader]<231211 17:23:43>$
  18. [svccnetlhs@HOST hdfsreader]<231211 17:23:44>$ cd libs/
  19. [svccnetlhs@HOST libs]<231211 17:23:54>$ ls
  20. activation-1.1.jar                     commons-beanutils-1.9.2.jar       curator-recipes-2.7.1.jar               hadoop-mapreduce-client-core-2.7.1.jar                  httpclient-4.1.2.jar           jetty-util-6.1.26.jar       parquet-hadoop-bundle-1.6.0rc3.jar
  21. aircompressor-0.3.jar                  commons-beanutils-core-1.8.0.jar  datanucleus-api-jdo-3.2.6.jar           hadoop-yarn-api-2.7.1.jar                               httpcore-4.1.2.jar             jline-2.12.jar              pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
  22. annotations-2.0.3.jar                  commons-cli-1.2.jar               datanucleus-core-3.2.10.jar             hadoop-yarn-common-2.7.1.jar                            jackson-core-asl-1.9.13.jar    jpam-1.1.jar                plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar
  23. ant-1.9.1.jar                          commons-codec-1.4.jar             datanucleus-rdbms-3.2.9.jar             hadoop-yarn-server-applicationhistoryservice-2.6.0.jar  jackson-jaxrs-1.9.13.jar       jsch-0.1.42.jar             protobuf-java-2.5.0.jar
  24. ant-launcher-1.9.1.jar                 commons-collections-3.2.1.jar     datax-common-0.0.1-SNAPSHOT.jar         hadoop-yarn-server-common-2.6.0.jar                     jackson-mapper-asl-1.9.13.jar  jsp-api-2.1.jar             servlet-api-2.5.jar
  25. antlr-2.7.7.jar                        commons-compiler-2.7.6.jar        derby-10.11.1.1.jar                     hadoop-yarn-server-resourcemanager-2.6.0.jar            jackson-xc-1.9.13.jar          jsr305-3.0.0.jar            slf4j-api-1.7.10.jar
  26. antlr-runtime-3.4.jar                  commons-compress-1.4.1.jar        eigenbase-properties-1.1.4.jar          hadoop-yarn-server-web-proxy-2.6.0.jar                  janino-2.7.6.jar               jta-1.1.jar                 slf4j-log4j12-1.7.10.jar
  27. aopalliance-1.0.jar                    commons-configuration-1.6.jar     fastjson2-2.0.23.jar                    hamcrest-core-1.3.jar                                   javacsv-2.0.jar                leveldbjni-all-1.8.jar      snappy-java-1.0.4.1.jar
  28. apache-curator-2.6.0.pom               commons-daemon-1.0.13.jar         geronimo-annotation_1.0_spec-1.1.1.jar  hive-ant-1.1.1.jar                                      javax.inject-1.jar             libfb303-0.9.2.jar          ST4-4.0.4.jar
  29. apacheds-i18n-2.0.0-M15.jar            commons-dbcp-1.4.jar              geronimo-jaspic_1.0_spec-1.0.jar        hive-cli-1.1.1.jar                                      java-xmlbuilder-0.4.jar        libthrift-0.9.2.jar         stax-api-1.0.1.jar
  30. apacheds-kerberos-codec-2.0.0-M15.jar  commons-digester-1.8.jar          geronimo-jta_1.1_spec-1.1.1.jar         hive-common-1.1.1.jar                                   jaxb-api-2.2.2.jar             log4j-1.2.17.jar            stax-api-1.0-2.jar
  31. apache-log4j-extras-1.2.17.jar         commons-httpclient-3.1.jar        groovy-all-2.1.6.jar                    hive-exec-1.1.1.jar                                     jaxb-impl-2.2.3-1.jar          log4j-api-2.17.1.jar        stringtemplate-3.2.1.jar
  32. api-asn1-api-1.0.0-M20.jar             commons-io-2.4.jar                gson-2.2.4.jar                          hive-hcatalog-core-1.1.1.jar                            jdo-api-3.0.1.jar              log4j-core-2.17.1.jar       velocity-1.5.jar
  33. api-util-1.0.0-M20.jar                 commons-lang-2.6.jar              guava-11.0.2.jar                        hive-metastore-1.1.1.jar                                jersey-client-1.9.jar          logback-classic-1.0.13.jar  xercesImpl-2.9.1.jar
  34. asm-3.1.jar                            commons-lang3-3.3.2.jar           guice-3.0.jar                           hive-serde-1.1.1.jar                                    jersey-core-1.9.jar            logback-core-1.0.13.jar     xml-apis-1.3.04.jar
  35. asm-commons-3.1.jar                    commons-logging-1.1.3.jar         guice-servlet-3.0.jar                   hive-service-1.1.1.jar                                  jersey-guice-1.9.jar           lzo-core-1.0.5.jar          xmlenc-0.52.jar
  36. asm-tree-3.1.jar                       commons-math3-3.1.1.jar           hadoop-aliyun-2.7.2.jar                 hive-shims-0.20S-1.1.1.jar                              jersey-json-1.9.jar            mail-1.4.1.jar              xz-1.0.jar
  37. avro-1.7.4.jar                         commons-net-3.1.jar               hadoop-annotations-2.7.1.jar            hive-shims-0.23-1.1.1.jar                               jersey-server-1.9.jar          netty-3.6.2.Final.jar       zookeeper-3.4.6.jar
  38. bonecp-0.8.0.RELEASE.jar               commons-pool-1.5.4.jar            hadoop-auth-2.7.1.jar                   hive-shims-1.1.1.jar                                    jets3t-0.9.0.jar               netty-all-4.0.23.Final.jar
  39. calcite-avatica-1.0.0-incubating.jar   cos_api-bundle-5.6.137.2.jar      hadoop-common-2.7.1.jar                 hive-shims-common-1.1.1.jar                             jettison-1.1.jar               opencsv-2.3.jar
  40. calcite-core-1.0.0-incubating.jar      curator-client-2.7.1.jar          hadoop-cos-3.1.0-8.3.2.jar              hive-shims-scheduler-1.1.1.jar                          jetty-6.1.26.jar               oro-2.0.8.jar
  41. calcite-linq4j-1.0.0-incubating.jar    curator-framework-2.6.0.jar       hadoop-hdfs-2.7.1.jar                   htrace-core-3.1.0-incubating.jar                        jetty-all-7.6.0.v20120127.jar  paranamer-2.3.jar
  42. [svccnetlhs@HOST libs]<231211 17:23:55>$
复制代码
 
DataX (StarRocksWriter) 插件
  1. [svccnetlhs@HOST datax]<231211 17:25:07>$ ls
  2. bin  conf  job  lib  log  log_perf  plugin  script  tmp
  3. [svccnetlhs@HOST datax]<231211 17:25:08>$ cd plugin/
  4. [svccnetlhs@HOST plugin]<231211 17:25:11>$ ls
  5. reader  writer
  6. [svccnetlhs@HOST plugin]<231211 17:25:11>$ cd writer/
  7. [svccnetlhs@HOST writer]<231211 17:25:13>$ ls
  8. adbpgwriter      clickhousewriter  doriswriter          ftpwriter        hbase11xsqlwriter  hdfswriter          kuduwriter     mysqlwriter         ocswriter     oscarwriter  postgresqlwriter  sqlserverwriter  tdenginewriter
  9. adswriter        databendwriter    drdswriter           gdbwriter        hbase11xwriter     hologresjdbcwriter  loghubwriter   neo4jwriter         odpswriter    osswriter    rdbmswriter       starrockswriter  tsdbwriter
  10. cassandrawriter  datahubwriter     elasticsearchwriter  hbase094xwriter  hbase20xsqlwriter  kingbaseeswriter    mongodbwriter  oceanbasev10writer  oraclewriter  otswriter    selectdbwriter    streamwriter     txtfilewriter
  11. [svccnetlhs@HOST writer]<231211 17:25:13>$ cd starrockswriter/
  12. [svccnetlhs@HOST starrockswriter]<231211 17:25:15>$ ls
  13. libs  plugin_job_template.json  plugin.json  starrockswriter-1.1.0.jar
  14. [svccnetlhs@HOST starrockswriter]<231211 17:25:16>$ ls libs/
  15. commons-codec-1.9.jar        commons-io-2.4.jar       commons-logging-1.1.1.jar  datax-common-0.0.1-SNAPSHOT.jar  fastjson2-2.0.23.jar  hamcrest-core-1.3.jar  httpcore-4.4.6.jar          logback-core-1.0.13.jar          plugin-rdbms-util-0.0.1-SNAPSHOT.jar
  16. commons-collections-3.0.jar  commons-lang3-3.3.2.jar  commons-math3-3.1.1.jar    druid-1.0.15.jar                 guava-r05.jar         httpclient-4.5.3.jar   logback-classic-1.0.13.jar  mysql-connector-java-5.1.46.jar  slf4j-api-1.7.10.jar
  17. [svccnetlhs@HOST starrockswriter]<231211 17:25:21>$
复制代码
注: 两个datax插件在文件开头可以进行下载。
 
DataX JSON

众所周知，DataX的是基于数据抽取、数据转换和数据加载三个步骤来实现数据流的搬迁。
Datax设计理念：

Datax框架设计：

Datax工作流程： 

 连接JSON:
  1. 模板1：<br>{
  2.     “content”: [
  3.         {
  4.             “reader”: {
  5.                 “name”: “hdfsreader”,
  6.                 “parameter”: {
  7.                     “column”: [
  8.                         {                                                             /************************************/
  9.                             “name”: “ts”,                                             /************************************/
  10.                             “type”: “string”,                                         /************************************/
  11.                             “value”: “2023-11-14”                                     /************************************/
  12.                         },                                                            /************************************/
  13.                         {                                                             /************************************/
  14.                             “index”: 0,                                               /************************************/
  15.                             “name”: “local_id”,                                       /************************************/
  16.                             “type”: “string”                                          /************************************/
  17.                         },                                                            /************************************/
  18.                         {                                                             /****1.由于cos文件中没有ts这个字段***/
  19.                             “index”: 1,                                               /****这里我则使用value指定一个固定值*/
  20.                             “name”: “encrypted_imei”,                                 /****value=2023-11-14代表当前path****/
  21.                             “type”: “string”                                          /****的分区数据, ********************/
  22.                         },                                                            /****此值在脚本中属于动态传参********/
  23.                         {                                                             /************************************/
  24.                             “index”: 2,                                               /****2.这里其他的字段使用了index*****/
  25.                             “name”: “encrypted_idfa”,                                 /****下标的形式取到每个字段的值******/
  26.                             “type”: “string”                                          /************************************/
  27.                         },                                                            /************************************/
  28.                         {                                                             /************************************/
  29.                             “index”: 3,                                               /************************************/
  30.                             “name”: “encrypted_mac”,                                  /************************************/
  31.                             “type”: “string”                                          /************************************/
  32.                         },                                                            /************************************/
  33.                         {                                                             /************************************/
  34.                             “index”: 4,                                               /************************************/
  35.                             “name”: “encrypted_android_id”,                           /************************************/
  36.                             “type”: “string”                                          /************************************/
  37.                         }                                                             /************************************/
  38.                     ],
  39.                     “defaultFS”: “cosn: //桶名/”,
  40.                     “encoding”: “UTF-8”,
  41.                     “fieldDelimiter”: ",",
  42.                     “fileType”: “orc”,
  43.                     “hadoopConfig”: {
  44.                         “fs.cosn.impl”: “org.apache.hadoop.fs.CosFileSystem”,
  45.                         “fs.cosn.tmp.dir”: “本地临时路径（随便）”,
  46.                         “fs.cosn.userinfo.region”: “ap-guangzhou”,
  47.                         “fs.cosn.userinfo.secretId”: "",
  48.                         “fs.cosn.userinfo.secretKey”: ""
  49.                     },
  50.                     “path”: "/sam/sam_dwd_user_action_cos_d/20231114/part-00011*"
  51.                 }
  52.             },
  53.             “writer”: {
  54.                 “name”: “starrockswriter”,
  55.                 “parameter”: {
  56.                     “column”: [
  57.                         “ts”,                                                        /******************************************/
  58.                         “local_id”,                                                  /******************************************/
  59.                         “encrypted_imei”,                                            /****StarRocks需要接收的字段名*************/
  60.                         “encrypted_idfa”,                                            /******************************************/
  61.                         “encrypted_mac”,                                             /******************************************/
  62.                         “encrypted_android_id”                                       /******************************************/
  63.                     ],
  64.                     “database”: “StarRocks库名”,
  65.                     “jdbcUrl”: “jdbc: mysql: //StarRocksFE_IP:9030/”,
  66.                     “loadProps”: {
  67.                         “max_filter_ratio”: 1
  68.                     },
  69.                     “loadUrl”: [
  70.                         “StarRocksFE_IP:8030”,
  71.                         “StarRocksFE_IP:8030”,
  72.                         “StarRocksFE_IP:8030”
  73.                     ],
  74.                     “password”: “StarRocks密码”,
  75.                     “postSql”: [
  76.                         
  77.                     ],
  78.                     “preSql”: [
  79.                         
  80.                     ],
  81.                     “table”: “StarRocks表名”,
  82.                     “username”: “StarRocks用户”
  83.                 }
  84.             }
  85.         }
  86.     ],
  87.     “setting”: {
  88.         “speed”: {
  89.             “byte”: -1,                                                            /********channel调整为3，不限速**********/
  90.             “channel”: 3                                                           /*********************************************/
  91.         }
  92.     }
  93. }
复制代码
  1. 模板2：<br>{
  2.     "job": {
  3.         "setting": {
  4.           "speed": {
  5.             "channel":3
  6.           },
  7.           "errorLimit": {}
  8.         },
  9.         "content": [{
  10.             "reader": {
  11.                 "name": "hdfsreader",
  12.                     "parameter": {
  13.                         "path": "/sam/sam_dwd_user_action_cos_d/20231114/part-*",
  14.                         "defaultFS": "cosn://*********/",
  15.                         "column": [
  16.                                 {"name":"ts","type":"string","value":"2023-11-14"},
  17.                                 {"name":"import_ds_","type":"string","index":0},
  18.                                 {"name":"unique_action_id","type":"string","index":1},
  19.                                 {"name":"action_time","type":"string","index":2},
  20.                                 {"name":"report_time","type":"string","index":3},
  21.                                 {"name":"action_type","type":"string","index":4},
  22.                                 {"name":"ka_id","type":"string","index":5},
  23.                                 {"name":"action_session_id","type":"string","index":6},
  24.                                 {"name":"uuid","type":"string","index":7},
  25.                                 {"name":"wx_app_id","type":"string","index":8},
  26.                                 {"name":"wx_open_id","type":"string","index":9},
  27.                                 {"name":"wx_union_id","type":"string","index":10},
  28.                                 {"name":"external_user_id","type":"string","index":11},
  29.                                 {"name":"merber_id","type":"string","index":12},
  30.                                 {"name":"local_id","type":"string","index":13},
  31.                                 {"name":"encrypted_imei","type":"string","index":14},
  32.                                 {"name":"encrypted_idfa","type":"string","index":15},
  33.                                 {"name":"encrypted_mac","type":"string","index":16},
  34.                                 {"name":"encrypted_android_id","type":"string","index":17},
  35.                                 {"name":"encrypted_qq","type":"string","index":18},
  36.                                 {"name":"encrypted_phone","type":"string","index":19},
  37.                                 {"name":"encrypting_algorithm","type":"string","index":20},
  38.                                 {"name":"chan_id","type":"string","index":21},
  39.                                 {"name":"chan_refer_app_id","type":"string","index":22},
  40.                                 {"name":"chan_shop_id","type":"string","index":23},
  41.                                 {"name":"chan_shop_name","type":"string","index":24},
  42.                                 {"name":"client_type","type":"string","index":25},
  43.                                 {"name":"client_name","type":"string","index":26},
  44.                                 {"name":"client_version","type":"string","index":27},
  45.                                 {"name":"sdk_version","type":"string","index":28},
  46.                                 {"name":"device_model","type":"string","index":29},
  47.                                 {"name":"ip","type":"string","index":30},
  48.                                 {"name":"user_agent","type":"string","index":31},
  49.                                 {"name":"page_path","type":"string","index":32},
  50.                                 {"name":"page_name","type":"string","index":33},
  51.                                 {"name":"referrer","type":"string","index":34},
  52.                                 {"name":"address","type":"string","index":35},
  53.                                 {"name":"city","type":"string","index":36},
  54.                                 {"name":"province","type":"string","index":37},
  55.                                 {"name":"country","type":"string","index":38},
  56.                                 {"name":"latitude","type":"string","index":39},
  57.                                 {"name":"longitude","type":"string","index":40},
  58.                                 {"name":"json_properties","type":"string","index":41},
  59.                                 {"name":"fdate","type":"string","index":42},
  60.                                 {"name":"tag_id","type":"string","index":43},
  61.                                 {"name":"tag_name","type":"string","index":44},
  62.                                 {"name":"chan_custom_id","type":"string","index":45},
  63.                                 {"name":"etl_load_time","type":"string","index":46},
  64.                                 {"name":"event_name","type":"string","index":47}
  65.                         ],
  66.                         "fileType": "orc",
  67.                         "encoding": "UTF-8",
  68.                         "hadoopConfig": {
  69.                             "fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem",
  70.                             "fs.cosn.userinfo.region": "ap-guangzhou",
  71.                             "fs.cosn.tmp.dir": "/u/chengken/starrocks/sam/data",
  72.                             "fs.cosn.userinfo.secretId": "***************",
  73.                             "fs.cosn.userinfo.secretKey": "**************",
  74.                             "fs.cosn.read.ahead.block.size": 1048576,
  75.                             "fs.cosn.read.ahead.queue.size": 2
  76.                         },
  77.                         "fieldDelimiter": ","
  78.                     }
  79.             },
  80.             "writer": {
  81.                  "name": "starrockswriter",
  82.                  "parameter": {
  83.                      "maxBatchRows":"5000000",
  84.                      "maxBatchSize":"5368709120",
  85.                      "username": "cndlopsns",
  86.                      "password": "lizhenghua1.",
  87.                      "database": "ods",
  88.                      "table": "buckets_tmp_20231205__sams_cos",
  89.                      "column": [
  90.                             "ts",
  91.                             "import_ds_",
  92.                             "unique_action_id",
  93.                             "action_time",
  94.                             "report_time",
  95.                             "action_type",
  96.                             "ka_id",
  97.                             "action_session_id",
  98.                             "uuid",
  99.                             "wx_app_id",
  100.                             "wx_open_id",
  101.                             "wx_union_id",
  102.                             "external_user_id",
  103.                             "merber_id",
  104.                             "local_id",
  105.                             "encrypted_imei",
  106.                             "encrypted_idfa",
  107.                             "encrypted_mac",
  108.                             "encrypted_android_id",
  109.                             "encrypted_qq",
  110.                             "encrypted_phone",
  111.                             "encrypting_algorithm",
  112.                             "chan_id",
  113.                             "chan_refer_app_id",
  114.                             "chan_shop_id",
  115.                             "chan_shop_name",
  116.                             "client_type",
  117.                             "client_name",
  118.                             "client_version",
  119.                             "sdk_version",
  120.                             "device_model",
  121.                             "ip",
  122.                             "user_agent",
  123.                             "page_path",
  124.                             "page_name",
  125.                             "referrer",
  126.                             "address",
  127.                             "city",
  128.                             "province",
  129.                             "country",
  130.                             "latitude",
  131.                             "longitude",
  132.                             "json_properties",
  133.                             "fdate",
  134.                             "tag_id",
  135.                             "tag_name",
  136.                             "chan_custom_id",
  137.                             "etl_load_time",
  138.                             "event_name"
  139.                   ],
  140.                      "preSql": [],
  141.                      "postSql": [],
  142.                      "jdbcUrl": "jdbc:mysql://***********:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8",
  143.                      "loadUrl": ["****1:8030","****2:8030","****3:8030"],
  144.                      "loadProps": {"max_filter_ratio":1}
  145.                  }
  146.              }
  147.         }]
  148.     }
  149. }
复制代码
 
启动

启动并顺利读到上游数据文件，然后异步写入StarRocks。
  1. /usr/bin/python2.7 /home/hadoop/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/hadoop/datax/cos-starrocks1.json
复制代码
  1. Run: /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms16g -Xmx16g" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json
  2. Output_____________
  4. DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
  5. Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
  8. 2023-12-11 17:49:54.918 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
  9. 2023-12-11 17:49:54.920 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
  10. 2023-12-11 17:49:54.945 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
  11. 2023-12-11 17:49:54.950 [main] INFO  Engine - the machine info  =>
  13.     osInfo:    Linux amd64 4.18.0-348.7.1.el8_5.x86_64
  14.     jvmInfo:    Oracle Corporation 1.8 25.112-b15
  15.     cpu num:    16
  17.     totalPhysicalMemory:    -0.00G
  18.     freePhysicalMemory:    -0.00G
  19.     maxFileDescriptorCount:    -1
  20.     currentOpenFileDescriptorCount:    -1
  22.     GC Names    [PS MarkSweep, PS Scavenge]
  24.     MEMORY_NAME                    | allocation_size                | init_size                     
  25.     PS Eden Space                  | 4,096.00MB                     | 4,096.00MB                     
  26.     Code Cache                     | 240.00MB                       | 2.44MB                        
  27.     Compressed Class Space         | 1,024.00MB                     | 0.00MB                        
  28.     PS Survivor Space              | 682.50MB                       | 682.50MB                       
  29.     PS Old Gen                     | 10,923.00MB                    | 10,923.00MB                    
  30.     Metaspace                      | -0.00MB                        | 0.00MB                        
  33. 2023-12-11 17:49:54.966 [main] INFO  Engine -
  34. {
  35.     "setting":{
  36.         "speed":{
  37.             "channel":3
  38.         },
  39.         "errorLimit":{
  40.             
  41.         }
  42.     },
  43.     "content":[
  44.         {
  45.             "reader":{
  46.                 "name":"hdfsreader",
  47.                 "parameter":{
  48.                     "path":"/sam/sam_dwd_user_action_cos_d/20231114/part-*",
  49.                     "defaultFS":"cosn://*********/",
  50.                     "column":[
  51.                         {
  52.                             "name":"ts",
  53.                             "type":"string",
  54.                             "value":"2023-11-14"
  55.                         },
  56.                         {
  57.                             "name":"import_ds_",
  58.                             "type":"string",
  59.                             "index":0
  60.                         },
  61.                         {
  62.                             "name":"unique_action_id",
  63.                             "type":"string",
  64.                             "index":1
  65.                         },
  66.                         {
  67.                             "name":"action_time",
  68.                             "type":"string",
  69.                             "index":2
  70.                         },
  71.                         {
  72.                             "name":"report_time",
  73.                             "type":"string",
  74.                             "index":3
  75.                         },
  76.                         {
  77.                             "name":"action_type",
  78.                             "type":"string",
  79.                             "index":4
  80.                         },
  81.                         {
  82.                             "name":"ka_id",
  83.                             "type":"string",
  84.                             "index":5
  85.                         },
  86.                         {
  87.                             "name":"action_session_id",
  88.                             "type":"string",
  89.                             "index":6
  90.                         },
  91.                         {
  92.                             "name":"uuid",
  93.                             "type":"string",
  94.                             "index":7
  95.                         },
  96.                         {
  97.                             "name":"wx_app_id",
  98.                             "type":"string",
  99.                             "index":8
  100.                         },
  101.                         {
  102.                             "name":"wx_open_id",
  103.                             "type":"string",
  104.                             "index":9
  105.                         },
  106.                         {
  107.                             "name":"wx_union_id",
  108.                             "type":"string",
  109.                             "index":10
  110.                         },
  111.                         {
  112.                             "name":"external_user_id",
  113.                             "type":"string",
  114.                             "index":11
  115.                         },
  116.                         {
  117.                             "name":"merber_id",
  118.                             "type":"string",
  119.                             "index":12
  120.                         },
  121.                         {
  122.                             "name":"local_id",
  123.                             "type":"string",
  124.                             "index":13
  125.                         },
  126.                         {
  127.                             "name":"encrypted_imei",
  128.                             "type":"string",
  129.                             "index":14
  130.                         },
  131.                         {
  132.                             "name":"encrypted_idfa",
  133.                             "type":"string",
  134.                             "index":15
  135.                         },
  136.                         {
  137.                             "name":"encrypted_mac",
  138.                             "type":"string",
  139.                             "index":16
  140.                         },
  141.                         {
  142.                             "name":"encrypted_android_id",
  143.                             "type":"string",
  144.                             "index":17
  145.                         },
  146.                         {
  147.                             "name":"encrypted_qq",
  148.                             "type":"string",
  149.                             "index":18
  150.                         },
  151.                         {
  152.                             "name":"encrypted_phone",
  153.                             "type":"string",
  154.                             "index":19
  155.                         },
  156.                         {
  157.                             "name":"encrypting_algorithm",
  158.                             "type":"string",
  159.                             "index":20
  160.                         },
  161.                         {
  162.                             "name":"chan_id",
  163.                             "type":"string",
  164.                             "index":21
  165.                         },
  166.                         {
  167.                             "name":"chan_refer_app_id",
  168.                             "type":"string",
  169.                             "index":22
  170.                         },
  171.                         {
  172.                             "name":"chan_shop_id",
  173.                             "type":"string",
  174.                             "index":23
  175.                         },
  176.                         {
  177.                             "name":"chan_shop_name",
  178.                             "type":"string",
  179.                             "index":24
  180.                         },
  181.                         {
  182.                             "name":"client_type",
  183.                             "type":"string",
  184.                             "index":25
  185.                         },
  186.                         {
  187.                             "name":"client_name",
  188.                             "type":"string",
  189.                             "index":26
  190.                         },
  191.                         {
  192.                             "name":"client_version",
  193.                             "type":"string",
  194.                             "index":27
  195.                         },
  196.                         {
  197.                             "name":"sdk_version",
  198.                             "type":"string",
  199.                             "index":28
  200.                         },
  201.                         {
  202.                             "name":"device_model",
  203.                             "type":"string",
  204.                             "index":29
  205.                         },
  206.                         {
  207.                             "name":"ip",
  208.                             "type":"string",
  209.                             "index":30
  210.                         },
  211.                         {
  212.                             "name":"user_agent",
  213.                             "type":"string",
  214.                             "index":31
  215.                         },
  216.                         {
  217.                             "name":"page_path",
  218.                             "type":"string",
  219.                             "index":32
  220.                         },
  221.                         {
  222.                             "name":"page_name",
  223.                             "type":"string",
  224.                             "index":33
  225.                         },
  226.                         {
  227.                             "name":"referrer",
  228.                             "type":"string",
  229.                             "index":34
  230.                         },
  231.                         {
  232.                             "name":"address",
  233.                             "type":"string",
  234.                             "index":35
  235.                         },
  236.                         {
  237.                             "name":"city",
  238.                             "type":"string",
  239.                             "index":36
  240.                         },
  241.                         {
  242.                             "name":"province",
  243.                             "type":"string",
  244.                             "index":37
  245.                         },
  246.                         {
  247.                             "name":"country",
  248.                             "type":"string",
  249.                             "index":38
  250.                         },
  251.                         {
  252.                             "name":"latitude",
  253.                             "type":"string",
  254.                             "index":39
  255.                         },
  256.                         {
  257.                             "name":"longitude",
  258.                             "type":"string",
  259.                             "index":40
  260.                         },
  261.                         {
  262.                             "name":"json_properties",
  263.                             "type":"string",
  264.                             "index":41
  265.                         },
  266.                         {
  267.                             "name":"fdate",
  268.                             "type":"string",
  269.                             "index":42
  270.                         },
  271.                         {
  272.                             "name":"tag_id",
  273.                             "type":"string",
  274.                             "index":43
  275.                         },
  276.                         {
  277.                             "name":"tag_name",
  278.                             "type":"string",
  279.                             "index":44
  280.                         },
  281.                         {
  282.                             "name":"chan_custom_id",
  283.                             "type":"string",
  284.                             "index":45
  285.                         },
  286.                         {
  287.                             "name":"etl_load_time",
  288.                             "type":"string",
  289.                             "index":46
  290.                         },
  291.                         {
  292.                             "name":"event_name",
  293.                             "type":"string",
  294.                             "index":47
  295.                         }
  296.                     ],
  297.                     "fileType":"orc",
  298.                     "encoding":"UTF-8",
  299.                     "hadoopConfig":{
  300.                         "fs.cosn.impl":"org.apache.hadoop.fs.CosFileSystem",
  301.                         "fs.cosn.userinfo.region":"ap-guangzhou",
  302.                         "fs.cosn.tmp.dir":"/u/chengken/starrocks/sam/data",
  303.                         "fs.cosn.userinfo.secretId":"**********",
  304.                         "fs.cosn.userinfo.secretKey":"****************",
  305.                         "fs.cosn.read.ahead.block.size":1048576,
  306.                         "fs.cosn.read.ahead.queue.size":2
  307.                     },
  308.                     "fieldDelimiter":","
  309.                 }
  310.             },
  311.             "writer":{
  312.                 "name":"starrockswriter",
  313.                 "parameter":{
  314.                     "maxBatchRows":"5000000",
  315.                     "maxBatchSize":"5368709120",
  316.                     "username":"cndlopsns",
  317.                     "password":"************",
  318.                     "database":"ods",
  319.                     "table":"buckets_tmp_20231205__sams_cos",
  320.                     "column":[
  321.                         "ts",
  322.                         "import_ds_",
  323.                         "unique_action_id",
  324.                         "action_time",
  325.                         "report_time",
  326.                         "action_type",
  327.                         "ka_id",
  328.                         "action_session_id",
  329.                         "uuid",
  330.                         "wx_app_id",
  331.                         "wx_open_id",
  332.                         "wx_union_id",
  333.                         "external_user_id",
  334.                         "merber_id",
  335.                         "local_id",
  336.                         "encrypted_imei",
  337.                         "encrypted_idfa",
  338.                         "encrypted_mac",
  339.                         "encrypted_android_id",
  340.                         "encrypted_qq",
  341.                         "encrypted_phone",
  342.                         "encrypting_algorithm",
  343.                         "chan_id",
  344.                         "chan_refer_app_id",
  345.                         "chan_shop_id",
  346.                         "chan_shop_name",
  347.                         "client_type",
  348.                         "client_name",
  349.                         "client_version",
  350.                         "sdk_version",
  351.                         "device_model",
  352.                         "ip",
  353.                         "user_agent",
  354.                         "page_path",
  355.                         "page_name",
  356.                         "referrer",
  357.                         "address",
  358.                         "city",
  359.                         "province",
  360.                         "country",
  361.                         "latitude",
  362.                         "longitude",
  363.                         "json_properties",
  364.                         "fdate",
  365.                         "tag_id",
  366.                         "tag_name",
  367.                         "chan_custom_id",
  368.                         "etl_load_time",
  369.                         "event_name"
  370.                     ],
  371.                     "preSql":[
  372.                         
  373.                     ],
  374.                     "postSql":[
  375.                         
  376.                     ],
  377.                     "jdbcUrl":"jdbc:mysql://192.168.1.121:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8",
  378.                     "loadUrl":[
  379.                         "192.168.1.121:8030",
  380.                         "192.168.1.122:8030",
  381.                         "192.168.1.123:8030"
  382.                     ],
  383.                     "loadProps":{
  384.                         "max_filter_ratio":1
  385.                     }
  386.                 }
  387.             }
  388.         }
  389.     ]
  390. }
  392. 2023-12-11 17:49:54.983 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=true
  393. 2023-12-11 17:49:54.983 [main] INFO  JobContainer - DataX jobContainer starts job.
  394. 2023-12-11 17:49:54.984 [main] INFO  JobContainer - Set jobId = 0
  395. 2023-12-11 17:49:54.994 [job-0] INFO  HdfsReader$Job - init() begin...
  396. 2023-12-11 17:49:55.250 [job-0] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":[]}
  397. 2023-12-11 17:49:55.250 [job-0] INFO  HdfsReader$Job - init() ok and end...
  398. 2023-12-11 17:49:55.279 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
  399. 2023-12-11 17:49:55.279 [job-0] INFO  JobContainer - DataX Reader.Job [hdfsreader] do prepare work .
  400. 2023-12-11 17:49:55.279 [job-0] INFO  HdfsReader$Job - prepare(), start to getAllFiles...
  401. 2023-12-11 17:49:55.279 [job-0] INFO  HdfsReader$Job - get HDFS all files in path = [/sam/sam_dwd_user_action_cos_d/20231114/part-*]
  402. 十二月 11, 2023 5:49:55 下午 org.apache.hadoop.util.NativeCodeLoader <clinit>
  403. 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  404. 2023-12-11 17:49:55.527 [job-0] INFO  RangerCredentialsClient - begin to init ranger client, impl []
  405. 2023-12-11 17:49:55.776 [job-0] INFO  CosNativeFileSystemStore - hadoop cos retry times: 200, cos client retry times: 5
  406. log4j:WARN No appenders could be found for logger (com.qcloud.cos.thirdparty.org.apache.http.client.protocol.RequestAddCookies).
  407. log4j:WARN Please initialize the log4j system properly.
  408. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  409. 2023-12-11 17:49:56.400 [job-0] INFO  CosFileSystem - The cos bucket is the normal bucket.
  410. 2023-12-11 17:49:56.418 [job-0] INFO  BufferPool - Initialize the buffer pool.
  411. 2023-12-11 17:49:56.419 [job-0] INFO  BufferPool - fs.cosn.upload.buffer.size is set to -1, so the 'mapped_disk' buffer will be used by default.
  412. 2023-12-11 17:49:56.419 [job-0] INFO  BufferPool - The type of the upload buffer pool is [MAPPED_DISK]. Buffer size:[-1]
  413. 2023-12-11 17:49:56.419 [job-0] INFO  BufferPool - tmp dir list
  414. 2023-12-11 17:49:56.796 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  415. 2023-12-11 17:49:56.953 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  416. 2023-12-11 17:49:57.014 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  417. 2023-12-11 17:49:57.192 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  418. 2023-12-11 17:49:57.244 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  419. 2023-12-11 17:49:57.392 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  420. 2023-12-11 17:49:57.443 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  421. 2023-12-11 17:49:57.586 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  422. 2023-12-11 17:49:57.645 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  423. 2023-12-11 17:49:57.782 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  424. 2023-12-11 17:49:57.837 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  425. 2023-12-11 17:49:58.007 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  426. 2023-12-11 17:49:58.066 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  427. 2023-12-11 17:49:58.233 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  428. 2023-12-11 17:49:58.292 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  429. 2023-12-11 17:49:58.444 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  430. 2023-12-11 17:49:58.510 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  431. 2023-12-11 17:49:58.694 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  432. 2023-12-11 17:49:58.749 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  433. 2023-12-11 17:49:58.975 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  434. 2023-12-11 17:49:59.030 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  435. 2023-12-11 17:49:59.181 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  436. 2023-12-11 17:49:59.232 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  437. 2023-12-11 17:49:59.393 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  438. 2023-12-11 17:49:59.449 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  439. 2023-12-11 17:49:59.602 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  440. 2023-12-11 17:49:59.656 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  441. 2023-12-11 17:49:59.836 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  442. 2023-12-11 17:49:59.888 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  443. 2023-12-11 17:50:00.038 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  444. 2023-12-11 17:50:00.106 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  445. 2023-12-11 17:50:00.325 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  446. 2023-12-11 17:50:00.382 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  447. 2023-12-11 17:50:00.551 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  448. 2023-12-11 17:50:00.606 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  449. 2023-12-11 17:50:00.769 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  450. 2023-12-11 17:50:00.826 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  451. 2023-12-11 17:50:00.969 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  452. 2023-12-11 17:50:01.021 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  453. 2023-12-11 17:50:01.165 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
  454. 2023-12-11 17:50:01.165 [job-0] INFO  HdfsReader$Job - 您即将读取的文件数为: [20], 列表为: [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
  455. 2023-12-11 17:50:01.166 [job-0] INFO  JobContainer - DataX Writer.Job [starrockswriter] do prepare work .
  456. 2023-12-11 17:50:01.168 [job-0] INFO  JobContainer - jobContainer starts to do split ...
  457. 2023-12-11 17:50:01.168 [job-0] INFO  JobContainer - Job set Channel-Number to 3 channels.
  458. 2023-12-11 17:50:01.168 [job-0] INFO  HdfsReader$Job - split() begin...
  459. 2023-12-11 17:50:01.173 [job-0] INFO  JobContainer - DataX Reader.Job [hdfsreader] splits to [20] tasks.
  460. 2023-12-11 17:50:01.174 [job-0] INFO  JobContainer - DataX Writer.Job [starrockswriter] splits to [20] tasks.
  461. 2023-12-11 17:50:01.200 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
  462. 2023-12-11 17:50:01.213 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
  463. 2023-12-11 17:50:01.215 [job-0] INFO  JobContainer - Running by standalone Mode.
  464. 2023-12-11 17:50:01.227 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [3] channels for [20] tasks.
  465. 2023-12-11 17:50:01.231 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to 209715200.
  466. 2023-12-11 17:50:01.232 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
  467. 2023-12-11 17:50:01.240 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[17] attemptCount[1] is started
  468. 2023-12-11 17:50:01.243 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
  469. 2023-12-11 17:50:01.244 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started
  470. 2023-12-11 17:50:01.288 [0-0-17-writer] INFO  HostUtils - IP 10.233.76.104 HOSTNAME pose-app-52211-pdc
  471. 2023-12-11 17:50:01.322 [0-0-17-reader] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
  472. 2023-12-11 17:50:01.324 [0-0-17-reader] INFO  Reader$Task - read start
  473. 2023-12-11 17:50:01.324 [0-0-17-reader] INFO  Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
  474. 2023-12-11 17:50:01.324 [0-0-17-reader] INFO  HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc].
  475. 2023-12-11 17:50:01.326 [0-0-0-reader] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
  476. 2023-12-11 17:50:01.326 [0-0-8-reader] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
  477. 2023-12-11 17:50:01.327 [0-0-0-reader] INFO  Reader$Task - read start
  478. 2023-12-11 17:50:01.327 [0-0-0-reader] INFO  Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
  479. 2023-12-11 17:50:01.328 [0-0-0-reader] INFO  HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc].
  480. 2023-12-11 17:50:01.328 [0-0-8-reader] INFO  Reader$Task - read start
  481. 2023-12-11 17:50:01.328 [0-0-8-reader] INFO  Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
  482. 2023-12-11 17:50:01.328 [0-0-8-reader] INFO  HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc].
  483. 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
  484. 信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
  485. 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
  486. 信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
  487. 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
  488. 信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
  489. 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
  490. 信息: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  491. 2023-12-11 17:50:01.646 [ORC_GET_SPLITS #1] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  492. 2023-12-11 17:50:01.776 [ORC_GET_SPLITS #1] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  493. 2023-12-11 17:50:01.778 [ORC_GET_SPLITS #1] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  494. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
  495. 信息: FooterCacheHitRatio: 0/1
  496. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
  497. 信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202185 duration=760 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
  498. 2023-12-11 17:50:02.288 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  499. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
  500. 信息: FooterCacheHitRatio: 0/1
  501. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
  502. 信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202296 duration=871 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
  503. 2023-12-11 17:50:02.369 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  504. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
  505. 信息: FooterCacheHitRatio: 0/1
  506. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
  507. 信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202433 duration=1008 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
  508. 2023-12-11 17:50:02.492 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  509. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  510. 信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
  511. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  512. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
  513. 2023-12-11 17:50:02.647 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  514. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  515. 信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
  516. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  517. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
  518. 2023-12-11 17:50:02.784 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  519. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  520. 信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
  521. 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  522. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
  523. 2023-12-11 17:50:02.867 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  524. 2023-12-11 17:50:11.235 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
  525. 2023-12-11 17:50:21.236 [job-0] INFO  StandAloneJobContainerCommunicator - Total 297312 records, 558177719 bytes | Speed 53.23MB/s, 29731 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.023s |  All Task WaitReaderTime 25.333s | Percentage 0.00%
  526. 2023-12-11 17:50:30.385 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  527. 十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  528. 信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039}
  529. 十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  530. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156410867, length: 9223372036854775807}
  531. 2023-12-11 17:50:30.731 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  532. 2023-12-11 17:50:30.907 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  533. 2023-12-11 17:50:31.114 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  534. 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  535. 信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039}
  536. 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  537. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156011050, length: 9223372036854775807}
  538. 2023-12-11 17:50:31.237 [job-0] INFO  StandAloneJobContainerCommunicator - Total 598944 records, 1125524739 bytes | Speed 54.11MB/s, 30163 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.040s |  All Task WaitReaderTime 43.878s | Percentage 0.00%
  539. 2023-12-11 17:50:31.280 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  540. 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  541. 信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 604159}
  542. 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  543. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156411995, length: 9223372036854775807}
  544. 2023-12-11 17:50:31.457 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  545. 2023-12-11 17:50:41.238 [job-0] INFO  StandAloneJobContainerCommunicator - Total 906144 records, 1702990600 bytes | Speed 55.07MB/s, 30720 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.056s |  All Task WaitReaderTime 62.147s | Percentage 0.00%
  546. 2023-12-11 17:50:51.241 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1203104 records, 2261515086 bytes | Speed 53.27MB/s, 29696 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.070s |  All Task WaitReaderTime 92.372s | Percentage 0.00%
  547. 2023-12-11 17:50:56.098 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  548. 十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  549. 信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839}
  550. 十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  551. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 311140141, length: 9223372036854775807}
  552. 2023-12-11 17:50:56.469 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  553. 2023-12-11 17:50:57.564 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  554. 十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  555. 信息: min key = {originalTxn: 0, bucket: -1, row: 604159}, max key = {originalTxn: 0, bucket: -1, row: 803839}
  556. 十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  557. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 312563739, length: 9223372036854775807}
  558. 2023-12-11 17:50:57.971 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  559. 2023-12-11 17:50:58.447 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  560. 十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  561. 信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839}
  562. 十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  563. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 310368886, length: 9223372036854775807}
  564. 2023-12-11 17:50:58.782 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  565. 2023-12-11 17:51:01.242 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1703232 records, 3203376993 bytes | Speed 89.82MB/s, 50012 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.093s |  All Task WaitReaderTime 127.642s | Percentage 0.00%
  566. 2023-12-11 17:51:11.242 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1829984 records, 3440997771 bytes | Speed 22.66MB/s, 12675 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.101s |  All Task WaitReaderTime 138.648s | Percentage 0.00%
  567. 2023-12-11 17:51:15.910 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  568. 十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  569. 信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799}
  570. 十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  571. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 417009783, length: 9223372036854775807}
  572. 2023-12-11 17:51:16.305 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  573. 2023-12-11 17:51:17.688 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  574. 十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  575. 信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799}
  576. 十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  577. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415835058, length: 9223372036854775807}
  578. 2023-12-11 17:51:18.017 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  579. 2023-12-11 17:51:18.695 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  580. 十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  581. 信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1105919}
  582. 十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  583. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415904892, length: 9223372036854775807}
  584. 2023-12-11 17:51:19.066 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  585. 2023-12-11 17:51:21.243 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2342208 records, 4403928517 bytes | Speed 91.83MB/s, 51222 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.126s |  All Task WaitReaderTime 178.482s | Percentage 0.00%
  586. 2023-12-11 17:51:31.245 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2466496 records, 4638515258 bytes | Speed 22.37MB/s, 12428 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.132s |  All Task WaitReaderTime 190.077s | Percentage 0.00%
  587. 2023-12-11 17:51:41.246 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2967264 records, 5579213090 bytes | Speed 89.71MB/s, 50076 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.156s |  All Task WaitReaderTime 229.517s | Percentage 0.00%
  588. 2023-12-11 17:51:41.359 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  589. 十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  590. 信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479}
  591. 十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  592. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 571347891, length: 9223372036854775807}
  593. 2023-12-11 17:51:41.747 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  594. 2023-12-11 17:51:41.998 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  595. 十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  596. 信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479}
  597. 十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  598. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 570444727, length: 9223372036854775807}
  599. 2023-12-11 17:51:42.390 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  600. 2023-12-11 17:51:44.650 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  601. 十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  602. 信息: min key = {originalTxn: 0, bucket: -1, row: 1105919}, max key = {originalTxn: 0, bucket: -1, row: 1305599}
  603. 十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  604. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 572342605, length: 9223372036854775807}
  605. 2023-12-11 17:51:44.978 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  606. 2023-12-11 17:51:51.248 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3307424 records, 6219717845 bytes | Speed 61.08MB/s, 34016 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.172s |  All Task WaitReaderTime 246.926s | Percentage 0.00%
  607. 2023-12-11 17:52:01.250 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3614624 records, 6797953036 bytes | Speed 55.14MB/s, 30720 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.187s |  All Task WaitReaderTime 279.975s | Percentage 0.00%
  608. 2023-12-11 17:52:02.446 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  609. 十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  610. 信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439}
  611. 十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  612. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 674573070, length: 9223372036854775807}
  613. 2023-12-11 17:52:02.859 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  614. 2023-12-11 17:52:03.369 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  615. 十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  616. 信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439}
  617. 十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  618. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 673856234, length: 9223372036854775807}
  619. 2023-12-11 17:52:03.751 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  620. 2023-12-11 17:52:04.567 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  621. 十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
  622. 信息: min key = {originalTxn: 0, bucket: -1, row: 1305599}, max key = {originalTxn: 0, bucket: -1, row: 1602559}
  623. 十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
  624. 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 675641495, length: 9223372036854775807}
  625. 2023-12-11 17:52:04.881 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
  626. 2023-12-11 17:52:11.251 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.202s |  All Task WaitReaderTime 297.463s | Percentage 0.00%
  627. 2023-12-11 17:52:21.252 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.215s |  All Task WaitReaderTime 331.832s | Percentage 0.00%
  628. ...
复制代码
StarRocks数据顺利加载：
  1. [Mon Dec 11 09:56:57 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] > show partitions from ods.buckets_tmp_20231205__sams_cos;
  2. +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
  3. | PartitionId | PartitionName | VisibleVersion | VisibleVersionTime  | VisibleVersionHash | State  | PartitionKey | Range                                                                      | DistributionKey | Buckets | ReplicationNum | StorageMedium | CooldownTime        | LastConsistencyCheckTime | DataSize | IsInMemory |
  4. +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
  5. | 513180658   | p20231114     | 1              | 2023-12-11 09:49:41 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-11-14]; ..types: [DATE]; keys: [2023-11-15]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | 16.3GB   | false      |
  6. | 504724038   | p20231115     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-11-15]; ..types: [DATE]; keys: [2023-11-16]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  7. | 504722860   | p20231116     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-11-16]; ..types: [DATE]; keys: [2023-11-17]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  8. | 504721682   | p20231206     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-06]; ..types: [DATE]; keys: [2023-12-07]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  9. | 504722271   | p20231207     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-07]; ..types: [DATE]; keys: [2023-12-08]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  10. | 504720504   | p20231208     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-08]; ..types: [DATE]; keys: [2023-12-09]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  11. | 504721093   | p20231209     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-09]; ..types: [DATE]; keys: [2023-12-10]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  12. | 505174943   | p20231210     | 1              | 2023-12-07 00:14:47 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-10]; ..types: [DATE]; keys: [2023-12-11]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  13. | 506968779   | p20231211     | 1              | 2023-12-08 00:04:57 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-11]; ..types: [DATE]; keys: [2023-12-12]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  14. | 508936021   | p20231212     | 1              | 2023-12-09 00:11:09 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-12]; ..types: [DATE]; keys: [2023-12-13]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  15. | 510079030   | p20231213     | 1              | 2023-12-10 00:05:38 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-13]; ..types: [DATE]; keys: [2023-12-14]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  16. | 510795860   | p20231214     | 1              | 2023-12-11 00:07:15 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-14]; ..types: [DATE]; keys: [2023-12-15]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
  17. +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
  18. 12 rows in set (0.002 sec)
  20. [Mon Dec 11 09:56:57 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] >
复制代码
查询正常：
  1. [Mon Dec 11 09:58:20 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] > select * from ods.buckets_tmp_20231205__sams_cos where ts='2023-11-14' limit 2;
  2. +------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
  3. | ts         | uuid             | import_ds_ | unique_action_id                     | action_time   | report_time   | action_type          | ka_id    | action_session_id                                 | wx_app_id | wx_open_id | wx_union_id | external_user_id | merber_id         | local_id | encrypted_imei                       | encrypted_idfa | encrypted_mac | encrypted_android_id | encrypted_qq | encrypted_phone | encrypting_algorithm | chan_id | chan_refer_app_id | chan_shop_id             | chan_shop_name                                                                                                     | client_type  | client_name | client_version | sdk_version | device_model | ip   | user_agent | page_path    | page_name | referrer | address | city | province | country | latitude | longitude | json_properties                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | fdate    | chan_custom_id | etl_load_time | tag_id    | tag_name     | event_name |
  4. +------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
  5. | 2023-11-14 | 8325 | 2023111412 | ed55491f-da41-4976-8601-49c4405d994f | 1699936160765 | 1699936132584 | element              | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL      | NULL       | NULL        | 274012627        | 10742100529761108 | NULL     | 9090 | NULL           | NULL          | NULL                 | NULL         | NULL            | NULL                 | NULL    | NULL              | 9991,6758,4834,6119,9996 | 苏州木渎DC                                     | sams-app-sdk | NULL        | NULL           | NULL        | NULL         | NULL | NULL       | HomeFragment | 首页      | NULL     | NULL    | NULL | NULL     | NULL    | NULL     | NULL      | *****                                                                                                                                                 | 20231114 | default        | NULL          | advantage | 普通会员     | NULL       |
  6. | 2023-11-14 | 8614 | 2023111412 | f30f3672-1bad-478c-ab19-16b736b850ef | 1699936161032 | 1699936133093 | expose_sku_component | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL      | NULL       | NULL        | 274012627        | 10742100529761108 | NULL     | 9090 | NULL           | NULL          | NULL                 | NULL         | NULL            | NULL                 | NULL    | NULL              | 9991,6758,4834,6119,9996 | 苏州木渎DC                                     | sams-app-sdk | NULL        | NULL           | NULL        | NULL         | NULL | NULL       | HomeFragment | 首页      | NULL     | NULL    | NULL | NULL     | NULL    | NULL     | NULL      | *****                           | 20231114 | default        | NULL          | advantage | 普通会员     | NULL       |
  7. +------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
  8. 2 rows in set (1.660 sec)
复制代码
速率不断太慢，Datax也就这样了：
  1. 2023-12-11 17:52:11.251 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.202s |  All Task WaitReaderTime 297.463s | Percentage 0.00%
  2. 2023-12-11 17:52:21.252 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.215s |  All Task WaitReaderTime 331.832s | Percentage 0.00%
复制代码
 
备注

之前有伙伴问到，为什么我的json里面字段用到“index”
  1. "column": [
  2.     {
  3.         "name": "ts",
  4.         "type": "string",
  5.         "value": "2023-11-14"
  6.     },
  7.     {
  8.         "name": "import_ds_",
  9.         "type": "string",
  10.         "index": 0
  11.     },
  12.     {
  13.         "name": "unique_action_id",
  14.         "type": "string",
  15.         "index": 1
  16.     },
  17.     {
  18.         "name": "action_time",
  19.         "type": "string",
  20.         "index": 2
  21.     },<br>    ....
  22.     ]
复制代码
这个地方经过尝试，原有的column1，column2，column3...的方式测试行不通， 因为我有个ts的字段需要造值。但如果使用
  1.     {
  2.         "name": "ts",
  3.         "type": "string",
  4.         "value": "2023-11-14"
  5.     },
  6.     {
  7.         "name": "import_ds_",
  8.         "type": "string",
  9.     },<br>    ....
复制代码
这种方式， 则抛出：由于您配置了type, 则至少需要配置 index 或 value，这是一件令人头疼的事。
  1. [svccnetlhs@HOST log]<231211 18:10:01>$ /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json
  3. DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
  4. Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
  7. 2023-12-11 18:10:30.213 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
  8. 2023-12-11 18:10:30.215 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
  9. 2023-12-11 18:10:30.226 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
  10. 2023-12-11 18:10:30.231 [main] INFO  Engine - the machine info  =>
  12.     osInfo:    Linux amd64 4.18.0-348.7.1.el8_5.x86_64
  13.     jvmInfo:    Oracle Corporation 1.8 25.112-b15
  14.     cpu num:    16
  16.     totalPhysicalMemory:    -0.00G
  17.     freePhysicalMemory:    -0.00G
  18.     maxFileDescriptorCount:    -1
  19.     currentOpenFileDescriptorCount:    -1
  21.     GC Names    [PS MarkSweep, PS Scavenge]
  23.     MEMORY_NAME                    | allocation_size                | init_size                     
  24.     PS Eden Space                  | 1,280.00MB                     | 256.00MB                       
  25.     Code Cache                     | 240.00MB                       | 2.44MB                        
  26.     Compressed Class Space         | 1,024.00MB                     | 0.00MB                        
  27.     PS Survivor Space              | 42.50MB                        | 42.50MB                        
  28.     PS Old Gen                     | 2,731.00MB                     | 683.00MB                       
  29.     Metaspace                      | -0.00MB                        | 0.00MB                        
  33. 2023-12-11 18:10:30.258 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=true
  34. 2023-12-11 18:10:30.258 [main] INFO  JobContainer - DataX jobContainer starts job.
  35. 2023-12-11 18:10:30.259 [main] INFO  JobContainer - Set jobId = 0
  36. 2023-12-11 18:10:30.269 [job-0] INFO  HdfsReader$Job - init() begin...
  37. 2023-12-11 18:10:30.274 [job-0] ERROR JobContainer - Exception when job run
  38. com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-06], Description:[没有 Index].  - 由于您配置了type, 则至少需要配置 index 或 value
  39.     at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
  40.     at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
  41.     at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
  42.     at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
  43.     at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
  44.     at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
  45.     at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
  46.     at com.alibaba.datax.core.Engine.start(Engine.java:86) [datax-core-0.0.1-SNAPSHOT.jar:na]
  47.     at com.alibaba.datax.core.Engine.entry(Engine.java:168) [datax-core-0.0.1-SNAPSHOT.jar:na]
  48.     at com.alibaba.datax.core.Engine.main(Engine.java:201) [datax-core-0.0.1-SNAPSHOT.jar:na]
  49. 2023-12-11 18:10:30.277 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
  50. 2023-12-11 18:10:30.279 [job-0] ERROR Engine -
  52. 经DataX智能分析,该任务最可能的错误原因是:
  53. com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-06], Description:[没有 Index].  - 由于您配置了type, 则至少需要配置 index 或 value
  54.     at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30)
  55.     at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150)
  56.     at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111)
  57.     at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50)
  58.     at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673)
  59.     at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303)
  60.     at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
  61.     at com.alibaba.datax.core.Engine.start(Engine.java:86)
  62.     at com.alibaba.datax.core.Engine.entry(Engine.java:168)
  63.     at com.alibaba.datax.core.Engine.main(Engine.java:201)
复制代码
那么我是如何取得每个字段精准的index？
这里我用到了 orc-tools-1.8.0-uber.jar 这个包把orc里面的字段先解析出来，
下载：https://repo1.maven.org/maven2/org/apache/orc/orc-tools/
  1. java -jar orc-tools-1.8.0-uber.jar meta <ORC文件>
复制代码
成功解析orc文件的元数据字段信息，Type: struct：代表的就是字段列与下标顺序。
  1. log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
  2. log4j:WARN Please initialize the log4j system properly.
  3. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  4. Processing data file part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000 [length: 290734159]
  5. Structure for part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000
  6. File Version: 0.12 with ORC_14 by ORC Java 1.6.14
  7. Rows: 849920
  8. Compression: SNAPPY
  9. Compression size: 131072
  10. Calendar: Julian/Gregorian
  11. Type: struct<import_ds_:int,unique_action_id:string,action_time:bigint,report_time:bigint,action_type:string,ka_id:bigint,action_session_id:string,uuid:string,wx_app_id:string,wx_open_id:string,wx_union_id:string,external_user_id:string,merber_id:string,local_id:string,encrypted_imei:string,encrypted_idfa:string,encrypted_mac:string,encrypted_android_id:string,encrypted_qq:string,encrypted_phone:string,encrypting_algorithm:string,chan_id:string,chan_refer_app_id:string,chan_shop_id:string,chan_shop_name:string,client_type:string,client_name:string,client_version:string,sdk_version:string,device_model:string,ip:string,user_agent:string,page_path:string,page_name:string,referrer:string,address:string,city:string,province:string,country:string,latitude:string,longitude:string,json_properties:string,fdate:string,tag_id:string,tag_name:string,chan_custom_id:string,etl_load_time:string>
  13. Stripe Statistics:
复制代码
 
 
完。
 

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！

本帖子中包含更多资源

您需要 登录 才可以下载或查看，没有账号？立即注册

x
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

飞不高

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

挺好的 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表