论坛 › 数据仓库与分析 › DataX vs 腾讯云COS对象存储 -> StarRocks集群

飞不高 发表于 2024-2-13 15:03:37

DataX vs 腾讯云COS对象存储 -> StarRocks集群

本文将介绍使用DataX读出Cos的Orc文件往StarRocks里面写。
 
需求: 需要将腾讯云cos上84TB的数据, 同步到StarRocks某个大表。正常每个分区数据量20~30亿，600GB。
工具：DataX
插件：hdfsreader、starrockswriter
对象存储COS：非融合

[*]hdfsreader：https://cloud.tencent.com/document/product/436/43654
[*]starrockswriter：https://docs.mirrorship.cn/zh/docs/loading/DataX-starrocks-writer
 
DataX

这里我使用的datax版本是 DataX (DATAX-OPENSOURCE-3.0)
<231211 17:17:11>$ tree bin/ conf/
bin/
├── datax.py
├── dxprof.py
└── perftrace.py
conf/
├── core.json
└── logback.xml

0 directories, 5 files
<231211 17:18:52>$ /bin/python3
python3   python3.6   python3.6m
<231211 17:18:52>$ /bin/python3 bin/datax.py

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


Usage: datax.py job-url-or-path

Options:
-h, --help            show this help message and exit

Product Env Options:
    Normal user use these options to set jvm parameters, job runtime mode
    etc. Make sure these options can be used in Product Env.

    -j <jvm parameters>, --jvm=<jvm parameters>
                        Set jvm parameters if necessary.
    --jobid=<job unique id>
                        Set job unique id when running by Distribute/Local
                        Mode.
    -m <job runtime mode>, --mode=<job runtime mode>
                        Set job runtime mode such as: standalone, local,
                        distribute. Default mode is standalone.
    -p <parameter used in job config>, --params=<parameter used in job config>
                        Set job parameter, eg: the source tableName you want
                        to set it by command, then you can use like this:
                        -p"-DtableName=your-table-name", if you have mutiple
                        parameters: -p"-DtableName=your-table-name
                        -DcolumnName=your-column-name".Note: you should config
                        in you job tableName with ${tableName}.
    -r <parameter used in view job config template>, --reader=<parameter used in view job config template>
                        View job config template, eg:
                        mysqlreader,streamreader
    -w <parameter used in view job config template>, --writer=<parameter used in view job config template>
                        View job config template, eg:
                        mysqlwriter,streamwriter

Develop/Debug Options:
    Developer use these options to trace more details of DataX.

    -d, --debug         Set to remote debug mode.
    --loglevel=<log level>
                        Set log level such as: debug, info, all etc.
<231211 17:19:06>$  
DataX (HdfsReader) 插件

<231211 17:23:29>$ ls
binconfjoblibloglog_perfpluginscripttmp
<231211 17:23:29>$
<231211 17:23:30>$ cd plugin/
<231211 17:23:32>$ ls
readerwriter
<231211 17:23:32>$ cd reader/
<231211 17:23:36>$ ls
cassandrareader   datahubreaderftpreaderhbase094xreaderhbase11xsqlreaderhdfsreader      loghubreader   mysqlreader         odpsreader      oraclereaderotsreader      postgresqlreadersqlserverreaderstreamreader    tsdbreader
clickhousereaderdrdsreader   gdbreaderhbase11xreader   hbase20xsqlreaderkingbaseesreadermongodbreaderoceanbasev10readeropentsdbreaderossreader   otsstreamreaderrdbmsreader       starrocksreadertdenginereadertxtfilereader
<231211 17:23:37>$ cd hdfsreader/
<231211 17:23:39>$ ls
hdfsreader-0.0.1-SNAPSHOT.jarlibsplugin_job_template.jsonplugin.json
<231211 17:23:40>$
<231211 17:23:42>$ pwd
/home/svccnetlhs/chengken/starrocks/datax/plugin/reader/hdfsreader
<231211 17:23:43>$
<231211 17:23:44>$ cd libs/
<231211 17:23:54>$ ls
activation-1.1.jar                     commons-beanutils-1.9.2.jar       curator-recipes-2.7.1.jar               hadoop-mapreduce-client-core-2.7.1.jar                  httpclient-4.1.2.jar         jetty-util-6.1.26.jar       parquet-hadoop-bundle-1.6.0rc3.jar
aircompressor-0.3.jar                  commons-beanutils-core-1.8.0.jardatanucleus-api-jdo-3.2.6.jar         hadoop-yarn-api-2.7.1.jar                               httpcore-4.1.2.jar             jline-2.12.jar            pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
annotations-2.0.3.jar                  commons-cli-1.2.jar               datanucleus-core-3.2.10.jar             hadoop-yarn-common-2.7.1.jar                            jackson-core-asl-1.9.13.jar    jpam-1.1.jar                plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar
ant-1.9.1.jar                        commons-codec-1.4.jar             datanucleus-rdbms-3.2.9.jar             hadoop-yarn-server-applicationhistoryservice-2.6.0.jarjackson-jaxrs-1.9.13.jar       jsch-0.1.42.jar             protobuf-java-2.5.0.jar
ant-launcher-1.9.1.jar               commons-collections-3.2.1.jar   datax-common-0.0.1-SNAPSHOT.jar         hadoop-yarn-server-common-2.6.0.jar                     jackson-mapper-asl-1.9.13.jarjsp-api-2.1.jar             servlet-api-2.5.jar
antlr-2.7.7.jar                        commons-compiler-2.7.6.jar      derby-10.11.1.1.jar                     hadoop-yarn-server-resourcemanager-2.6.0.jar            jackson-xc-1.9.13.jar          jsr305-3.0.0.jar            slf4j-api-1.7.10.jar
antlr-runtime-3.4.jar                  commons-compress-1.4.1.jar      eigenbase-properties-1.1.4.jar          hadoop-yarn-server-web-proxy-2.6.0.jar                  janino-2.7.6.jar               jta-1.1.jar               slf4j-log4j12-1.7.10.jar
aopalliance-1.0.jar                  commons-configuration-1.6.jar   fastjson2-2.0.23.jar                  hamcrest-core-1.3.jar                                 javacsv-2.0.jar                leveldbjni-all-1.8.jar      snappy-java-1.0.4.1.jar
apache-curator-2.6.0.pom               commons-daemon-1.0.13.jar         geronimo-annotation_1.0_spec-1.1.1.jarhive-ant-1.1.1.jar                                    javax.inject-1.jar             libfb303-0.9.2.jar          ST4-4.0.4.jar
apacheds-i18n-2.0.0-M15.jar            commons-dbcp-1.4.jar            geronimo-jaspic_1.0_spec-1.0.jar      hive-cli-1.1.1.jar                                    java-xmlbuilder-0.4.jar      libthrift-0.9.2.jar         stax-api-1.0.1.jar
apacheds-kerberos-codec-2.0.0-M15.jarcommons-digester-1.8.jar          geronimo-jta_1.1_spec-1.1.1.jar         hive-common-1.1.1.jar                                 jaxb-api-2.2.2.jar             log4j-1.2.17.jar            stax-api-1.0-2.jar
apache-log4j-extras-1.2.17.jar         commons-httpclient-3.1.jar      groovy-all-2.1.6.jar                  hive-exec-1.1.1.jar                                     jaxb-impl-2.2.3-1.jar          log4j-api-2.17.1.jar      stringtemplate-3.2.1.jar
api-asn1-api-1.0.0-M20.jar             commons-io-2.4.jar                gson-2.2.4.jar                        hive-hcatalog-core-1.1.1.jar                            jdo-api-3.0.1.jar            log4j-core-2.17.1.jar       velocity-1.5.jar
api-util-1.0.0-M20.jar               commons-lang-2.6.jar            guava-11.0.2.jar                        hive-metastore-1.1.1.jar                              jersey-client-1.9.jar          logback-classic-1.0.13.jarxercesImpl-2.9.1.jar
asm-3.1.jar                            commons-lang3-3.3.2.jar         guice-3.0.jar                           hive-serde-1.1.1.jar                                    jersey-core-1.9.jar            logback-core-1.0.13.jar   xml-apis-1.3.04.jar
asm-commons-3.1.jar                  commons-logging-1.1.3.jar         guice-servlet-3.0.jar                   hive-service-1.1.1.jar                                  jersey-guice-1.9.jar         lzo-core-1.0.5.jar          xmlenc-0.52.jar
asm-tree-3.1.jar                     commons-math3-3.1.1.jar         hadoop-aliyun-2.7.2.jar               hive-shims-0.20S-1.1.1.jar                              jersey-json-1.9.jar            mail-1.4.1.jar            xz-1.0.jar
avro-1.7.4.jar                         commons-net-3.1.jar               hadoop-annotations-2.7.1.jar            hive-shims-0.23-1.1.1.jar                               jersey-server-1.9.jar          netty-3.6.2.Final.jar       zookeeper-3.4.6.jar
bonecp-0.8.0.RELEASE.jar               commons-pool-1.5.4.jar            hadoop-auth-2.7.1.jar                   hive-shims-1.1.1.jar                                    jets3t-0.9.0.jar               netty-all-4.0.23.Final.jar
calcite-avatica-1.0.0-incubating.jar   cos_api-bundle-5.6.137.2.jar      hadoop-common-2.7.1.jar               hive-shims-common-1.1.1.jar                           jettison-1.1.jar               opencsv-2.3.jar
calcite-core-1.0.0-incubating.jar      curator-client-2.7.1.jar          hadoop-cos-3.1.0-8.3.2.jar            hive-shims-scheduler-1.1.1.jar                        jetty-6.1.26.jar               oro-2.0.8.jar
calcite-linq4j-1.0.0-incubating.jar    curator-framework-2.6.0.jar       hadoop-hdfs-2.7.1.jar                   htrace-core-3.1.0-incubating.jar                        jetty-all-7.6.0.v20120127.jarparanamer-2.3.jar
<231211 17:23:55>$  
DataX (StarRocksWriter) 插件

<231211 17:25:07>$ ls
binconfjoblibloglog_perfpluginscripttmp
<231211 17:25:08>$ cd plugin/
<231211 17:25:11>$ ls
readerwriter
<231211 17:25:11>$ cd writer/
<231211 17:25:13>$ ls
adbpgwriter      clickhousewriterdoriswriter          ftpwriter      hbase11xsqlwriterhdfswriter          kuduwriter   mysqlwriter         ocswriter   oscarwriterpostgresqlwritersqlserverwritertdenginewriter
adswriter      databendwriter    drdswriter         gdbwriter      hbase11xwriter   hologresjdbcwriterloghubwriter   neo4jwriter         odpswriter    osswriter    rdbmswriter       starrockswritertsdbwriter
cassandrawriterdatahubwriter   elasticsearchwriterhbase094xwriterhbase20xsqlwriterkingbaseeswriter    mongodbwriteroceanbasev10writeroraclewriterotswriter    selectdbwriter    streamwriter   txtfilewriter
<231211 17:25:13>$ cd starrockswriter/
<231211 17:25:15>$ ls
libsplugin_job_template.jsonplugin.jsonstarrockswriter-1.1.0.jar
<231211 17:25:16>$ ls libs/
commons-codec-1.9.jar      commons-io-2.4.jar       commons-logging-1.1.1.jardatax-common-0.0.1-SNAPSHOT.jarfastjson2-2.0.23.jarhamcrest-core-1.3.jarhttpcore-4.4.6.jar          logback-core-1.0.13.jar          plugin-rdbms-util-0.0.1-SNAPSHOT.jar
commons-collections-3.0.jarcommons-lang3-3.3.2.jarcommons-math3-3.1.1.jar    druid-1.0.15.jar               guava-r05.jar         httpclient-4.5.3.jar   logback-classic-1.0.13.jarmysql-connector-java-5.1.46.jarslf4j-api-1.7.10.jar
<231211 17:25:21>$注: 两个datax插件在文件开头可以进行下载。
 
DataX JSON

众所周知，DataX的是基于数据抽取、数据转换和数据加载三个步骤来实现数据流的搬迁。
Datax设计理念：
https://img2023.cnblogs.com/blog/1198387/202312/1198387-20231211172908398-1231155883.png
Datax框架设计：
https://img2023.cnblogs.com/blog/1198387/202312/1198387-20231211173023584-1891142961.png
Datax工作流程： 
https://img2023.cnblogs.com/blog/1198387/202312/1198387-20231211173036233-58115793.png
 连接JSON:
模板1：<br>{
    “content”: [
      {
            “reader”: {
                “name”: “hdfsreader”,
                “parameter”: {
                  “column”: [
                        {                                                             /************************************/
                            “name”: “ts”,                                             /************************************/
                            “type”: “string”,                                       /************************************/
                            “value”: “2023-11-14”                                     /************************************/
                        },                                                            /************************************/
                        {                                                             /************************************/
                            “index”: 0,                                             /************************************/
                            “name”: “local_id”,                                       /************************************/
                            “type”: “string”                                          /************************************/
                        },                                                            /************************************/
                        {                                                             /****1.由于cos文件中没有ts这个字段***/
                            “index”: 1,                                             /****这里我则使用value指定一个固定值*/
                            “name”: “encrypted_imei”,                                 /****value=2023-11-14代表当前path****/
                            “type”: “string”                                          /****的分区数据, ********************/
                        },                                                            /****此值在脚本中属于动态传参********/
                        {                                                             /************************************/
                            “index”: 2,                                             /****2.这里其他的字段使用了index*****/
                            “name”: “encrypted_idfa”,                                 /****下标的形式取到每个字段的值******/
                            “type”: “string”                                          /************************************/
                        },                                                            /************************************/
                        {                                                             /************************************/
                            “index”: 3,                                             /************************************/
                            “name”: “encrypted_mac”,                                  /************************************/
                            “type”: “string”                                          /************************************/
                        },                                                            /************************************/
                        {                                                             /************************************/
                            “index”: 4,                                             /************************************/
                            “name”: “encrypted_android_id”,                           /************************************/
                            “type”: “string”                                          /************************************/
                        }                                                             /************************************/
                  ],
                  “defaultFS”: “cosn: //桶名/”,
                  “encoding”: “UTF-8”,
                  “fieldDelimiter”: ",",
                  “fileType”: “orc”,
                  “hadoopConfig”: {
                        “fs.cosn.impl”: “org.apache.hadoop.fs.CosFileSystem”,
                        “fs.cosn.tmp.dir”: “本地临时路径（随便）”,
                        “fs.cosn.userinfo.region”: “ap-guangzhou”,
                        “fs.cosn.userinfo.secretId”: "",
                        “fs.cosn.userinfo.secretKey”: ""
                  },
                  “path”: "/sam/sam_dwd_user_action_cos_d/20231114/part-00011*"
                }
            },
            “writer”: {
                “name”: “starrockswriter”,
                “parameter”: {
                  “column”: [
                        “ts”,                                                      /******************************************/
                        “local_id”,                                                /******************************************/
                        “encrypted_imei”,                                          /****StarRocks需要接收的字段名*************/
                        “encrypted_idfa”,                                          /******************************************/
                        “encrypted_mac”,                                             /******************************************/
                        “encrypted_android_id”                                       /******************************************/
                  ],
                  “database”: “StarRocks库名”,
                  “jdbcUrl”: “jdbc: mysql: //StarRocksFE_IP:9030/”,
                  “loadProps”: {
                        “max_filter_ratio”: 1
                  },
                  “loadUrl”: [
                        “StarRocksFE_IP:8030”,
                        “StarRocksFE_IP:8030”,
                        “StarRocksFE_IP:8030”
                  ],
                  “password”: “StarRocks密码”,
                  “postSql”: [
                        
                  ],
                  “preSql”: [
                        
                  ],
                  “table”: “StarRocks表名”,
                  “username”: “StarRocks用户”
                }
            }
      }
    ],
    “setting”: {
      “speed”: {
            “byte”: -1,                                                            /********channel调整为3，不限速**********/
            “channel”: 3                                                         /*********************************************/
      }
    }
}模板2：<br>{
    "job": {
      "setting": {
          "speed": {
            "channel":3
          },
          "errorLimit": {}
      },
      "content": [{
            "reader": {
                "name": "hdfsreader",
                  "parameter": {
                        "path": "/sam/sam_dwd_user_action_cos_d/20231114/part-*",
                        "defaultFS": "cosn://*********/",
                        "column": [
                              {"name":"ts","type":"string","value":"2023-11-14"},
                              {"name":"import_ds_","type":"string","index":0},
                              {"name":"unique_action_id","type":"string","index":1},
                              {"name":"action_time","type":"string","index":2},
                              {"name":"report_time","type":"string","index":3},
                              {"name":"action_type","type":"string","index":4},
                              {"name":"ka_id","type":"string","index":5},
                              {"name":"action_session_id","type":"string","index":6},
                              {"name":"uuid","type":"string","index":7},
                              {"name":"wx_app_id","type":"string","index":8},
                              {"name":"wx_open_id","type":"string","index":9},
                              {"name":"wx_union_id","type":"string","index":10},
                              {"name":"external_user_id","type":"string","index":11},
                              {"name":"merber_id","type":"string","index":12},
                              {"name":"local_id","type":"string","index":13},
                              {"name":"encrypted_imei","type":"string","index":14},
                              {"name":"encrypted_idfa","type":"string","index":15},
                              {"name":"encrypted_mac","type":"string","index":16},
                              {"name":"encrypted_android_id","type":"string","index":17},
                              {"name":"encrypted_qq","type":"string","index":18},
                              {"name":"encrypted_phone","type":"string","index":19},
                              {"name":"encrypting_algorithm","type":"string","index":20},
                              {"name":"chan_id","type":"string","index":21},
                              {"name":"chan_refer_app_id","type":"string","index":22},
                              {"name":"chan_shop_id","type":"string","index":23},
                              {"name":"chan_shop_name","type":"string","index":24},
                              {"name":"client_type","type":"string","index":25},
                              {"name":"client_name","type":"string","index":26},
                              {"name":"client_version","type":"string","index":27},
                              {"name":"sdk_version","type":"string","index":28},
                              {"name":"device_model","type":"string","index":29},
                              {"name":"ip","type":"string","index":30},
                              {"name":"user_agent","type":"string","index":31},
                              {"name":"page_path","type":"string","index":32},
                              {"name":"page_name","type":"string","index":33},
                              {"name":"referrer","type":"string","index":34},
                              {"name":"address","type":"string","index":35},
                              {"name":"city","type":"string","index":36},
                              {"name":"province","type":"string","index":37},
                              {"name":"country","type":"string","index":38},
                              {"name":"latitude","type":"string","index":39},
                              {"name":"longitude","type":"string","index":40},
                              {"name":"json_properties","type":"string","index":41},
                              {"name":"fdate","type":"string","index":42},
                              {"name":"tag_id","type":"string","index":43},
                              {"name":"tag_name","type":"string","index":44},
                              {"name":"chan_custom_id","type":"string","index":45},
                              {"name":"etl_load_time","type":"string","index":46},
                              {"name":"event_name","type":"string","index":47}
                        ],
                        "fileType": "orc",
                        "encoding": "UTF-8",
                        "hadoopConfig": {
                            "fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem",
                            "fs.cosn.userinfo.region": "ap-guangzhou",
                            "fs.cosn.tmp.dir": "/u/chengken/starrocks/sam/data",
                            "fs.cosn.userinfo.secretId": "***************",
                            "fs.cosn.userinfo.secretKey": "**************",
                            "fs.cosn.read.ahead.block.size": 1048576,
                            "fs.cosn.read.ahead.queue.size": 2
                        },
                        "fieldDelimiter": ","
                  }
            },
            "writer": {
               "name": "starrockswriter",
               "parameter": {
                     "maxBatchRows":"5000000",
                     "maxBatchSize":"5368709120",
                     "username": "cndlopsns",
                     "password": "lizhenghua1.",
                     "database": "ods",
                     "table": "buckets_tmp_20231205__sams_cos",
                     "column": [
                            "ts",
                            "import_ds_",
                            "unique_action_id",
                            "action_time",
                            "report_time",
                            "action_type",
                            "ka_id",
                            "action_session_id",
                            "uuid",
                            "wx_app_id",
                            "wx_open_id",
                            "wx_union_id",
                            "external_user_id",
                            "merber_id",
                            "local_id",
                            "encrypted_imei",
                            "encrypted_idfa",
                            "encrypted_mac",
                            "encrypted_android_id",
                            "encrypted_qq",
                            "encrypted_phone",
                            "encrypting_algorithm",
                            "chan_id",
                            "chan_refer_app_id",
                            "chan_shop_id",
                            "chan_shop_name",
                            "client_type",
                            "client_name",
                            "client_version",
                            "sdk_version",
                            "device_model",
                            "ip",
                            "user_agent",
                            "page_path",
                            "page_name",
                            "referrer",
                            "address",
                            "city",
                            "province",
                            "country",
                            "latitude",
                            "longitude",
                            "json_properties",
                            "fdate",
                            "tag_id",
                            "tag_name",
                            "chan_custom_id",
                            "etl_load_time",
                            "event_name"
                  ],
                     "preSql": [],
                     "postSql": [],
                     "jdbcUrl": "jdbc:mysql://***********:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8",
                     "loadUrl": ["****1:8030","****2:8030","****3:8030"],
                     "loadProps": {"max_filter_ratio":1}
               }
             }
      }]
    }
} 
启动

启动并顺利读到上游数据文件，然后异步写入StarRocks。
/usr/bin/python2.7 /home/hadoop/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/hadoop/datax/cos-starrocks1.jsonRun: /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms16g -Xmx16g" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json
Output_____________

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2023-12-11 17:49:54.918 INFOMessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-12-11 17:49:54.920 INFOMessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo
2023-12-11 17:49:54.945 INFOVMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-12-11 17:49:54.950 INFOEngine - the machine info=>

    osInfo:    Linux amd64 4.18.0-348.7.1.el8_5.x86_64
    jvmInfo:    Oracle Corporation 1.8 25.112-b15
    cpu num:    16

    totalPhysicalMemory:    -0.00G
    freePhysicalMemory:    -0.00G
    maxFileDescriptorCount:    -1
    currentOpenFileDescriptorCount:    -1

    GC Names   

    MEMORY_NAME                  | allocation_size                | init_size                     
    PS Eden Space                  | 4,096.00MB                     | 4,096.00MB                     
    Code Cache                     | 240.00MB                     | 2.44MB                        
    Compressed Class Space         | 1,024.00MB                     | 0.00MB                        
    PS Survivor Space            | 682.50MB                     | 682.50MB                     
    PS Old Gen                     | 10,923.00MB                  | 10,923.00MB                  
    Metaspace                      | -0.00MB                        | 0.00MB                        


2023-12-11 17:49:54.966 INFOEngine -
{
    "setting":{
      "speed":{
            "channel":3
      },
      "errorLimit":{
            
      }
    },
    "content":[
      {
            "reader":{
                "name":"hdfsreader",
                "parameter":{
                  "path":"/sam/sam_dwd_user_action_cos_d/20231114/part-*",
                  "defaultFS":"cosn://*********/",
                  "column":[
                        {
                            "name":"ts",
                            "type":"string",
                            "value":"2023-11-14"
                        },
                        {
                            "name":"import_ds_",
                            "type":"string",
                            "index":0
                        },
                        {
                            "name":"unique_action_id",
                            "type":"string",
                            "index":1
                        },
                        {
                            "name":"action_time",
                            "type":"string",
                            "index":2
                        },
                        {
                            "name":"report_time",
                            "type":"string",
                            "index":3
                        },
                        {
                            "name":"action_type",
                            "type":"string",
                            "index":4
                        },
                        {
                            "name":"ka_id",
                            "type":"string",
                            "index":5
                        },
                        {
                            "name":"action_session_id",
                            "type":"string",
                            "index":6
                        },
                        {
                            "name":"uuid",
                            "type":"string",
                            "index":7
                        },
                        {
                            "name":"wx_app_id",
                            "type":"string",
                            "index":8
                        },
                        {
                            "name":"wx_open_id",
                            "type":"string",
                            "index":9
                        },
                        {
                            "name":"wx_union_id",
                            "type":"string",
                            "index":10
                        },
                        {
                            "name":"external_user_id",
                            "type":"string",
                            "index":11
                        },
                        {
                            "name":"merber_id",
                            "type":"string",
                            "index":12
                        },
                        {
                            "name":"local_id",
                            "type":"string",
                            "index":13
                        },
                        {
                            "name":"encrypted_imei",
                            "type":"string",
                            "index":14
                        },
                        {
                            "name":"encrypted_idfa",
                            "type":"string",
                            "index":15
                        },
                        {
                            "name":"encrypted_mac",
                            "type":"string",
                            "index":16
                        },
                        {
                            "name":"encrypted_android_id",
                            "type":"string",
                            "index":17
                        },
                        {
                            "name":"encrypted_qq",
                            "type":"string",
                            "index":18
                        },
                        {
                            "name":"encrypted_phone",
                            "type":"string",
                            "index":19
                        },
                        {
                            "name":"encrypting_algorithm",
                            "type":"string",
                            "index":20
                        },
                        {
                            "name":"chan_id",
                            "type":"string",
                            "index":21
                        },
                        {
                            "name":"chan_refer_app_id",
                            "type":"string",
                            "index":22
                        },
                        {
                            "name":"chan_shop_id",
                            "type":"string",
                            "index":23
                        },
                        {
                            "name":"chan_shop_name",
                            "type":"string",
                            "index":24
                        },
                        {
                            "name":"client_type",
                            "type":"string",
                            "index":25
                        },
                        {
                            "name":"client_name",
                            "type":"string",
                            "index":26
                        },
                        {
                            "name":"client_version",
                            "type":"string",
                            "index":27
                        },
                        {
                            "name":"sdk_version",
                            "type":"string",
                            "index":28
                        },
                        {
                            "name":"device_model",
                            "type":"string",
                            "index":29
                        },
                        {
                            "name":"ip",
                            "type":"string",
                            "index":30
                        },
                        {
                            "name":"user_agent",
                            "type":"string",
                            "index":31
                        },
                        {
                            "name":"page_path",
                            "type":"string",
                            "index":32
                        },
                        {
                            "name":"page_name",
                            "type":"string",
                            "index":33
                        },
                        {
                            "name":"referrer",
                            "type":"string",
                            "index":34
                        },
                        {
                            "name":"address",
                            "type":"string",
                            "index":35
                        },
                        {
                            "name":"city",
                            "type":"string",
                            "index":36
                        },
                        {
                            "name":"province",
                            "type":"string",
                            "index":37
                        },
                        {
                            "name":"country",
                            "type":"string",
                            "index":38
                        },
                        {
                            "name":"latitude",
                            "type":"string",
                            "index":39
                        },
                        {
                            "name":"longitude",
                            "type":"string",
                            "index":40
                        },
                        {
                            "name":"json_properties",
                            "type":"string",
                            "index":41
                        },
                        {
                            "name":"fdate",
                            "type":"string",
                            "index":42
                        },
                        {
                            "name":"tag_id",
                            "type":"string",
                            "index":43
                        },
                        {
                            "name":"tag_name",
                            "type":"string",
                            "index":44
                        },
                        {
                            "name":"chan_custom_id",
                            "type":"string",
                            "index":45
                        },
                        {
                            "name":"etl_load_time",
                            "type":"string",
                            "index":46
                        },
                        {
                            "name":"event_name",
                            "type":"string",
                            "index":47
                        }
                  ],
                  "fileType":"orc",
                  "encoding":"UTF-8",
                  "hadoopConfig":{
                        "fs.cosn.impl":"org.apache.hadoop.fs.CosFileSystem",
                        "fs.cosn.userinfo.region":"ap-guangzhou",
                        "fs.cosn.tmp.dir":"/u/chengken/starrocks/sam/data",
                        "fs.cosn.userinfo.secretId":"**********",
                        "fs.cosn.userinfo.secretKey":"****************",
                        "fs.cosn.read.ahead.block.size":1048576,
                        "fs.cosn.read.ahead.queue.size":2
                  },
                  "fieldDelimiter":","
                }
            },
            "writer":{
                "name":"starrockswriter",
                "parameter":{
                  "maxBatchRows":"5000000",
                  "maxBatchSize":"5368709120",
                  "username":"cndlopsns",
                  "password":"************",
                  "database":"ods",
                  "table":"buckets_tmp_20231205__sams_cos",
                  "column":[
                        "ts",
                        "import_ds_",
                        "unique_action_id",
                        "action_time",
                        "report_time",
                        "action_type",
                        "ka_id",
                        "action_session_id",
                        "uuid",
                        "wx_app_id",
                        "wx_open_id",
                        "wx_union_id",
                        "external_user_id",
                        "merber_id",
                        "local_id",
                        "encrypted_imei",
                        "encrypted_idfa",
                        "encrypted_mac",
                        "encrypted_android_id",
                        "encrypted_qq",
                        "encrypted_phone",
                        "encrypting_algorithm",
                        "chan_id",
                        "chan_refer_app_id",
                        "chan_shop_id",
                        "chan_shop_name",
                        "client_type",
                        "client_name",
                        "client_version",
                        "sdk_version",
                        "device_model",
                        "ip",
                        "user_agent",
                        "page_path",
                        "page_name",
                        "referrer",
                        "address",
                        "city",
                        "province",
                        "country",
                        "latitude",
                        "longitude",
                        "json_properties",
                        "fdate",
                        "tag_id",
                        "tag_name",
                        "chan_custom_id",
                        "etl_load_time",
                        "event_name"
                  ],
                  "preSql":[
                        
                  ],
                  "postSql":[
                        
                  ],
                  "jdbcUrl":"jdbc:mysql://192.168.1.121:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8",
                  "loadUrl":[
                        "192.168.1.121:8030",
                        "192.168.1.122:8030",
                        "192.168.1.123:8030"
                  ],
                  "loadProps":{
                        "max_filter_ratio":1
                  }
                }
            }
      }
    ]
}

2023-12-11 17:49:54.983 INFOPerfTrace - PerfTrace traceId=job_-1, isEnable=true
2023-12-11 17:49:54.983 INFOJobContainer - DataX jobContainer starts job.
2023-12-11 17:49:54.984 INFOJobContainer - Set jobId = 0
2023-12-11 17:49:54.994 INFOHdfsReader$Job - init() begin...
2023-12-11 17:49:55.250 INFOHdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":[]}
2023-12-11 17:49:55.250 INFOHdfsReader$Job - init() ok and end...
2023-12-11 17:49:55.279 INFOJobContainer - jobContainer starts to do prepare ...
2023-12-11 17:49:55.279 INFOJobContainer - DataX Reader.Job do prepare work .
2023-12-11 17:49:55.279 INFOHdfsReader$Job - prepare(), start to getAllFiles...
2023-12-11 17:49:55.279 INFOHdfsReader$Job - get HDFS all files in path =
十二月 11, 2023 5:49:55 下午 org.apache.hadoop.util.NativeCodeLoader <clinit>
警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-12-11 17:49:55.527 INFORangerCredentialsClient - begin to init ranger client, impl []
2023-12-11 17:49:55.776 INFOCosNativeFileSystemStore - hadoop cos retry times: 200, cos client retry times: 5
log4j:WARN No appenders could be found for logger (com.qcloud.cos.thirdparty.org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2023-12-11 17:49:56.400 INFOCosFileSystem - The cos bucket is the normal bucket.
2023-12-11 17:49:56.418 INFOBufferPool - Initialize the buffer pool.
2023-12-11 17:49:56.419 INFOBufferPool - fs.cosn.upload.buffer.size is set to -1, so the 'mapped_disk' buffer will be used by default.
2023-12-11 17:49:56.419 INFOBufferPool - The type of the upload buffer pool is . Buffer size:[-1]
2023-12-11 17:49:56.419 INFOBufferPool - tmp dir list
2023-12-11 17:49:56.796 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:56.953 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.014 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.192 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.244 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.392 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.443 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.586 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.645 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.782 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.837 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.007 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.066 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.233 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.292 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.444 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.510 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.694 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.749 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.975 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.030 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.181 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.232 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.393 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.449 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.602 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.656 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.836 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.888 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.038 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.106 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.325 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.382 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.551 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.606 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.769 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.826 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.969 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:50:01.021 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:01.165 INFOHdfsReader$Job - 是类型的文件, 将该文件加入source files列表
2023-12-11 17:50:01.165 INFOHdfsReader$Job - 您即将读取的文件数为: , 列表为:
2023-12-11 17:50:01.166 INFOJobContainer - DataX Writer.Job do prepare work .
2023-12-11 17:50:01.168 INFOJobContainer - jobContainer starts to do split ...
2023-12-11 17:50:01.168 INFOJobContainer - Job set Channel-Number to 3 channels.
2023-12-11 17:50:01.168 INFOHdfsReader$Job - split() begin...
2023-12-11 17:50:01.173 INFOJobContainer - DataX Reader.Job splits to tasks.
2023-12-11 17:50:01.174 INFOJobContainer - DataX Writer.Job splits to tasks.
2023-12-11 17:50:01.200 INFOJobContainer - jobContainer starts to do schedule ...
2023-12-11 17:50:01.213 INFOJobContainer - Scheduler starts taskGroups.
2023-12-11 17:50:01.215 INFOJobContainer - Running by standalone Mode.
2023-12-11 17:50:01.227 INFOTaskGroupContainer - taskGroupId= start channels for tasks.
2023-12-11 17:50:01.231 INFOChannel - Channel set byte_speed_limit to 209715200.
2023-12-11 17:50:01.232 INFOChannel - Channel set record_speed_limit to -1, No tps activated.
2023-12-11 17:50:01.240 INFOTaskGroupContainer - taskGroup taskId attemptCount is started
2023-12-11 17:50:01.243 INFOTaskGroupContainer - taskGroup taskId attemptCount is started
2023-12-11 17:50:01.244 INFOTaskGroupContainer - taskGroup taskId attemptCount is started
2023-12-11 17:50:01.288 INFOHostUtils - IP 10.233.76.104 HOSTNAME pose-app-52211-pdc
2023-12-11 17:50:01.322 INFOHdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2023-12-11 17:50:01.324 INFOReader$Task - read start
2023-12-11 17:50:01.324 INFOReader$Task - reading file :
2023-12-11 17:50:01.324 INFOHdfsReader$Job - Start Read orcfile .
2023-12-11 17:50:01.326 INFOHdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2023-12-11 17:50:01.326 INFOHdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2023-12-11 17:50:01.327 INFOReader$Task - read start
2023-12-11 17:50:01.327 INFOReader$Task - reading file :
2023-12-11 17:50:01.328 INFOHdfsReader$Job - Start Read orcfile .
2023-12-11 17:50:01.328 INFOReader$Task - read start
2023-12-11 17:50:01.328 INFOReader$Task - reading file :
2023-12-11 17:50:01.328 INFOHdfsReader$Job - Start Read orcfile .
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
信息: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2023-12-11 17:50:01.646 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:01.776 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:01.778 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
信息: FooterCacheHitRatio: 0/1
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202185 duration=760 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
2023-12-11 17:50:02.288 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
信息: FooterCacheHitRatio: 0/1
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202296 duration=871 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
2023-12-11 17:50:02.369 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
信息: FooterCacheHitRatio: 0/1
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202433 duration=1008 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
2023-12-11 17:50:02.492 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
2023-12-11 17:50:02.647 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
2023-12-11 17:50:02.784 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
2023-12-11 17:50:02.867 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:11.235 INFOStandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.000s |All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-12-11 17:50:21.236 INFOStandAloneJobContainerCommunicator - Total 297312 records, 558177719 bytes | Speed 53.23MB/s, 29731 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.023s |All Task WaitReaderTime 25.333s | Percentage 0.00%
2023-12-11 17:50:30.385 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039}
十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156410867, length: 9223372036854775807}
2023-12-11 17:50:30.731 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:30.907 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:31.114 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039}
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156011050, length: 9223372036854775807}
2023-12-11 17:50:31.237 INFOStandAloneJobContainerCommunicator - Total 598944 records, 1125524739 bytes | Speed 54.11MB/s, 30163 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.040s |All Task WaitReaderTime 43.878s | Percentage 0.00%
2023-12-11 17:50:31.280 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 604159}
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156411995, length: 9223372036854775807}
2023-12-11 17:50:31.457 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:41.238 INFOStandAloneJobContainerCommunicator - Total 906144 records, 1702990600 bytes | Speed 55.07MB/s, 30720 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.056s |All Task WaitReaderTime 62.147s | Percentage 0.00%
2023-12-11 17:50:51.241 INFOStandAloneJobContainerCommunicator - Total 1203104 records, 2261515086 bytes | Speed 53.27MB/s, 29696 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.070s |All Task WaitReaderTime 92.372s | Percentage 0.00%
2023-12-11 17:50:56.098 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839}
十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 311140141, length: 9223372036854775807}
2023-12-11 17:50:56.469 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:57.564 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 604159}, max key = {originalTxn: 0, bucket: -1, row: 803839}
十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 312563739, length: 9223372036854775807}
2023-12-11 17:50:57.971 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:58.447 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839}
十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 310368886, length: 9223372036854775807}
2023-12-11 17:50:58.782 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:01.242 INFOStandAloneJobContainerCommunicator - Total 1703232 records, 3203376993 bytes | Speed 89.82MB/s, 50012 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.093s |All Task WaitReaderTime 127.642s | Percentage 0.00%
2023-12-11 17:51:11.242 INFOStandAloneJobContainerCommunicator - Total 1829984 records, 3440997771 bytes | Speed 22.66MB/s, 12675 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.101s |All Task WaitReaderTime 138.648s | Percentage 0.00%
2023-12-11 17:51:15.910 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799}
十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 417009783, length: 9223372036854775807}
2023-12-11 17:51:16.305 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:17.688 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799}
十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415835058, length: 9223372036854775807}
2023-12-11 17:51:18.017 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:18.695 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1105919}
十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415904892, length: 9223372036854775807}
2023-12-11 17:51:19.066 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:21.243 INFOStandAloneJobContainerCommunicator - Total 2342208 records, 4403928517 bytes | Speed 91.83MB/s, 51222 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.126s |All Task WaitReaderTime 178.482s | Percentage 0.00%
2023-12-11 17:51:31.245 INFOStandAloneJobContainerCommunicator - Total 2466496 records, 4638515258 bytes | Speed 22.37MB/s, 12428 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.132s |All Task WaitReaderTime 190.077s | Percentage 0.00%
2023-12-11 17:51:41.246 INFOStandAloneJobContainerCommunicator - Total 2967264 records, 5579213090 bytes | Speed 89.71MB/s, 50076 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.156s |All Task WaitReaderTime 229.517s | Percentage 0.00%
2023-12-11 17:51:41.359 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479}
十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 571347891, length: 9223372036854775807}
2023-12-11 17:51:41.747 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:41.998 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479}
十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 570444727, length: 9223372036854775807}
2023-12-11 17:51:42.390 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:44.650 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1105919}, max key = {originalTxn: 0, bucket: -1, row: 1305599}
十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 572342605, length: 9223372036854775807}
2023-12-11 17:51:44.978 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:51.248 INFOStandAloneJobContainerCommunicator - Total 3307424 records, 6219717845 bytes | Speed 61.08MB/s, 34016 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.172s |All Task WaitReaderTime 246.926s | Percentage 0.00%
2023-12-11 17:52:01.250 INFOStandAloneJobContainerCommunicator - Total 3614624 records, 6797953036 bytes | Speed 55.14MB/s, 30720 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.187s |All Task WaitReaderTime 279.975s | Percentage 0.00%
2023-12-11 17:52:02.446 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439}
十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 674573070, length: 9223372036854775807}
2023-12-11 17:52:02.859 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:52:03.369 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439}
十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 673856234, length: 9223372036854775807}
2023-12-11 17:52:03.751 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:52:04.567 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1305599}, max key = {originalTxn: 0, bucket: -1, row: 1602559}
十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 675641495, length: 9223372036854775807}
2023-12-11 17:52:04.881 INFOCosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:52:11.251 INFOStandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.202s |All Task WaitReaderTime 297.463s | Percentage 0.00%
2023-12-11 17:52:21.252 INFOStandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.215s |All Task WaitReaderTime 331.832s | Percentage 0.00%
...StarRocks数据顺利加载：
:['default_cluster:cndlopsns']>: > show partitions from ods.buckets_tmp_20231205__sams_cos;
+-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
| PartitionId | PartitionName | VisibleVersion | VisibleVersionTime| VisibleVersionHash | State| PartitionKey | Range                                                                      | DistributionKey | Buckets | ReplicationNum | StorageMedium | CooldownTime      | LastConsistencyCheckTime | DataSize | IsInMemory |
+-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
| 513180658   | p20231114   | 1            | 2023-12-11 09:49:41 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | 16.3GB   | false      |
| 504724038   | p20231115   | 1            | 2023-12-06 13:47:20 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 504722860   | p20231116   | 1            | 2023-12-06 13:47:20 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 504721682   | p20231206   | 1            | 2023-12-06 13:47:20 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 504722271   | p20231207   | 1            | 2023-12-06 13:47:20 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 504720504   | p20231208   | 1            | 2023-12-06 13:47:20 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 504721093   | p20231209   | 1            | 2023-12-06 13:47:20 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 505174943   | p20231210   | 1            | 2023-12-07 00:14:47 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 506968779   | p20231211   | 1            | 2023-12-08 00:04:57 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 508936021   | p20231212   | 1            | 2023-12-09 00:11:09 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 510079030   | p20231213   | 1            | 2023-12-10 00:05:38 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
| 510795860   | p20231214   | 1            | 2023-12-11 00:07:15 | 0                  | NORMAL | ts         | ; keys: ; ..types: ; keys: ; ) | uuid            | 196   | 2            | HDD         | 9999-12-31 15:59:59 | NULL                     | .000   | false      |
+-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
12 rows in set (0.002 sec)

:['default_cluster:cndlopsns']>: > 查询正常：
:['default_cluster:cndlopsns']>: > select * from ods.buckets_tmp_20231205__sams_cos where ts='2023-11-14' limit 2;
+------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
| ts         | uuid             | import_ds_ | unique_action_id                     | action_time   | report_time   | action_type          | ka_id    | action_session_id                                 | wx_app_id | wx_open_id | wx_union_id | external_user_id | merber_id         | local_id | encrypted_imei                     | encrypted_idfa | encrypted_mac | encrypted_android_id | encrypted_qq | encrypted_phone | encrypting_algorithm | chan_id | chan_refer_app_id | chan_shop_id             | chan_shop_name                                                                                                   | client_type| client_name | client_version | sdk_version | device_model | ip   | user_agent | page_path    | page_name | referrer | address | city | province | country | latitude | longitude | json_properties                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | fdate    | chan_custom_id | etl_load_time | tag_id    | tag_name   | event_name |
+------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
| 2023-11-14 | 8325 | 2023111412 | ed55491f-da41-4976-8601-49c4405d994f | 1699936160765 | 1699936132584 | element            | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL      | NULL       | NULL      | 274012627      | 10742100529761108 | NULL   | 9090 | NULL         | NULL          | NULL               | NULL         | NULL            | NULL               | NULL    | NULL            | 9991,6758,4834,6119,9996 | 苏州木渎DC                                     | sams-app-sdk | NULL      | NULL         | NULL      | NULL         | NULL | NULL       | HomeFragment | 首页      | NULL   | NULL    | NULL | NULL   | NULL    | NULL   | NULL      | *****                                                                                                                                                 | 20231114 | default      | NULL          | advantage | 普通会员   | NULL       |
| 2023-11-14 | 8614 | 2023111412 | f30f3672-1bad-478c-ab19-16b736b850ef | 1699936161032 | 1699936133093 | expose_sku_component | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL      | NULL       | NULL      | 274012627      | 10742100529761108 | NULL   | 9090 | NULL         | NULL          | NULL               | NULL         | NULL            | NULL               | NULL    | NULL            | 9991,6758,4834,6119,9996 | 苏州木渎DC                                     | sams-app-sdk | NULL      | NULL         | NULL      | NULL         | NULL | NULL       | HomeFragment | 首页      | NULL   | NULL    | NULL | NULL   | NULL    | NULL   | NULL      | *****                           | 20231114 | default      | NULL          | advantage | 普通会员   | NULL       |
+------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
2 rows in set (1.660 sec)速率不断太慢，Datax也就这样了：
2023-12-11 17:52:11.251 INFOStandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.202s |All Task WaitReaderTime 297.463s | Percentage 0.00%
2023-12-11 17:52:21.252 INFOStandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.215s |All Task WaitReaderTime 331.832s | Percentage 0.00% 
备注

之前有伙伴问到，为什么我的json里面字段用到“index”
"column": [
    {
      "name": "ts",
      "type": "string",
      "value": "2023-11-14"
    },
    {
      "name": "import_ds_",
      "type": "string",
      "index": 0
    },
    {
      "name": "unique_action_id",
      "type": "string",
      "index": 1
    },
    {
      "name": "action_time",
      "type": "string",
      "index": 2
    },<br>    ....
    ]这个地方经过尝试，原有的column1，column2，column3...的方式测试行不通， 因为我有个ts的字段需要造值。但如果使用
    {
      "name": "ts",
      "type": "string",
      "value": "2023-11-14"
    },
    {
      "name": "import_ds_",
      "type": "string",
    },<br>    ....这种方式， 则抛出：由于您配置了type, 则至少需要配置 index 或 value，这是一件令人头疼的事。
<231211 18:10:01>$ /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2023-12-11 18:10:30.213 INFOMessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-12-11 18:10:30.215 INFOMessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo
2023-12-11 18:10:30.226 INFOVMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-12-11 18:10:30.231 INFOEngine - the machine info=>

    osInfo:    Linux amd64 4.18.0-348.7.1.el8_5.x86_64
    jvmInfo:    Oracle Corporation 1.8 25.112-b15
    cpu num:    16

    totalPhysicalMemory:    -0.00G
    freePhysicalMemory:    -0.00G
    maxFileDescriptorCount:    -1
    currentOpenFileDescriptorCount:    -1

    GC Names   

    MEMORY_NAME                  | allocation_size                | init_size                     
    PS Eden Space                  | 1,280.00MB                     | 256.00MB                     
    Code Cache                     | 240.00MB                     | 2.44MB                        
    Compressed Class Space         | 1,024.00MB                     | 0.00MB                        
    PS Survivor Space            | 42.50MB                        | 42.50MB                        
    PS Old Gen                     | 2,731.00MB                     | 683.00MB                     
    Metaspace                      | -0.00MB                        | 0.00MB                        



2023-12-11 18:10:30.258 INFOPerfTrace - PerfTrace traceId=job_-1, isEnable=true
2023-12-11 18:10:30.258 INFOJobContainer - DataX jobContainer starts job.
2023-12-11 18:10:30.259 INFOJobContainer - Set jobId = 0
2023-12-11 18:10:30.269 INFOHdfsReader$Job - init() begin...
2023-12-11 18:10:30.274 ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:, Description:[没有 Index].- 由于您配置了type, 则至少需要配置 index 或 value
    at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) ~
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150) ~
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111) ~
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50) ~
    at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673) ~
    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303) ~
    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) ~
    at com.alibaba.datax.core.Engine.start(Engine.java:86)
    at com.alibaba.datax.core.Engine.entry(Engine.java:168)
    at com.alibaba.datax.core.Engine.main(Engine.java:201)
2023-12-11 18:10:30.277 INFOStandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |All Task WaitWriterTime 0.000s |All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-12-11 18:10:30.279 ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:, Description:[没有 Index].- 由于您配置了type, 则至少需要配置 index 或 value
    at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30)
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150)
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111)
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50)
    at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673)
    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303)
    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
    at com.alibaba.datax.core.Engine.start(Engine.java:86)
    at com.alibaba.datax.core.Engine.entry(Engine.java:168)
    at com.alibaba.datax.core.Engine.main(Engine.java:201)那么我是如何取得每个字段精准的index？
这里我用到了 orc-tools-1.8.0-uber.jar 这个包把orc里面的字段先解析出来，
下载：https://repo1.maven.org/maven2/org/apache/orc/orc-tools/
java -jar orc-tools-1.8.0-uber.jar meta <ORC文件>成功解析orc文件的元数据字段信息，Type: struct：代表的就是字段列与下标顺序。
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Processing data file part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000
Structure for part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000
File Version: 0.12 with ORC_14 by ORC Java 1.6.14
Rows: 849920
Compression: SNAPPY
Compression size: 131072
Calendar: Julian/Gregorian
Type: struct<import_ds_:int,unique_action_id:string,action_time:bigint,report_time:bigint,action_type:string,ka_id:bigint,action_session_id:string,uuid:string,wx_app_id:string,wx_open_id:string,wx_union_id:string,external_user_id:string,merber_id:string,local_id:string,encrypted_imei:string,encrypted_idfa:string,encrypted_mac:string,encrypted_android_id:string,encrypted_qq:string,encrypted_phone:string,encrypting_algorithm:string,chan_id:string,chan_refer_app_id:string,chan_shop_id:string,chan_shop_name:string,client_type:string,client_name:string,client_version:string,sdk_version:string,device_model:string,ip:string,user_agent:string,page_path:string,page_name:string,referrer:string,address:string,city:string,province:string,country:string,latitude:string,longitude:string,json_properties:string,fdate:string,tag_id:string,tag_name:string,chan_custom_id:string,etl_load_time:string>

Stripe Statistics: 
 
完。
 

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！

查看完整版本: DataX vs 腾讯云COS对象存储 -> StarRocks集群