数据仓库和贸易智能分析:ClickHouse擅长处置惩罚大规模的数据,可以用于构建企业级的数据仓库,支持复杂的OLAP查询,可用实时数仓，适合各种贸易分析和报表应用。它是一个数据库： 数据库既有存储引擎，也有查询引擎。ClickHouse 可以有用地从各种来源获取数据，其查询引擎可提供低延长查询相应。
实时分析和监控:ClickHouse以毫秒级的相应时间处置惩罚大量数据,非常适合实时监控、非常检测和故障预警等场景。
Web分析:ClickHouse可以收罗和分析网站访问日志,计算各种指标如UV、PV、跳出率等,为网站运营提供数据支持。
用户行为分析：精细化运营分析，日活，留存率分析，路径分析，有序漏斗转化率分析，Session 分析等。
物联网数据分析:ClickHouse可以处置惩罚海量的传感器数据,用于分析设备运行状态、猜测故障等。
广告技术:ClickHouse可以处置惩罚广告点击、展示等海量数据,为广告优化和结果分析提供支持。
金融交易分析:ClickHouse可以处置惩罚交易数据,为金融市场分析提供实时计算本领，欺诈检测，低延长查询对于辨认任何欺诈活动和防止损失。
游戏数据分析:ClickHouse可以收集和分析游戏用户行为数据,为游戏运营和产物优化提供依据。
酒店管理行业：利用 ClickHouse 相识最新的预订、订价、收入和物业绩效信息。制造商在 ClickHouse 数据库的基础上创建物流规划体系。
媒体和娱乐公司：使用 ClickHouse 监控流媒体质量、管理广告投放和推送个性化促销信息。
云服务提供商: ClickHouse 的基础上创建云资源监控和网络分析。
version: '3.8'
services:
clickhouse-01:
image: "clickhouse/clickhouse-server{CHVER:-latest}"
user: "101:101"
container_name: clickhouse-01
hostname: clickhouse-01
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.1
volumes:
- ${PWD}/fs/volumes/clickhouse-01/etc/clickhouse-server/config.d/config.xml:/etc/clickhouse-server/config.d/config.xml
- ${PWD}/fs/volumes/clickhouse-01/etc/clickhouse-server/users.d/users.xml:/etc/clickhouse-server/users.d/users.xml
ports:
- "127.0.0.1:8123:8123"
- "127.0.0.1:9000:9000"
depends_on:
- clickhouse-keeper-01
- clickhouse-keeper-02
- clickhouse-keeper-03
clickhouse-02:
image: "clickhouse/clickhouse-server
user: "101:101"
container_name: clickhouse-02
hostname: clickhouse-02
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.2
volumes:
- ${PWD}/fs/volumes/clickhouse-02/etc/clickhouse-server/config.d/config.xml:/etc/clickhouse-server/config.d/config.xml
- ${PWD}/fs/volumes/clickhouse-02/etc/clickhouse-server/users.d/users.xml:/etc/clickhouse-server/users.d/users.xml
ports:
- "127.0.0.1:8124:8123"
- "127.0.0.1:9001:9000"
depends_on:
- clickhouse-keeper-01
- clickhouse-keeper-02
- clickhouse-keeper-03
clickhouse-03:
image: "clickhouse/clickhouse-server
user: "101:101"
container_name: clickhouse-03
hostname: clickhouse-03
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.3
volumes:
- ${PWD}/fs/volumes/clickhouse-03/etc/clickhouse-server/config.d/config.xml:/etc/clickhouse-server/config.d/config.xml
- ${PWD}/fs/volumes/clickhouse-03/etc/clickhouse-server/users.d/users.xml:/etc/clickhouse-server/users.d/users.xml
ports:
- "127.0.0.1:8125:8123"
- "127.0.0.1:9002:9000"
depends_on:
- clickhouse-keeper-01
- clickhouse-keeper-02
- clickhouse-keeper-03
clickhouse-04:
image: "clickhouse/clickhouse-server
user: "101:101"
container_name: clickhouse-04
hostname: clickhouse-04
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.4
volumes:
- ${PWD}/fs/volumes/clickhouse-04/etc/clickhouse-server/config.d/config.xml:/etc/clickhouse-server/config.d/config.xml
- ${PWD}/fs/volumes/clickhouse-04/etc/clickhouse-server/users.d/users.xml:/etc/clickhouse-server/users.d/users.xml
ports:
- "127.0.0.1:8126:8123"
- "127.0.0.1:9003:9000"
depends_on:
- clickhouse-keeper-01
- clickhouse-keeper-02
- clickhouse-keeper-03
clickhouse-keeper-01:
image: "clickhouse/clickhouse-keeper
user: "101:101"
container_name: clickhouse-keeper-01
hostname: clickhouse-keeper-01
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.5
volumes:
- ${PWD}/fs/volumes/clickhouse-keeper-01/etc/clickhouse-keeper/keeper_config.xml:/etc/clickhouse-keeper/keeper_config.xml
ports:
- "127.0.0.1:9181:9181"
clickhouse-keeper-02:
image: "clickhouse/clickhouse-keeper
user: "101:101"
container_name: clickhouse-keeper-02
hostname: clickhouse-keeper-02
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.6
volumes:
- ${PWD}/fs/volumes/clickhouse-keeper-02/etc/clickhouse-keeper/keeper_config.xml:/etc/clickhouse-keeper/keeper_config.xml
ports:
- "127.0.0.1:9182:9181"
clickhouse-keeper-03:
image: "clickhouse/clickhouse-keeper
user: "101:101"
container_name: clickhouse-keeper-03
hostname: clickhouse-keeper-03
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.7
volumes:
- ${PWD}/fs/volumes/clickhouse-keeper-03/etc/clickhouse-keeper/keeper_config.xml:/etc/clickhouse-keeper/keeper_config.xml
ports:
- "127.0.0.1:9183:9181"
ch-proxy:
image: contentsquareplatform/chproxy:v1.26.4
platform: linux/amd64
container_name: ch-proxy
hostname: ch-proxy
networks:
cluster_2S_2R_ch_proxy:
ipv4_address: 192.168.9.10
ports:
- "127.0.0.1:443:443"
- "127.0.0.1:80:80"
volumes:
- ${PWD}/fs/volumes/ch-proxy/config/config.yml:/opt/config.yml
depends_on:
- clickhouse-01
- clickhouse-02
- clickhouse-03
- clickhouse-04
command: [-config, /opt/config.yml]
networks:
cluster_2S_2R_ch_proxy:
driver: bridge
ipam:
config:
- subnet: 192.168.9.0/24
gateway: 192.168.9.254
version: '3'
services:
clickhouse:
container_name: clickhouse
image: clickhouse/clickhouse-server:23.8
restart: always
ports:
- 8123:8123 # http
- 9000:9000 # native protocol
- 9363:9363 # metrics server
volumes:
- clickhouse-data:/var/lib/clickhouse/
- clickhouse-logs:/var/log/clickhouse-server/
- ./config_files/clickhouse/etc/clickhouse-server:/etc/clickhouse-server
cap_add:
- SYS_NICE
- NET_ADMIN
- IPC_LOCK
ulimits:
nofile:
soft: 262144
hard: 262144
depends_on:
- minio
- minio-bucket-creator
prometheus:
container_name: prometheus
image: prom/prometheus:v2.37.9
restart: always
ports:
- 9090:9090
volumes:
- ./config_files/prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yaml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
depends_on:
- minio-prometheus-monitoring-cluster
- minio-prometheus-monitoring-node
- minio-prometheus-monitoring-bucket
grafana:
container_name: grafana
image: grafana/grafana:10.0.3-ubuntu
ports:
- 3000:3000
volumes:
- grafana_data:/var/lib/grafana
- ./config_files/grafana/etc/grafana/provisioning/dashboards/:/etc/grafana/provisioning/dashboards/
- ./config_files/grafana/etc/grafana/provisioning/datasources/:/etc/grafana/provisioning/datasources/
- ./config_files/grafana/var/lib/grafana/dashboards/:/var/lib/grafana/dashboards/
# uncomment the following line, if you don't have access to grafana.com plugins and download a plugin manually (read README.md file)
- ./config_files/grafana/var/lib/grafana/plugins/:/var/lib/grafana/plugins/
depends_on:
- prometheus
minio:
container_name: minio
image: minio/minio:RELEASE.2023-09-07T02-05-02Z
restart: always
env_file:
- .env
ports:
- 9001:9001
- 9002:9002
volumes:
- minio_data:/var/lib/minio/data
command: minio server /var/lib/minio/data --address 0.0.0.0:9002 --console-address ":9001"
minio-bucket-creator:
container_name: minio-bucket-creator
image: minio/mc:RELEASE.2023-09-13T23-08-58Z
env_file:
- .env
entrypoint: >
/bin/bash -c "
/usr/bin/mc alias set myminio http://minio:9002 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD;
/usr/bin/mc mb myminio/${MINIO_CLICKHOUSE_BACKUP_BUCKET:-clickhouse};
/usr/bin/mc anonymous set public myminio/${MINIO_CLICKHOUSE_BACKUP_BUCKET:-clickhouse};
exit 0;
"
depends_on:
- minio
minio-prometheus-monitoring-cluster:
container_name: minio-prometheus-monitoring-cluster
image: minio/mc:RELEASE.2023-09-13T23-08-58Z
env_file:
- .env
volumes:
- ./config_files/prometheus/templates/prometheus.yaml:/home/prometheus-template.yaml:ro
- ./config_files/prometheus/prometheus.yaml:/home/prometheus.yaml
entrypoint: >
/bin/bash -c "
cp /home/prometheus-template.yaml /home/prometheus.yaml;
/usr/bin/mc alias set myminio http://minio:9002 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD > /dev/null;
/usr/bin/mc admin prometheus generate myminio cluster | sed '1d' | awk '{print \" \" $0}' >> /home/prometheus.yaml;
"
depends_on:
- minio
minio-prometheus-monitoring-node:
container_name: minio-prometheus-monitoring-node
image: minio/mc:RELEASE.2023-09-13T23-08-58Z
env_file:
- .env
volumes:
- ./config_files/prometheus/prometheus.yaml:/home/prometheus.yaml
entrypoint: >
/bin/bash -c "
/usr/bin/mc alias set myminio http://minio:9002 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD > /dev/null;
/usr/bin/mc admin prometheus generate myminio node | sed '1d' | awk '{print \" \" $0}' >> /home/prometheus.yaml;
"
depends_on:
- minio
- minio-prometheus-monitoring-cluster
minio-prometheus-monitoring-bucket:
container_name: minio-prometheus-monitoring-bucket
image: minio/mc:RELEASE.2023-09-13T23-08-58Z
env_file:
- .env
volumes:
- ./config_files/prometheus/prometheus.yaml:/home/prometheus.yaml
entrypoint: >
/bin/bash -c "
/usr/bin/mc alias set myminio http://minio:9002 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD > /dev/null;
/usr/bin/mc admin prometheus generate myminio bucket | sed '1d' | awk '{print \" \" $0}' >> /home/prometheus.yaml;
"
depends_on:
- minio
- minio-prometheus-monitoring-cluster
- minio-prometheus-monitoring-node
volumes:
clickhouse-data:
clickhouse-logs:
prometheus_data:
grafana_data:
minio_data:
数据分片：提到了CH集群分片，这通常意味着数据被分割成多个部门，分布在差别的服务器或节点上，以提高处置惩罚效率和可扩展性。
实时与离线处置惩罚：体系支持实时写入和离线写入，这表明架构设计了处置惩罚差别数据流的本领，可以或许顺应差别的业务场景。
Spark与Flink任务：提到了Spark任务和Flink/FlinkSQL任务，这表明体系使用了这两种流行的大数据处置惩罚框架来处置惩罚数据。
位图索引：提到了位图当地表和标签用户位图表，位图索引是一种高效的数据布局，用于快速检索和存储大量数据。
用户ID哈希处置惩罚：用户ID通过Hash处置惩罚，这有助于保护用户隐私，同时也可以减少数据存储的需求。
Kafka消息队列：使用Kafka作为消息队列，这有助于实现数据的异步处置惩罚息争耦。
HTTP调用与自界说插件：体系支持通过HTTP调用自界说CH插件，这提供了机动性，允许根据须要扩展或修改体系功能。
安全与性能优化：提到了Rate Limiter（限流器）、Security（安全）、Customer Plugins（自界说插件）、Circuit Breaker（断路器）和Routing（路由），这些都是体系性能和安全性的紧张保障。
API网关：使用Proxy（如Apisix）作为API网关，这有助于管理API请求，提供负载均衡、服务发现等功能。
数据转换：用户行为数据结果须要转换成bitmap和标签位图结果，这表明体系须要进行数据格式的转换以顺应差别的查询需求。
多节点处置惩罚：提到了CH Shard1、CH Shard2、CH ShardN，这表明体系设计了多节点处置惩罚本领，以支持大规模的数据和高并发请求。
| 欢迎光临 IT评测·应用市场-qidao123.com技术社区 (https://dis.qidao123.com/) | Powered by Discuz! X3.4 |