k8s实战案例之部署Nginx+Tomcat+NFS实现动静分离

道家人  金牌会员 | 2023-6-5 10:26:19 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 850|帖子 850|积分 2550

1、基于镜像分层构建及自定义镜像运行Nginx及Java服务并基于NFS实现动静分离

1.1、业务镜像设计规划


根据业务的不同，我们可以导入官方基础镜像，在官方基础镜像的基础上自定义需要用的工具和环境，然后构建成自定义出自定义基础镜像，后续再基于自定义基础镜像，来构建不同服务的基础镜像，最后基于服务的自定义基础镜像构建出对应业务镜像；最后将这些镜像上传至本地harbor仓库，然后通过k8s配置清单，将对应业务运行至k8s集群之上；
1.2、Nginx+Tomcat+NFS实现动静分离架构图


客户端通过负载均衡器的反向代理来访问k8s上的服务， nginx pod和tomcat pod 由k8s svc 资源进行关联；所有数据（静态资源和动态资源）通过存储挂载至对应pod中；nginx作为服务入口，它负责接收客户端的请求，同时响应静态资源（到存储上读取，比如js文件，css文件，图片等）；后端动态资源，由nginx将请求转发至后端tomcat server 完成（tomcat负责数据写入，比如用户的上传的图片等等）；
2、自定义centos基础镜像构建
  1. root@k8s-master01:~/k8s-data/dockerfile/system/centos# ls
  2. CentOS7-aliyun-Base.repo  CentOS7-aliyun-epel.repo  Dockerfile  build-command.sh  filebeat-7.12.1-x86_64.rpm
  3. root@k8s-master01:~/k8s-data/dockerfile/system/centos# cat Dockerfile
  4. #自定义Centos 基础镜像
  5. FROM centos:7.9.2009
  6. ADD filebeat-7.12.1-x86_64.rpm /tmp
  7. # 添加阿里源
  8. ADD CentOS7-aliyun-Base.repo  CentOS7-aliyun-epel.repo /etc/yum.repos.d/  
  9. # 自定义安装工具和环境
  10. RUN yum makecache &&yum install -y /tmp/filebeat-7.12.1-x86_64.rpm vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop &&  rm -rf /etc/localtime /tmp/filebeat-7.12.1-x86_64.rpm && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime  && useradd nginx -u 2088
  11. root@k8s-master01:~/k8s-data/dockerfile/system/centos# cat build-command.sh
  12. #!/bin/bash
  13. #docker build -t  harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 .
  14. #docker push harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
  16. /usr/local/bin/nerdctl build -t harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 .
  17. /usr/local/bin/nerdctl push harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
  18. root@k8s-master01:~/k8s-data/dockerfile/system/centos#
复制代码
2.1、构建自定义centos基础镜像


2.2、验证自定义centos基础镜像

在harbor上验证镜像是否正常上传？

运行镜像为容器，验证对应镜像是否有我们添加的工具和环境？

3、基于自定义centos基础镜像构建nginx镜像
  1. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# ls
  2. Dockerfile  build-command.sh  nginx-1.22.0.tar.gz
  3. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# cat Dockerfile
  4. #Nginx Base Image
  5. # 导入自定义centos基础镜像
  6. FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
  7. # 安装编译环境
  8. RUN yum install -y vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop
  9. # 添加nginx源码至/usr/local/src/
  10. ADD nginx-1.22.0.tar.gz /usr/local/src/
  11. # 编译nginx
  12. RUN cd /usr/local/src/nginx-1.22.0 && ./configure  && make && make install && ln -sv  /usr/local/nginx/sbin/nginx /usr/sbin/nginx  &&rm -rf /usr/local/src/nginx-1.22.0.tar.gz
  13. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# cat build-command.sh
  14. #!/bin/bash
  15. #docker build -t harbor.ik8s.cc/pub-images/nginx-base:v1.18.0  .
  16. #docker push  harbor.ik8s.cc/pub-images/nginx-base:v1.18.0
  18. nerdctl build -t  harbor.ik8s.cc/pub-images/nginx-base:v1.22.0  .
  19. nerdctl push harbor.ik8s.cc/pub-images/nginx-base:v1.22.0
  20. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base#
复制代码
3.1、构建自定义nginx基础镜像


3.2、验证自定义nginx基础镜像

验证nginx基础镜像是否上传至harbor？

把nginx基础镜像运行为容器，看看nginx是否正常安装？

能够将nginx基础镜像运行为容器，并在容器内部启动nginx，表示nginx基础镜像就构建好了；
3.3、构建自定义nginx业务镜像
  1. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# ls
  2. Dockerfile  app1.tar.gz  build-command.sh  index.html  nginx.conf  webapp
  3. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat Dockerfile
  4. #Nginx 1.22.0
  5. # 导入nginx基础镜像
  6. FROM harbor.ik8s.cc/pub-images/nginx-base:v1.22.0
  7. # 添加nginx配置文件
  8. ADD nginx.conf /usr/local/nginx/conf/nginx.conf
  9. # 添加业务代码
  10. ADD app1.tar.gz  /usr/local/nginx/html/webapp/
  11. ADD index.html  /usr/local/nginx/html/index.html
  12. # 创建静态资源挂载路径
  13. RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/images
  14. # 暴露端口
  15. EXPOSE 80 443
  16. # 运行nginx
  17. CMD ["nginx"]
  18. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat nginx.conf
  19. user  nginx nginx;
  20. worker_processes  auto;
  21. daemon off;
  22. events {
  23.     worker_connections  1024;
  24. }
  26. http {
  27.     include       mime.types;
  28.     default_type  application/octet-stream;
  29.     #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
  30.     #                  '$status $body_bytes_sent "$http_referer" '
  31.     #                  '"$http_user_agent" "$http_x_forwarded_for"';
  32.     #access_log  logs/access.log  main;
  33.     sendfile        on;
  34.     #tcp_nopush     on;
  35.     #keepalive_timeout  0;
  36.     keepalive_timeout  65;
  37.     upstream  tomcat_webserver {
  38.         server  magedu-tomcat-app1-service.magedu:80;
  39.     }
  41.     server {
  42.         listen       80;
  43.         server_name  localhost;
  44.         location / {
  45.             root   html;
  46.             index  index.html index.htm;
  47.         }
  48.         location /webapp {
  49.             root   html;
  50.             index  index.html index.htm;
  51.         }
  52.         location /app1 {
  53.              proxy_pass  http://tomcat_webserver;
  54.              proxy_set_header   Host    $host;
  55.              proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  56.              proxy_set_header X-Real-IP $remote_addr;
  57.         }
  58.         error_page   500 502 503 504  /50x.html;
  59.         location = /50x.html {
  60.             root   html;
  61.         }
  62.     }
  63. }
  64. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat build-command.sh
  65. #!/bin/bash
  66. TAG=$1
  67. #docker build -t harbor.ik8s.cc/magedu/nginx-web1:${TAG} .
  68. #echo "镜像构建完成，即将上传到harbor"
  69. #sleep 1
  70. #docker push harbor.ik8s.cc/magedu/nginx-web1:${TAG}
  71. #echo "镜像上传到harbor完成"
  73. nerdctl build -t harbor.ik8s.cc/magedu/nginx-web1:${TAG} .
  74. nerdctl push harbor.ik8s.cc/magedu/nginx-web1:${TAG}
  75. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx#
复制代码
上述Dockerfile中主要基于nginx基础镜像添加业务代码，添加配置，以及定义运行nginx和暴露服务端口；

3.4、验证自定义nginx业务镜像

验证nginx业务镜像是否上传至harbor？

运行nginx业务镜像为容器，看看对应业务是否能够正常访问？

这里提示找不到magedu-tomcat-app1-service.magedu:80这个upstream ，这是因为我们在配置文件中写死了nginx调用后端tomcat在k8s中 svc的地址；所以该镜像只能运行在k8s环境中，并且在运行该镜像前对应k8s环境中，tomcat的svc必须存在；
4、基于自定义centos基础镜像构建tomcat镜像

4.1、基于自定义centos基础镜像构建jdk基础镜像
  1. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# ll
  2. total 190464
  3. drwxr-xr-x 2 root root      4096 Jun  4 04:04 ./
  4. drwxr-xr-x 6 root root      4096 Aug  9  2022 ../
  5. -rw-r--r-- 1 root root       389 Jun  4 04:04 Dockerfile
  6. -rw-r--r-- 1 root root       259 Jun  4 04:02 build-command.sh
  7. -rw-r--r-- 1 root root 195013152 Jun 22  2021 jdk-8u212-linux-x64.tar.gz
  8. -rw-r--r-- 1 root root      2105 Jun 22  2021 profile
  9. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat Dockerfile
  10. #JDK Base Image
  11. # 导入自定义centos基础镜像
  12. FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
  13. # 安装jdk环境
  14. ADD jdk-8u212-linux-x64.tar.gz /usr/local/src/
  15. RUN ln -sv /usr/local/src/jdk1.8.0_212 /usr/local/jdk
  16. ADD profile /etc/profile
  17. ENV JAVA_HOME /usr/local/jdk
  18. ENV JRE_HOME $JAVA_HOME/jre
  19. ENV CLASSPATH $JAVA_HOME/lib/:$JRE_HOME/lib/
  20. ENV PATH $PATH:$JAVA_HOME/bin
  21. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat build-command.sh
  22. #!/bin/bash
  23. #docker build -t harbor.ik8s.cc/pub-images/jdk-base:v8.212  .
  24. #sleep 1
  25. #docker push  harbor.ik8s.cc/pub-images/jdk-base:v8.212
  27. nerdctl build -t  harbor.ik8s.cc/pub-images/jdk-base:v8.212  .
  28. nerdctl push  harbor.ik8s.cc/pub-images/jdk-base:v8.212
  29. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212#
复制代码
4.1.1、构建自定义jdk基础镜像


4.1.2、验证自定义jdk基础镜像

验证jdk基础镜像是否上传至harbor?

运行jdk基础镜像为容器，看看jdk环境是否安装？

能够正常在容器内部执行java 命令表示jdk镜像构建没有问题；
4.2、基于自定义jdk镜像构建tomcat基础镜像
  1. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# ll
  2. total 9508
  3. drwxr-xr-x 2 root root    4096 Jun  4 04:23 ./
  4. drwxr-xr-x 6 root root    4096 Aug  9  2022 ../
  5. -rw-r--r-- 1 root root     390 Jun  4 04:22 Dockerfile
  6. -rw-r--r-- 1 root root 9717059 Jun 22  2021 apache-tomcat-8.5.43.tar.gz
  7. -rw-r--r-- 1 root root     275 Jun  4 04:23 build-command.sh
  8. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat Dockerfile
  9. #Tomcat 8.5.43基础镜像
  10. # 导入自定义jdk镜像
  11. FROM harbor.ik8s.cc/pub-images/jdk-base:v8.212
  12. # 创建tomcat安装目录、数据目录和日志目录
  13. RUN mkdir /apps /data/tomcat/webapps /data/tomcat/logs -pv
  14. # 安装tomcat
  15. ADD apache-tomcat-8.5.43.tar.gz  /apps
  16. RUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-8.5.43 /apps/tomcat && chown -R tomcat.tomcat /apps /data -R
  17. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat build-command.sh
  18. #!/bin/bash
  19. #docker build -t harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43  .
  20. #sleep 3
  21. #docker push  harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43
  23. nerdctl  build -t harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43  .
  24. nerdctl  push harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43
  25. root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43#
复制代码
4.2.1、构建自定义tomcat基础镜像


4.2.2、验证自定义tomcat基础镜像

验证自定义tomcat基础镜像是否上传至harbor？

将自定义tomcat镜像运行为容器，看看tomcat是否可正常访问呢？
  1. root@k8s-node01:~# nerdctl run -it --rm -p 8080:8080 harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 /bin/bash
  2. WARN[0000] skipping verifying HTTPS certs for "harbor.ik8s.cc"
  3. harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43:                                    resolved       |++++++++++++++++++++++++++++++++++++++|
  4. manifest-sha256:ba362a6c099d965ffae43aae04e3ecb15e86e39d9d88259f4be72378b75bd1e5: done           |++++++++++++++++++++++++++++++++++++++|
  5. config-sha256:1d5e5f5e1dcb8ce1c7f7d6ce9fcc967275309c14baaaddbf3aa86de26054d1d5:   done           |++++++++++++++++++++++++++++++++++++++|
  6. layer-sha256:fa5fdb4dc02a5e79b212be196324b9936efbc850390f86283498a0e01b344ec3:    done           |++++++++++++++++++++++++++++++++++++++|
  7. layer-sha256:00bab63d153828acf58242f1781bd40769cc8b69659f37a2a49918ff3bfca68c:    done           |++++++++++++++++++++++++++++++++++++++|
  8. layer-sha256:6ddb864b9c4e53f3079ec4839ea3bace75b0d5d3daeae0ae910c171646bc4f96:    done           |++++++++++++++++++++++++++++++++++++++|
  9. elapsed: 2.4 s                                                                    total:  18.6 M (7.7 MiB/s)                                       
  10. [root@f5752bba588f /]# ll /apps/
  11. total 4
  12. drwxr-xr-x 1 tomcat tomcat 4096 Jun  4 12:25 apache-tomcat-8.5.43
  13. lrwxrwxrwx 1 tomcat tomcat   26 Jun  4 12:25 tomcat -> /apps/apache-tomcat-8.5.43
  14. [root@f5752bba588f /]# cd /apps/tomcat/bin/
  15. [root@f5752bba588f bin]# ls
  16. bootstrap.jar       catalina.sh  commons-daemon-native.tar.gz  configtest.sh  digest.sh         shutdown.bat  startup.sh            tool-wrapper.bat  version.sh
  17. catalina-tasks.xml  ciphers.bat  commons-daemon.jar            daemon.sh      setclasspath.bat  shutdown.sh   tomcat-juli.jar       tool-wrapper.sh
  18. catalina.bat        ciphers.sh   configtest.bat                digest.bat     setclasspath.sh   startup.bat   tomcat-native.tar.gz  version.bat
  19. [root@f5752bba588f bin]# ./catalina.sh run
  20. Using CATALINA_BASE:   /apps/tomcat
  21. Using CATALINA_HOME:   /apps/tomcat
  22. Using CATALINA_TMPDIR: /apps/tomcat/temp
  23. Using JRE_HOME:        /usr/local/jdk/jre
  24. Using CLASSPATH:       /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar
  25. 04-Jun-2023 12:40:29.845 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.5.43
  26. 04-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Jul 4 2019 20:53:15 UTC
  27. 04-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:         8.5.43.0
  28. 04-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
  29. 04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.15.0-72-generic
  30. 04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
  31. 04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/local/src/jdk1.8.0_212/jre
  32. 04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_212-b10
  33. 04-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
  34. 04-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /apps/apache-tomcat-8.5.43
  35. 04-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /apps/apache-tomcat-8.5.43
  36. 04-Jun-2023 12:40:29.854 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/apps/tomcat/conf/logging.properties
  37. 04-Jun-2023 12:40:29.854 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
  38. 04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
  39. 04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
  40. 04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
  41. 04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
  42. 04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/apps/tomcat
  43. 04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/apps/tomcat
  44. 04-Jun-2023 12:40:29.857 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/apps/tomcat/temp
  45. 04-Jun-2023 12:40:29.857 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
  46. 04-Jun-2023 12:40:30.166 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
  47. 04-Jun-2023 12:40:30.193 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
  48. 04-Jun-2023 12:40:30.232 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"]
  49. 04-Jun-2023 12:40:30.236 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
  50. 04-Jun-2023 12:40:30.237 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1262 ms
  51. 04-Jun-2023 12:40:30.303 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
  52. 04-Jun-2023 12:40:30.303 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.43
  53. 04-Jun-2023 12:40:30.324 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/docs]
  54. 04-Jun-2023 12:40:30.918 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/docs] has finished in [593] ms
  55. 04-Jun-2023 12:40:30.919 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/examples]
  56. 04-Jun-2023 12:40:31.524 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/examples] has finished in [605] ms
  57. 04-Jun-2023 12:40:31.524 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/ROOT]
  58. 04-Jun-2023 12:40:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/ROOT] has finished in [16] ms
  59. 04-Jun-2023 12:40:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/manager]
  60. 04-Jun-2023 12:40:31.565 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/manager] has finished in [25] ms
  61. 04-Jun-2023 12:40:31.565 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/host-manager]
  62. 04-Jun-2023 12:40:31.597 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/host-manager] has finished in [32] ms
  63. 04-Jun-2023 12:40:31.600 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
  64. 04-Jun-2023 12:40:31.608 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
  65. 04-Jun-2023 12:40:31.611 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1373 ms
复制代码
访问tomcat

能够正常访问tomcat,说明tomcat基础镜像就构建的没有问题；
4.3、基于自定义tomcat镜像构建tomcat业务镜像
  1. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# ll
  2. total 23588
  3. drwxr-xr-x  3 root root     4096 Jun  4 04:58 ./
  4. drwxr-xr-x 11 root root     4096 Aug  9  2022 ../
  5. -rw-r--r--  1 root root      603 Jun  4 04:57 Dockerfile
  6. -rw-r--r--  1 root root      144 May 11 06:26 app1.tar.gz
  7. -rwxr-xr-x  1 root root      261 Jun  4 04:58 build-command.sh*
  8. -rwxr-xr-x  1 root root    23611 Jun 22  2021 catalina.sh*
  9. -rw-r--r--  1 root root 24086235 Jun 22  2021 filebeat-7.5.1-x86_64.rpm
  10. -rw-r--r--  1 root root      667 Oct 24  2021 filebeat.yml
  11. drwxr-xr-x  2 root root     4096 May 11 06:26 myapp/
  12. -rwxr-xr-x  1 root root      372 Jan 22  2022 run_tomcat.sh*
  13. -rw-r--r--  1 root root     6462 Oct 10  2021 server.xml
  14. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat Dockerfile
  15. #tomcat web1
  16. # 导入自定义tomcat镜像
  17. FROM harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43
  18. # 添加启动脚本和配置文件
  19. ADD catalina.sh /apps/tomcat/bin/catalina.sh
  20. ADD server.xml /apps/tomcat/conf/server.xml
  21. ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
  22. # 添加业务代码
  23. ADD app1.tar.gz /data/tomcat/webapps/app1/
  24. #ADD filebeat.yml /etc/filebeat/filebeat.yml
  25. RUN chown  -R nginx.nginx /data/ /apps/
  26. #ADD filebeat-7.5.1-x86_64.rpm /tmp/
  27. #RUN cd /tmp && yum localinstall -y filebeat-7.5.1-amd64.deb
  28. # 暴露端口
  29. EXPOSE 8080 8443
  30. # 启动tomcat
  31. CMD ["/apps/tomcat/bin/run_tomcat.sh"]
  32. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat build-command.sh
  33. #!/bin/bash
  34. TAG=$1
  35. #docker build -t  harbor.ik8s.cc/magedu/tomcat-app1:${TAG} .
  36. #sleep 3
  37. #docker push  harbor.ik8s.cc/magedu/tomcat-app1:${TAG}
  39. nerdctl build -t  harbor.ik8s.cc/magedu/tomcat-app1:${TAG} .
  40. nerdctl push  harbor.ik8s.cc/magedu/tomcat-app1:${TAG}
  41. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1#
  42. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat run_tomcat.sh
  43. #!/bin/bash
  44. #echo "nameserver 223.6.6.6" > /etc/resolv.conf
  45. #echo "192.168.7.248 k8s-vip.example.com" >> /etc/hosts
  47. #/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat &
  48. su - nginx -c "/apps/tomcat/bin/catalina.sh start"
  49. tail -f /etc/hosts
  50. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1#
复制代码
4.3.1、构建自定义tomcat业务镜像


4.3.2、验证自定义tomcat业务镜像

验证自定义tomcat业务镜像是否上传至harbor？

将自定义tomcat业务镜像运行为容器，看看对应业务是否正常访问？

访问业务

能够正常访问app1说明业务容器的镜像构建没有问题；
5、在k8s环境中运行tomcat
  1. root@k8s-master01:~/k8s-data/yaml/namespaces# ls
  2. magedu-ns.yaml
  3. root@k8s-master01:~/k8s-data/yaml/namespaces# cat magedu-ns.yaml
  4. apiVersion: v1
  5. kind: Namespace
  6. metadata:
  7.   name: magedu
  8. root@k8s-master01:~/k8s-data/yaml/namespaces# cd ../magedu/tomcat-app1/
  9. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# ll
  10. total 16
  11. drwxr-xr-x  2 root root 4096 Jun  4 06:16 ./
  12. drwxr-xr-x 12 root root 4096 Aug  9  2022 ../
  13. -rw-r--r--  1 root root  596 Jun 22  2021 hpa.yaml
  14. -rw-r--r--  1 root root 1849 Jun  4 05:18 tomcat-app1.yaml
  15. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# cat hpa.yaml
  16. #apiVersion: autoscaling/v2beta1
  17. apiVersion: autoscaling/v1
  18. kind: HorizontalPodAutoscaler
  19. metadata:
  20.   namespace: magedu
  21.   name: magedu-tomcat-app1-podautoscaler
  22.   labels:
  23.     app: magedu-tomcat-app1
  24.     version: v2beta1
  25. spec:
  26.   scaleTargetRef:
  27.     apiVersion: apps/v1
  28.     #apiVersion: extensions/v1beta1
  29.     kind: Deployment
  30.     name: magedu-tomcat-app1-deployment
  31.   minReplicas: 2
  32.   maxReplicas: 20
  33.   targetCPUUtilizationPercentage: 60
  34.   #metrics:
  35.   #- type: Resource
  36.   #  resource:
  37.   #    name: cpu
  38.   #    targetAverageUtilization: 60
  39.   #- type: Resource
  40.   #  resource:
  41.   #    name: memory
  43. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# cat tomcat-app1.yaml
  44. kind: Deployment
  45. #apiVersion: extensions/v1beta1
  46. apiVersion: apps/v1
  47. metadata:
  48.   labels:
  49.     app: magedu-tomcat-app1-deployment-label
  50.   name: magedu-tomcat-app1-deployment
  51.   namespace: magedu
  52. spec:
  53.   replicas: 1
  54.   selector:
  55.     matchLabels:
  56.       app: magedu-tomcat-app1-selector
  57.   template:
  58.     metadata:
  59.       labels:
  60.         app: magedu-tomcat-app1-selector
  61.     spec:
  62.       containers:
  63.       - name: magedu-tomcat-app1-container
  64.         image: harbor.ik8s.cc/magedu/tomcat-app1:v1
  65.         #command: ["/apps/tomcat/bin/run_tomcat.sh"]
  66.         imagePullPolicy: IfNotPresent
  67.         #imagePullPolicy: Always
  68.         ports:
  69.         - containerPort: 8080
  70.           protocol: TCP
  71.           name: http
  72.         env:
  73.         - name: "password"
  74.           value: "123456"
  75.         - name: "age"
  76.           value: "18"
  77.         #resources:
  78.         #  limits:
  79.         #    cpu: 1
  80.         #    memory: "512Mi"
  81.         #  requests:
  82.         #    cpu: 500m
  83.         #    memory: "512Mi"
  84.         volumeMounts:
  85.         - name: magedu-images
  86.           mountPath: /usr/local/nginx/html/webapp/images
  87.           readOnly: false
  88.         - name: magedu-static
  89.           mountPath: /usr/local/nginx/html/webapp/static
  90.           readOnly: false
  91.       volumes:
  92.       - name: magedu-images
  93.         nfs:
  94.           server: 192.168.0.42
  95.           path: /data/k8sdata/magedu/images
  96.       - name: magedu-static
  97.         nfs:
  98.           server: 192.168.0.42
  99.           path: /data/k8sdata/magedu/static
  100. #      nodeSelector:
  101. #        project: magedu
  102. #        app: tomcat
  103. ---
  104. kind: Service
  105. apiVersion: v1
  106. metadata:
  107.   labels:
  108.     app: magedu-tomcat-app1-service-label
  109.   name: magedu-tomcat-app1-service
  110.   namespace: magedu
  111. spec:
  112.   type: NodePort
  113.   ports:
  114.   - name: http
  115.     port: 80
  116.     protocol: TCP
  117.     targetPort: 8080
  118.     nodePort: 30092
  119.   selector:
  120.     app: magedu-tomcat-app1-selector
  121. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1#
复制代码
5.1、准备后端存储相关目录
  1. root@harbor:~# mkdir -pv /data/k8sdata/magedu/images
  2. mkdir: created directory '/data/k8sdata/magedu'
  3. mkdir: created directory '/data/k8sdata/magedu/images'
  4. root@harbor:~# mkdir -pv /data/k8sdata/magedu/static
  5. mkdir: created directory '/data/k8sdata/magedu/static'
  6. root@harbor:~# cat /etc/exports
  7. # /etc/exports: the access control list for filesystems which may be exported
  8. #               to NFS clients.  See exports(5).
  9. #
  10. # Example for NFSv2 and NFSv3:
  11. # /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
  12. #
  13. # Example for NFSv4:
  14. # /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
  15. # /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
  16. #
  17. /data/k8sdata/kuboard *(rw,no_root_squash)
  18. /data/volumes *(rw,no_root_squash)
  19. /pod-vol *(rw,no_root_squash)
  20. /data/k8sdata/myserver *(rw,no_root_squash)
  21. /data/k8sdata/mysite *(rw,no_root_squash)
  23. /data/k8sdata/magedu/images *(rw,no_root_squash)
  24. /data/k8sdata/magedu/static *(rw,no_root_squash)
  25. root@harbor:~# exportfs -av
  26. exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/kuboard".
  27.   Assuming default behaviour ('no_subtree_check').
  28.   NOTE: this default has changed since nfs-utils version 1.0.x
  30. exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/volumes".
  31.   Assuming default behaviour ('no_subtree_check').
  32.   NOTE: this default has changed since nfs-utils version 1.0.x
  34. exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/pod-vol".
  35.   Assuming default behaviour ('no_subtree_check').
  36.   NOTE: this default has changed since nfs-utils version 1.0.x
  38. exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/myserver".
  39.   Assuming default behaviour ('no_subtree_check').
  40.   NOTE: this default has changed since nfs-utils version 1.0.x
  42. exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/mysite".
  43.   Assuming default behaviour ('no_subtree_check').
  44.   NOTE: this default has changed since nfs-utils version 1.0.x
  46. exportfs: /etc/exports [7]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/images".
  47.   Assuming default behaviour ('no_subtree_check').
  48.   NOTE: this default has changed since nfs-utils version 1.0.x
  50. exportfs: /etc/exports [8]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/static".
  51.   Assuming default behaviour ('no_subtree_check').
  52.   NOTE: this default has changed since nfs-utils version 1.0.x
  54. exporting *:/data/k8sdata/magedu/static
  55. exporting *:/data/k8sdata/magedu/images
  56. exporting *:/data/k8sdata/mysite
  57. exporting *:/data/k8sdata/myserver
  58. exporting *:/pod-vol
  59. exporting *:/data/volumes
  60. exporting *:/data/k8sdata/kuboard
  61. root@harbor:~#
复制代码
5.2、将tomcat业务部署至k8s
  1. root@k8s-master01:~/k8s-data/yaml# cd namespaces/
  2. root@k8s-master01:~/k8s-data/yaml/namespaces# ls
  3. magedu-ns.yaml
  4. root@k8s-master01:~/k8s-data/yaml/namespaces# kubectl apply  -f magedu-ns.yaml
  5. namespace/magedu created
  6. root@k8s-master01:~/k8s-data/yaml/namespaces# ls
  7. magedu-ns.yaml
  8. root@k8s-master01:~/k8s-data/yaml/namespaces# cd ../magedu/tomcat-app1/
  9. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# ls
  10. hpa.yaml  tomcat-app1.yaml
  11. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl apply -f .
  12. horizontalpodautoscaler.autoscaling/magedu-tomcat-app1-podautoscaler created
  13. deployment.apps/magedu-tomcat-app1-deployment created
  14. service/magedu-tomcat-app1-service created
  15. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1#
复制代码
5.3、验证tomcat pod是否正常running?服务是否可正常访问？
  1. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl get pods -n magedu
  2. NAME                                             READY   STATUS    RESTARTS   AGE
  3. magedu-tomcat-app1-deployment-7754c8549c-prglk   1/1     Running   0          2m2s
  4. magedu-tomcat-app1-deployment-7754c8549c-xmg9l   1/1     Running   0          2m17s
  5. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl get svc -n magedu   
  6. NAME                         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
  7. magedu-tomcat-app1-service   NodePort   10.100.129.23   <none>        80:30092/TCP   2m23s
  8. root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1#
复制代码
访问k8s集群节点的30092端口，看看对应tomcat服务是否能够正常访问？

能够通过k8s集群节点访问tomcat服务，说明tomcat服务已经正常部署至k8s环境；
5.4、更改tomcat service 类型为无头服务，重新apply配置清单，让后端tomcat服务只能在k8s内部环境访问


6、在k8s环境中运行nginx，实现nginx+tomcat动静分离
  1. root@k8s-master01:~/k8s-data/yaml/magedu/nginx# cat nginx.yaml
  2. kind: Deployment
  3. apiVersion: apps/v1
  4. metadata:
  5.   labels:
  6.     app: magedu-nginx-deployment-label
  7.   name: magedu-nginx-deployment
  8.   namespace: magedu
  9. spec:
  10.   replicas: 1
  11.   selector:
  12.     matchLabels:
  13.       app: magedu-nginx-selector
  14.   template:
  15.     metadata:
  16.       labels:
  17.         app: magedu-nginx-selector
  18.     spec:
  19.       containers:
  20.       - name: magedu-nginx-container
  21.         image: harbor.ik8s.cc/magedu/nginx-web1:v1
  22.         #command: ["/apps/tomcat/bin/run_tomcat.sh"]
  23.         #imagePullPolicy: IfNotPresent
  24.         imagePullPolicy: Always
  25.         ports:
  26.         - containerPort: 80
  27.           protocol: TCP
  28.           name: http
  29.         - containerPort: 443
  30.           protocol: TCP
  31.           name: https
  32.         env:
  33.         - name: "password"
  34.           value: "123456"
  35.         - name: "age"
  36.           value: "20"
  37.         resources:
  38.           limits:
  39.             cpu: 500m
  40.             memory: 512Mi
  41.           requests:
  42.             cpu: 500m
  43.             memory: 256Mi
  45.         volumeMounts:
  46.         - name: magedu-images
  47.           mountPath: /usr/local/nginx/html/webapp/images
  48.           readOnly: false
  49.         - name: magedu-static
  50.           mountPath: /usr/local/nginx/html/webapp/static
  51.           readOnly: false
  52.       volumes:
  53.       - name: magedu-images
  54.         nfs:
  55.           server: 192.168.0.42
  56.           path: /data/k8sdata/magedu/images
  57.       - name: magedu-static
  58.         nfs:
  59.           server: 192.168.0.42
  60.           path: /data/k8sdata/magedu/static
  61.       #nodeSelector:
  62.       #  group: magedu
  64. ---
  65. kind: Service
  66. apiVersion: v1
  67. metadata:
  68.   labels:
  69.     app: magedu-nginx-service-label
  70.   name: magedu-nginx-service
  71.   namespace: magedu
  72. spec:
  73.   type: NodePort
  74.   ports:
  75.   - name: http
  76.     port: 80
  77.     protocol: TCP
  78.     targetPort: 80
  79.     nodePort: 30090
  80.   - name: https
  81.     port: 443
  82.     protocol: TCP
  83.     targetPort: 443
  84.     nodePort: 30091
  85.   selector:
  86.     app: magedu-nginx-selector
  87. root@k8s-master01:~/k8s-data/yaml/magedu/nginx#
复制代码
6.1、部署nginx业务部署至k8s
  1. root@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl apply -f .
  2. deployment.apps/magedu-nginx-deployment created
  3. service/magedu-nginx-service created
  4. root@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl get pod -n magedu
  5. NAME                                             READY   STATUS    RESTARTS   AGE
  6. magedu-nginx-deployment-5589bbf4bc-6gd2w         1/1     Running   0          14s
  7. magedu-tomcat-app1-deployment-7754c8549c-c7rtb   1/1     Running   0          8m7s
  8. magedu-tomcat-app1-deployment-7754c8549c-prglk   1/1     Running   0          19m
  9. root@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl get svc -n magedu
  10. NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
  11. magedu-nginx-service         NodePort    10.100.94.118   <none>        80:30090/TCP,443:30091/TCP   24s
  12. magedu-tomcat-app1-service   ClusterIP   10.100.129.23   <none>        80/TCP                       19m
  13. root@k8s-master01:~/k8s-data/yaml/magedu/nginx#
复制代码
6.2、验证nginx 服务是否正常可访问？



能够通过访问k8s集群节点的30090正常访问到nginx，说明nginx服务已经正常部署至k8s环境；
6.3、通过nginx 访问tomcat服务，看看tomcat是否能够被nginx代理？


能够通过nginx访问后端tomcat服务，说明nginx能够正常代理tomcat服务；
7、在负载均衡器上代理nginx服务
  1. root@k8s-ha01:~# cat /etc/haproxy/haproxy.cfg
  2. global
  3.         log /dev/log    local0
  4.         log /dev/log    local1 notice
  5.         chroot /var/lib/haproxy
  6.         stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  7.         stats timeout 30s
  8.         user haproxy
  9.         group haproxy
  10.         daemon
  12.         # Default SSL material locations
  13.         ca-base /etc/ssl/certs
  14.         crt-base /etc/ssl/private
  16.         # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
  17.         ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  18.         ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  19.         ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
  21. defaults
  22.         log     global
  23.         mode    http
  24.         option  httplog
  25.         option  dontlognull
  26.         timeout connect 5000
  27.         timeout client  50000
  28.         timeout server  50000
  29.         errorfile 400 /etc/haproxy/errors/400.http
  30.         errorfile 403 /etc/haproxy/errors/403.http
  31.         errorfile 408 /etc/haproxy/errors/408.http
  32.         errorfile 500 /etc/haproxy/errors/500.http
  33.         errorfile 502 /etc/haproxy/errors/502.http
  34.         errorfile 503 /etc/haproxy/errors/503.http
  35.         errorfile 504 /etc/haproxy/errors/504.http
  37. listen k8s_apiserver_6443
  38. bind 192.168.0.111:6443
  39. mode tcp
  40. #balance leastconn
  41. server k8s-master01 192.168.0.31:6443 check inter 2000 fall 3 rise 5
  42. server k8s-master02 192.168.0.32:6443 check inter 2000 fall 3 rise 5
  43. server k8s-master03 192.168.0.33:6443 check inter 2000 fall 3 rise 5
  46. listen nginx-svc-80
  47. bind 192.168.0.111:80
  48. mode tcp
  49. server k8s-node01 192.168.0.34:30090 check inter 2000 fall 3 rise 5
  50. server k8s-node02 192.168.0.35:30090 check inter 2000 fall 3 rise 5
  51. server k8s-node03 192.168.0.36:30090 check inter 2000 fall 3 rise 5
  52. root@k8s-ha01:~# systemctl restart haproxy
  53. root@k8s-ha01:~#
复制代码
8、验证通过vip访问nginx服务，看看是否能够正常访问？


9、在存储服务器上上传静态资源，看看nginx是正常读取静态资源？
  1. root@k8s-master01:~/ubuntu/html# scp -rp * 192.168.0.42:/data/k8sdata/magedu/images/
  2. root@192.168.0.42's password:
  3. 1.jpg                                                                                                                                    100%   40KB   8.6MB/s   00:00   
  4. index.html                                                                                                                               100%  277   282.5KB/s   00:00   
  5. root@k8s-master01:~/ubuntu/html#
复制代码
验证资源是否正常上传至存储对应目录？
  1. root@harbor:~# ll /data/k8sdata/magedu/images/
  2. total 16
  3. drwxr-xr-x 3 root root 4096 Jun  4 07:56 ./
  4. drwxr-xr-x 4 root root 4096 Jun  4 06:32 ../
  5. drwxr-xr-x 2 root root 4096 May 31 18:37 images/
  6. -rw-r--r-- 1 root root  277 Aug  5  2022 index.html
  7. root@harbor:~#
复制代码
访问nginx服务，看看对应资源是否能够被读取？

        出处：https://www.cnblogs.com/qiuhom-1874/        本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接，否则保留追究法律责任的权利.
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！

本帖子中包含更多资源

您需要 登录 才可以下载或查看，没有账号？立即注册

x
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

道家人

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

挺好的 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表