基于nerdctl+buildkitd+containerd实现镜像构建
1、容器技术简介容器技术除了的docker之外，还有coreOS的rkt、google的gvisor、以及docker开源的containerd、redhat的podman、阿⾥的pouch等，为了保证容器⽣态的标准性和健康可持续发展，包括Linux 基⾦会、Docker、微软、红帽、⾕歌和IBM等公司在2015年6⽉共同成⽴了⼀个叫open container（OCI）的组织，其⽬的就是制定开放的标准的容器规范，⽬前OCI⼀共发布了两个规范，分别是runtime spec和image format spec，有了这两个规范，不同的容器公司开发的容器只要兼容这两个规范，就可以保证容器的可移植性和相互可操作性。
containerd官网：https://containerd.io/
gvisor官网：https://gvisor.dev/
podman官网：https://podman.io
pouch项目地址：https://github.com/alibaba/pouch
buildkit: 从Docker公司的开源出来的⼀个镜像构建⼯具包，⽀持OCI标准的镜像构建,项目地址https://github.com/moby/buildkit
2、buildkitd组成部分
https://img2023.cnblogs.com/blog/1503305/202305/1503305-20230529003335494-1506673074.png
buildkitd(服务端)，⽬前⽀持runc和containerd作为镜像构建环境，默认是runc，可以更换为containerd。
buildctl(客户端)，负责解析Dockerfile⽂件，并向服务端buildkitd发出构建请求。
3、部署buildkitd
3.1、下载二进制包
wget https://github.com/moby/buildkit/releases/download/v0.11.6/buildkit-v0.11.6.linux-amd64.tar.gz解压压缩包，将二进制文件软连接至path环境变量
root@k8s-master01:/usr/local/src# ls
buildkit-v0.11.6.linux-amd64.tar.gz
root@k8s-master01:/usr/local/src# tar xf buildkit-v0.11.6.linux-amd64.tar.gz
root@k8s-master01:/usr/local/src# ls
binbuildkit-v0.11.6.linux-amd64.tar.gz
root@k8s-master01:/usr/local/src# cd bin
root@k8s-master01:/usr/local/src/bin# ls
buildctl buildkit-qemu-arm buildkit-qemu-mips64 buildkit-qemu-ppc64lebuildkit-qemu-s390xbuildkitd
buildkit-qemu-aarch64buildkit-qemu-i386buildkit-qemu-mips64elbuildkit-qemu-riscv64buildkit-runc
root@k8s-master01:/usr/local/src/bin# ln -s /usr/local/src/bin/* /usr/local/bin/
root@k8s-master01:/usr/local/src/bin# buildctl --help
NAME:
buildctl - build utility
USAGE:
buildctl command
VERSION:
v0.11.6
COMMANDS:
du disk usage
prune clean up build cache
build, bbuild
debug debug utilities
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--debug enable debug output in logs
--addr value buildkitd address (default: "unix:///run/buildkit/buildkitd.sock")
--tlsservername valuebuildkitd server name for certificate validation
--tlscacert value CA certificate for validation
--tlscert value client certificate
--tlskey value client key
--tlsdir value directory containing CA certificate, client certificate, and client key
--timeout value timeout backend connection after value seconds (default: 5)
--help, -h show help
--version, -v print the version
root@k8s-master01:/usr/local/src/bin# 能够正常在bash中执行buildkit --help ，表示对应命令已经正常软连接至path环境中。
3.2、提供buildkit.socket文件
root@k8s-master01:/usr/local/src/bin# cat /lib/systemd/system/buildkit.socket
Description=BuildKit
Documentation=https://github.com/moby/buildkit
ListenStream=%t/buildkit/buildkitd.sock
WantedBy=sockets.target
root@k8s-master01:/usr/local/src/bin# 3.3、提供buildkit.service文件
root@k8s-master01:/usr/local/src/bin# cat /lib/systemd/system/buildkitd.service
Description=BuildKit
Requires=buildkit.socket
After=buildkit.socketDocumentation=https://github.com/moby/buildkit
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true
WantedBy=multi-user.target
root@k8s-master01:/usr/local/src/bin# 3.4、启动buildiktd服务
root@k8s-master01:/usr/local/src/bin# systemctl daemon-reload
root@k8s-master01:/usr/local/src/bin# systemctl enable buildkitd
Created symlink /etc/systemd/system/multi-user.target.wants/buildkitd.service → /lib/systemd/system/buildkitd.service.
root@k8s-master01:/usr/local/src/bin# systemctl restart buildkitd
root@k8s-master01:/usr/local/src/bin# systemctl status buildkitd
● buildkitd.service - BuildKit
Loaded: loaded (/lib/systemd/system/buildkitd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-05-31 13:32:38 UTC; 4s ago
Main PID: 12191 (buildkitd)
Tasks: 9 (limit: 4571)
Memory: 7.8M
CPU: 125ms
CGroup: /system.slice/buildkitd.service
└─12191 /usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true
May 31 13:32:38 k8s-master01.ik8s.cc systemd: Started BuildKit.
May 31 13:32:38 k8s-master01.ik8s.cc buildkitd: time="2023-05-31T13:32:38Z" level=warning msg="using host network as the default"
May 31 13:32:38 k8s-master01.ik8s.cc buildkitd: time="2023-05-31T13:32:38Z" level=warning msg="git source cannot be enabled: failed to find git
binary: exec: \"git\": executable file not found in $PATH"
May 31 13:32:38 k8s-master01.ik8s.cc buildkitd: time="2023-05-31T13:32:38Z" level=info msg="found worker \"kffvitvra0b27yi956wa4xcal\", labels=m
ap[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproject.buildkit.worker.containerd.uuid:f27e8d21-8b1d-44ac-9233-f55b9b028fc0 o
rg.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:k8s-master01.ik8s.cc org.mobyproject.buildkit.worker.networ
k:host org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64
/v2 linux/386]"
May 31 13:32:38 k8s-master01.ik8s.cc buildkitd: time="2023-05-31T13:32:38Z" level=info msg="found 1 workers, default=\"kffvitvra0b27yi956wa4xcal
\""
May 31 13:32:38 k8s-master01.ik8s.cc buildkitd: time="2023-05-31T13:32:38Z" level=warning msg="currently, only the default worker can be used."
May 31 13:32:38 k8s-master01.ik8s.cc buildkitd: time="2023-05-31T13:32:38Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
root@k8s-master01:/usr/local/src/bin# 4、基于nginx代理⾃签名证书harbor并实现https
4.1、将harbor修改为http协议
停止harbor服务
root@harbor:/app/harbor# docker-compose down
[+] Running 13/13
✔ Container notary-server Removed 0.8s
✔ Container nginx Removed 1.7s
✔ Container harbor-jobservice Removed 0.7s
✔ Container trivy-adapter Removed 0.6s
✔ Container notary-signer Removed 0.5s
✔ Container harbor-portal Removed 0.3s
✔ Container harbor-core Removed 3.5s
✔ Container harbor-db Removed 0.7s
✔ Container registry Removed 0.5s
✔ Container redis Removed 0.5s
✔ Network harbor_notary-sig Removed 0.8s
✔ Network harbor_harbor Removed 0.2s
✔ Network harbor_harbor-notaryRemoved 0.5s
root@harbor:/app/harbor# 修改harbor配置文件，注释https和证书的配置
https://img2023.cnblogs.com/blog/1503305/202305/1503305-20230531214152116-1957275795.png
重新prepare
root@harbor:/app/harbor# ls
LICENSEcommoncommon.shdocker-compose.ymlharbor.v2.8.0.tar.gzharbor.ymlharbor.yml.tmplinstall.shprepare
root@harbor:/app/harbor# ./prepare
prepare base dir is set to /app/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/notary/notary-signer-ca.crt
Clearing the configuration file: /config/notary/notary-signer.key
Clearing the configuration file: /config/notary/server-config.postgres.json
Clearing the configuration file: /config/notary/server_env
Clearing the configuration file: /config/notary/notary-signer.crt
Clearing the configuration file: /config/notary/signer_env
Clearing the configuration file: /config/notary/signer-config.postgres.json
Clearing the configuration file: /config/notary/root.crt
Clearing the configuration file: /config/trivy-adapter/env
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/nginx/conf.d/notary.upstream.conf
Clearing the configuration file: /config/nginx/conf.d/notary.server.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/portal/nginx.conf
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
root@harbor:/app/harbor# 启动harbor
root@harbor:/app/harbor# docker-compose up -d
[+] Running 10/10
✔ Network harbor_harbor Created 0.2s
✔ Container harbor-log Started 1.2s
✔ Container registry Started 2.5s
✔ Container harbor-portal Started 2.7s
✔ Container registryctl Started 2.7s
✔ Container harbor-db Started 2.7s
✔ Container redis Started 2.6s
✔ Container harbor-core Started 3.2s
✔ Container harbor-jobserviceStarted 4.2s
✔ Container nginx Started 4.3s
root@harbor:/app/harbor# 如果你给harbor添加的有service文件，也可以使用systemctl stop/start harbor来停止/启动harbor
验证harbor是否可以正常登录？
https://img2023.cnblogs.com/blog/1503305/202305/1503305-20230531233824420-458774571.png
4.2、nginx实现harbor的反向代理
4.2.1、nginx安装及配置
root@k8s-deploy:~# cd /usr/local/src/
root@k8s-deploy:/usr/local/src# wget https://nginx.org/download/nginx-1.22.0.tar.gz
--2023-05-31 13:59:55--https://nginx.org/download/nginx-1.22.0.tar.gz
Resolving nginx.org (nginx.org)... 52.58.199.22, 3.125.197.172, 2a05:d014:edb:5704::6, ...
Connecting to nginx.org (nginx.org)|52.58.199.22|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1073322 (1.0M)
Saving to: ‘nginx-1.22.0.tar.gz’
nginx-1.22.0.tar.gz 100%[=======================================================================================>] 1.02M37.8KB/s in 27s
2023-05-31 14:00:23 (38.9 KB/s) - ‘nginx-1.22.0.tar.gz’ saved
root@k8s-deploy:/usr/local/src# tar xf nginx-1.22.0.tar.gz
root@k8s-deploy:/usr/local/src# cd nginx-1.22.0
root@k8s-deploy:/usr/local/src/nginx-1.22.0# ./configure --prefix=/apps/nginx \
> --with-http_ssl_module \
> --with-http_v2_module \
> --with-http_realip_module \
> --with-http_stub_status_module \
> --with-http_gzip_static_module \
> --with-pcre \
> --with-stream \
> --with-stream_ssl_module \
> --with-stream_realip_modulehttps://img2023.cnblogs.com/blog/1503305/202305/1503305-20230531221323938-1135518205.png
上述报错没有找到http模块依赖的PCRE库
解决办法：安装libpcre3 libpcre3-dev
root@k8s-deploy:/usr/local/src/nginx-1.22.0# apt install libpcre3 libpcre3-dev -y安装上述依赖后，重新configure
https://img2023.cnblogs.com/blog/1503305/202305/1503305-20230531221739442-1509276467.png
上述报错没有找到ssl模块依赖的openssl库
解决办法：安装openssl libssl-dev
root@k8s-deploy:/usr/local/src/nginx-1.22.0# apt install openssl libssl-dev -yhttps://img2023.cnblogs.com/blog/1503305/202305/1503305-20230531222119746-1541296320.png
执行上述./configure 没有报错，并且能够看到上述截图，表示编译环境通过检查，可以进行下一步编译和安装
编译安装nginx
root@k8s-deploy:/usr/local/src/nginx-1.22.0# make && make install创建证书⽬录，并生成私钥，证书
4.2.1.1、创建证书⽬录,并生成ca私钥
root@k8s-deploy:/apps/nginx/sbin# mkdir /apps/nginx/certs
root@k8s-deploy:/apps/nginx/sbin# cd /apps/nginx/certs
root@k8s-deploy:/apps/nginx/certs# openssl genrsa -out ca.key 40964.2.1.2、用ca私钥生成ca自签名证书
root@k8s-deploy:/apps/nginx/certs# openssl req -x509 -new -nodes -sha512 -days 3650 \
>-subj "/C=CN/ST=Beijing/L=Beijing/O=Personal-CA/OU=Personal/CN=CA.com" \
>-key ca.key \
>-out ca.crt
root@k8s-deploy:/apps/nginx/certs# ls
ca.crtca.key
root@k8s-deploy:/apps/nginx/certs# 4.2.1.3、生成harbor服务器私钥
root@k8s-deploy:/apps/nginx/certs# openssl genrsa -out magedu.net.key 4096
root@k8s-deploy:/apps/nginx/certs# ls
ca.crtca.keymagedu.net.key
root@k8s-deploy:/apps/nginx/certs# 4.2.1.4、用harbor服务器私钥生成生成证书签名请求csr文件
root@k8s-deploy:/apps/nginx/certs# openssl req -sha512 -new \
> -subj "/C=CN/ST=Beijing/L=Beijing/O=test/OU=Personal/CN=magedu.net" \
> -key magedu.net.key \
> -out magedu.net.csr
root@k8s-deploy:/apps/nginx/certs# ls
ca.crtca.keymagedu.net.csrmagedu.net.key
root@k8s-deploy:/apps/nginx/certs# 4.2.1.5、使用ca证书和私钥为harbor服务器签发证书
root@k8s-deploy:/apps/nginx/certs# openssl x509 -req -sha512 -days 3650 \
> -CA ca.crt -CAkey ca.key -CAcreateserial \
> -in magedu.net.csr \
> -out magedu.net.crt
Certificate request self-signature ok
subject=C = CN, ST = Beijing, L = Beijing, O = test, OU = Personal, CN = magedu.net
root@k8s-deploy:/apps/nginx/certs# ls
ca.crtca.keymagedu.net.crtmagedu.net.csrmagedu.net.key
root@k8s-deploy:/apps/nginx/certs# 编辑nginx配置文件，配置证书和反向代理harbor
root@k8s-deploy:/apps/nginx/certs# cat /apps/nginx/conf/nginx.conf
worker_processes1;
events {
worker_connections1024;
}
http {
client_max_body_size 1000m;
server {
listen 80;
listen 443 ssl;
ssl_certificate /apps/nginx/certs/magedu.net.crt;
ssl_certificate_key /apps/nginx/certs/magedu.net.key;
ssl_session_cache shared:sslcache:20m;
ssl_session_timeout 10m;
location / {
if ($scheme = http ){
rewrite / https://harbor.magedu.net permanent;
}
proxy_pass http://192.168.0.42;
}
}
}
root@k8s-deploy:/apps/nginx/certs# 检查nginx配置文件语法
root@k8s-deploy:/apps/nginx/certs# cd ..
root@k8s-deploy:/apps/nginx# /apps/nginx/sbin/nginx -t
nginx: the configuration file /apps/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /apps/nginx/conf/nginx.conf test is successful
root@k8s-deploy:/apps/nginx# 启动nginx
root@k8s-deploy:/apps/nginx# /apps/nginx/sbin/nginx
root@k8s-deploy:/apps/nginx# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:80 0.0.0.0:*
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 511 0.0.0.0:443 0.0.0.0:*
root@k8s-deploy:/apps/nginx# 4.2.2、浏览器访问测试nginx 看看是否反向代理给harbor？是否为https？
https://img2023.cnblogs.com/blog/1503305/202306/1503305-20230601010246499-2106340086.gif
查看证书信息
https://img2023.cnblogs.com/blog/1503305/202306/1503305-20230601011120180-1633242569.png
4.2.3、buildkitd配置⽂件
root@k8s-master01:/usr/local/src/bin# cat /etc/buildkit/buildkitd.toml
http = true
insecure = true
root@k8s-master01:4.2.4、nerdctl配置⽂件
root@k8s-master01:/usr/local/src/bin# cat /etc/nerdctl/nerdctl.toml
namespace = "k8s.io"
debug = false
debug_full = false
insecure_registry = true
root@k8s-master01:/usr/local/src/bin#5、测试镜像构建
5.1、nerdctl常⽤命令
5.1.1 给nerdctl添加命令补全
root@k8s-master01:~# echo "source
页:
[1]