录制课程-集群部署-3-部署kubernetes1.22.10

kubernetes1.22.10部署

准备工作

兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及那些没有包管理器的发行版提供了通用说明。
每台机器 2 GB 或更多 RAM（任何更少都会为您的应用程序留下很小的空间）。
2 个 CPU 或更多。
集群中所有机器之间的完整网络连接（公共或专用网络都可以）。
每个节点的唯一主机名、MAC 地址和 product_uuid。有关更多详细信息，请参见此处。
您的机器上的某些端口是开放的。有关更多详细信息，请参见此处。
交换Swap分区。必须禁用Swap才能使 kubelet 正常工作。

我的服务器配置列表

没有必要按照我的这个配置去操作个人建议实验环境：正常演示环境2核2G就够了

需要开放的端口

虽然 etcd 端口包含在控制平面部分，但您也可以在外部或自定义端口上托管自己的 etcd 集群。可以覆盖所有默认端口号。当使用自定义端口时，这些端口需要打开而不是此处提到的默认值。一个常见的例子是 API 服务器端口，有时会切换到 443。或者，默认端口保持原样，API 服务器放在负载均衡器后面，该负载均衡器监听 443 并将请求路由到默认端口上的 API 服务器。

准备主机地址

修改每一台主机的/etc/hosts配置

# vim /etc/hosts
10.1.6.45 containerd-kube-master
10.1.6.46 containerd-kube-work1
10.1.6.47 containerd-kube-work2

关闭swap分区以及防火墙

进入fstab后找到你挂载的swap分区注释即可.

[root@bogon ~]# swapoff -a
[root@localhost ~]# echo "vm.swappiness = 0" >> /etc/sysctl.conf
[root@bogon ~]# vim /etc/fstab
# /dev/mapper/rl-swap     none                    swap    defaults        0 0
[root@localhost ~]# systemctl stop firewalld && systemctl disable firewalld # 关闭并且禁用防火墙

所有内容准备完成后重启三台服务器!

Containerd的基础安装和操作

本文档后续基于Containerd+RockyLinux+Kubeadmin部署Kubernetes1.24版本的环境。

需要注意的是,根据Kubernetes官方给出的公告。Kubernetes 1.20x版本将会废弃对Docker的支持

参考连接

通过阿里云镜像源安装

官方镜像站
三台主机全部执行此操作

[root@containerd-kube-master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@containerd-kube-master ~]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@containerd-kube-master ~]# yum -y install containerd.io

查看一下containerd的版本

[root@containerd-kube-master ~]# containerd -v
containerd containerd.io 1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6

生成containerd的配置文件

三台主机全部执行此操作默认情况下在/etc/containerd/config.toml已经有这个文件了,但是里面是一些简短的配置.

[root@containerd-kube-master containerd]# mkdir - /etc/containerd/
[root@containerd-kube-master containerd]# containerd config default | tee /etc/containerd/config.toml # 生成contained的默认配置

修改sandbox_img

pause：此镜像是kubernetes的基础容器
三台主机全部执行此操作由于部分用户无法进入k8s.gcr.io资源地址,需要对此地址进行替换.

[root@containerd-kube-master containerd]# vim /etc/containerd/config.toml
sandbox_image = "k8s.gcr.io/pause:3.6"  # 找到此选项并且修改为: registry.aliyuncs.com/google_containers/pause:3.6
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
  runtime_type = "io.containerd.runtime.v1.linux"# 修改为io.containerd.runtime.v1.linux

修改Systemd Cgroup驱动

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

然后重启containerd

 systemctl restart containerd

设置Crictl 查找的容器运行时

crictl config runtime-endpoint unix:///var/run/containerd/containerd.sock

(额外) 设置Containerd的私有仓库

如果你想用自己的私有仓库的话，可以通过如下方式进行设定

[plugins."io.containerd.grpc.v1.cri".image_decryption]
  key_model = "node"
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = ""
  [plugins."io.containerd.grpc.v1.cri".registry.auths]
  [plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."10.1.6.15".auth]
	   username = "" # 私有仓库账号
	   password = "" # 私有仓库密码
  [plugins."io.containerd.grpc.v1.cri".registry.headers]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.1.6.15"]
	endpoint = [""] # 私有仓库地址

启动containerd

三台主机全部执行此操作保证Active: active(running)的状态即可

[root@containerd-kube-master containerd]# systemctl restart containerd
[root@containerd-kube-master containerd]# systemctl enable containerd
[root@containerd-kube-master containerd]# systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-09-05 02:53:02 EDT; 6s ago
     Docs: <https://containerd.io>
  Process: 8465 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 8467 (containerd)
    Tasks: 12
   Memory: 25.2M
   CGroup: /system.slice/containerd.service
           └─8467 /usr/bin/containerd
 
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159619104-04:00" level=info msg="Start subscribing containerd event"
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159662811-04:00" level=info msg="Start recovering state"
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159718042-04:00" level=info msg="Start event monitor"
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159737174-04:00" level=info msg="Start snapshots syncer"
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159750064-04:00" level=info msg="Start cni network conf syncer for default"
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159756351-04:00" level=info msg="Start streaming server"
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159868546-04:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.159906215-04:00" level=info msg=serving... address=/run/containerd/containerd.sock
Sep 05 02:53:02 containerd-kube-master containerd[8467]: time="2022-09-05T02:53:02.160196660-04:00" level=info msg="containerd successfully booted in 0.021144s"
Sep 05 02:53:02 containerd-kube-master systemd[1]: Started containerd container runtime.

配置IPV4转发

三台全部执行

cat <<EOF | tee /etc/modules-load.d/kubernetes1.24.conf
overlay
br_netfilter
EOF
cat <<EOF | tee /etc/sysctl.d/kubernetes1.24-forsys.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
[root@containerd-kube-master containerd]# modprobe br_netfilter
[root@containerd-kube-master ~]# sysctl --system

kubernetes安装

kubernetes的安装依旧是基于aliyun

通过阿里云镜像源安装

三台全部安装由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用 yum install -y --nogpgcheck kubelet kubeadm kubectl 安装

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
EOF
[root@containerd-kube-work1 ~]# yum install -y kubelet-1.22.10 kubeadm-1.22.10 kubectl-1.22.10

可以通过yum --showduplicates list kubelet查看当前仓库中可用的版本

安装命令提示

安装后可以使用tab进行快捷提示

[root@containerd-kube-master ~]# yum -y install bash-completion
[root@containerd-kube-master ~]# source <(kubeadm completion bash) && source <(kubectl completion bash)

如果你想要永久的使其生效,请把他们加入到.bashrc当中

cd ~
[root@containerd-kube-master ~]# vim .bashrc
# .bashrc
 
# User specific aliases and functions
 
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
 
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
source <(kubeadm completion bash)
source <(kubectl completion bash)
source <(crictl completion bash)

启动kubelet

[root@containerd-kube-master containerd]# systemctl enable kubelet

初始化集群配置信息

[root@containerd-kube-master containerd]# kubeadm config print init-defaults > init.yaml

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.1.6.45 # 初始化的第一台master主机的地址
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: kubernetes-master # 定义主机的唯一名称
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers # 修改为阿里云地址
kind: ClusterConfiguration
kubernetesVersion: 1.22.10
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12   # Pod使用的子网地址
scheduler: {}

[root@containerd-kube-master containerd]# kubeadm init --config=init.yaml   # 等待镜像Pull完成
[init] Using Kubernetes version: v1.24.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

创建admin配置目录

[root@containerd-kube-master containerd]# mkdir -p $HOME/.kube
[root@containerd-kube-master containerd]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@containerd-kube-master containerd]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

创建集群网络

因为flannel不支持网络隔离,所以不想用了!

不再基于flannel,而是基于calico

[root@containerd-kube-master .kube]# curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml -O
# -->这个地址好像是用不了了,现在404了...
curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O

编辑calico.yaml

- name: CALICO_IPV4POOL_CIDR # 修改CIDR为Kubernetes的子网地址,即初始化集群的serviceSubnet
    value: "10.96.0.0/12"

创建calico网络

[root@containerd-kube-master ~]# kubectl apply -f calico.yaml

加入集群

如果初始化成功会出现Your Kubernetes control-plane has initialized successfully!

在node节点执行

[root@containerd-kube-work1 containerd]# kubeadm join 10.1.6.45:6443 --token abcdef.0123456789abcdef \\
        --discovery-token-ca-cert-hash sha256:417d4385cc4f0cc572b106a6a13bf59fd865421f12a401f3660afa6ca19e500d

验证集群

查看master节点的Pod是否全部启动

[root@containerd-kube-master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-6799f5f4b4-pf8w8    1/1     Running   0          2m16s
kube-system   calico-node-lzcjk                           1/1     Running   0          54s
kube-system   calico-node-mrqkd                           1/1     Running   0          2m16s
kube-system   calico-node-r45bc                           1/1     Running   0          71s
kube-system   coredns-74586cf9b6-gkmbl                    1/1     Running   0          2m37s
kube-system   coredns-74586cf9b6-tgh6f                    1/1     Running   0          2m37s
kube-system   etcd-kubernetes-master                      1/1     Running   2          2m42s
kube-system   kube-apiserver-kubernetes-master            1/1     Running   2          2m43s
kube-system   kube-controller-manager-kubernetes-master   1/1     Running   2          2m43s
kube-system   kube-proxy-mx4bg                            1/1     Running   0          54s
kube-system   kube-proxy-ssw98                            1/1     Running   0          71s
kube-system   kube-proxy-tpgfj                            1/1     Running   0          2m38s
kube-system   kube-scheduler-kubernetes-master            1/1     Running   2          2m43s

从master上查看节点是否已经全部Ready

[root@containerd-kube-master ~]# kubectl get nodes
NAME                    STATUS   ROLES           AGE     VERSION
containerd-kube-work1   Ready    <none>          55s     v1.24.2
containerd-kube-work2   Ready    <none>          38s     v1.24.2
containerd-kube-master  Ready    control-plane   2m29s   v1.24.2

到此为止,1.24的kubernetes已经安装完毕

提一个小问题

看一下你们的coredns是否在同一个节点上,如果在同一个节点上,建议重新分配一下coredns保证其高可用性

[root@containerd-kube-master ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE   IP            NODE                    NOMINATED NODE   READINESS GATES
kube-system   coredns-74586cf9b6-gkmbl                    1/1     Running   0          32m   10.105.56.1   kubernetes-master       <none>           <none>
kube-system   coredns-74586cf9b6-tgh6f                    1/1     Running   0          32m   10.105.56.3   kubernetes-master       <none>           <none>

重新分配coredns

[root@containerd-kube-master ~]# kubectl --namespace kube-system rollout restart deployment coredns

问题解决

使用crictl image出现`WARN[0000] image connect using default endpoints`

出现该问题的原因是由于crictl不知道使用那个sock导致的

[root@containerd-kube-master ~]# crictl image
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.

解决方法如下

默认情况下containerd的sock存放于/run/containerd/containerd.sock

# 重新设置一下使用的runtime-endpoint
crictl config runtime-endpoint unix:///run/containerd/containerd.sock

默认生成的crictl存放在/etc/crictl.yaml

[root@containerd-kube-master containerd]# vim /etc/crictl.yaml

编辑配置文件

runtime-endpoint: "unix:///run/containerd/containerd.sock"
image-endpoint: "unix:///run/containerd/containerd.sock" # 新版本增加了image-endpoint需要手动更改
timeout: 10
debug: false
pull-image-on-create: false
disable-pull-on-run: false

[root@containerd-kube-master containerd]# systemctl daemon-reload
[root@containerd-kube-master containerd]# crictl image
IMAGE               TAG                 IMAGE ID            SIZE

Master主集群加入token过期如何处理

默认情况下,该token只有24小时,如果token值过期的话需要重新生成

--discovery-token-ca-cert-hash sha256:793106d3b4305808d55c3cdb211f89a768bec86ecef64264b131dc8f2548da16

查看当前master集群的token列表

[root@containerd-kube-master .kube]# kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   8h          2022-09-06T10:10:05Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

重新生成一份token

[root@containerd-kube-master .kube]# kubeadm token create

通过证书hashtokne

[root@containerd-kube-master .kube]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

kubernetes1.22.10部署

准备工作

我的服务器配置列表

需要开放的端口

准备主机地址

关闭swap分区以及防火墙

Containerd的基础安装和操作

通过阿里云镜像源安装

生成containerd的配置文件

修改sandbox_img

修改Systemd Cgroup驱动

设置Crictl 查找的容器运行时

(额外) 设置Containerd的私有仓库

启动containerd

配置IPV4转发

kubernetes安装

通过阿里云镜像源安装

安装命令提示

启动kubelet

初始化集群配置信息

创建admin配置目录

创建集群网络

加入集群

验证集群

提一个小问题

问题解决

使用crictl image出现WARN[0000] image connect using default endpoints

Master主集群加入token过期如何处理

使用crictl image出现`WARN[0000] image connect using default endpoints`