# Kubernetes Upgrade 1.20.x to v1.22.x Using Kubespray - Docker to Containerd Migration

This is a template post but below steps tested on our clusters, so you can use as it is (By adapting the kubespray configurations in this article according to your own configuration, of course).

## Procedure

### 1\. Upgrade k8s to v1.21.6 using kubespray v2.17

All steps are done on the bootstrap (or one of the master) node using root user

```bash
sudo su -

export CLUSTER_NAME=<put cluster name here>
```

Uninstall old ansible package

```bash
pip uninstall ansible
```

Install newer python packages and optional packages

```bash
yum -y install python3-pip python3-devel vim wget unzip tmux
# tmux is optional, but it is useful to have it
```

Create virtualenv if not exist

```bash
pip3 install virtualenv
mkdir -p ~/.virtualenvs
pip3 install virtualenvwrapper

echo 'export WORKON_HOME=$HOME/.virtualenvs' >> ~/.bashrc
echo 'source /usr/bin/virtualenvwrapper.sh' >> ~/.bashrc
source ~/.bashrc
```

Install required kubespray release

```bash
mkdir -p /kubespray
cd /kubespray
wget https://github.com/kubernetes-sigs/kubespray/archive/release-2.17.zip
unzip release-2.17.zip
cd kubespray-release-2.17
```

Create virtualenv and install requirements

```bash
mkvirtualenv -p python3 kspray_217
pip install -r requirements.txt

mkdir -p inventory/$CLUSTER_NAME
cp -rfp inventory/sample/* inventory/$CLUSTER_NAME/
cp ../kubespray-release-2.16/inventory/$CLUSTER_NAME/inventory.cfg inventory/$CLUSTER_NAME/inventory.cfg
# if you didn't use kubespray v2.16, you can create a new inventory.cfg file using the sample file
```

Edit inventory/`$CLUSTER_NAME`/inventory.cfg if needed (e.g. add/remove nodes, etc)

```bash
vim inventory/$CLUSTER_NAME/inventory.cfg
```

Edit inventory/`$CLUSTER_NAME`/group\_vars/k8s\_cluster/k8s-cluster.yml

* Change kubernetes\_version to 1.21.6
    
* Uncomment and change kube\_token\_auth: true
    
* cluster\_name: $CLUSTER\_NAME
    
* kube\_network\_plugin: calico (Use your network plugin instead of calico if you use other network plugin)
    

```bash
vim inventory/$CLUSTER_NAME/group_vars/k8s_cluster/k8s-cluster.yml
```

Edit inventory/`$CLUSTER_NAME`/group\_vars/all/all.yml

* kube\_read\_only\_port: 10255
    

```bash
vim inventory/$CLUSTER_NAME/group_vars/all/all.yml
```

Apply upgrade

```bash
ansible-playbook upgrade-cluster.yml -b -i inventory/$CLUSTER_NAME/inventory.cfg --user root
```

---

### Migrate to Containerd on Worker Nodes

Cordon and drain one of the worker nodes

```bash
kubectl cordon <worker node name>
kubectl drain <worker node name> --ignore-daemonsets --delete-emptydir-data`
```

Stop kubelet and docker services

```bash
systemctl stop kubelet docker containerd
systemctl status kubelet docker containerd
```

Remove docker and containerd packages

```bash
yum -y remove docker-docker-ce-cli.x86_64 docker-ce-rootless-extras.x86_64 \
docker-ce.x86_64 docker-scan-plugin.x86_64 containerd
```

Install containerd. We need specific version of containerd

```bash
yum -y install yum-utils
yum-config-manager --add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum -y install containerd.io-1.4.9-3.1.el7
```

Configure containerd

```bash
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
```

Edit /etc/containerd/config.toml

* Add `SystemdCgroup = true` under \[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options\]
    

```bash
 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true
```

Enable & Start containerd

```bash
systemctl enable containerd
systemctl start containerd
systemctl status containerd
```

Append these two `KUBELET_ARGS` parameters to the `/etc/kubernetes/kubelet.env` file

```bash
KUBELET_ARGS=" \
--container-runtime=remote \
--container-runtime-endpoint=unix:///run/containerd/containerd.sock \
"
```

Restart kubelet service and make sure it is running correctly. It may take a few minutes for Kubelet to fully boot up and error logs may print during this time. Ignore 'Not found $NODENAME' logs.

```bash
systemctl restart kubelet
systemctl status kubelet
```

Check docker, containerd and kubelet services

```bash
systemctl status docker containerd kubelet
#docker should be stopped
```

Reboot, just to make sure everything restarts fresh before the node is uncordoned

```bash
reboot

systemctl status docker containerd kubelet
#docker should be stopped
```

Finally, uncordon the node

```bash
kubectl uncordon <node name>
```

### Migrate to Containerd on Master Nodes

Follow the same steps as worker nodes

Cordon and drain one of the master nodes

```bash
kubectl cordon <master node name>
kubectl drain --ignore-daemonsets --delete-emptydir-data <master node name>
```

Stop etcd, kubelet, containerd and docker services

```bash
systemctl stop kubelet etcd docker containerd
systemctl status kubelet docker etcd containerd
```

Remove docker and containerd packages

```bash
yum -y remove docker-docker-ce-cli.x86_64 docker-ce-rootless-extras.x86_64 \
docker-ce.x86_64 docker-scan-plugin.x86_64 containerd
```

Install containerd. We need specific version of containerd

```bash
yum -y install yum-utils
yum-config-manager --add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum -y install containerd.io-1.4.9-3.1.el7
```

Configure containerd

```bash
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
```

Edit /etc/containerd/config.toml

* Add `SystemdCgroup = true` under \[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options\]
    

```bash
 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true
```

```bash
vim /etc/containerd/config.toml
```

Enable & Start containerd

```bash
systemctl enable containerd
systemctl start containerd
systemctl status containerd
```

Append these two `KUBELET_ARGS` parameters to the `/etc/kubernetes/kubelet.env` file

```bash
vim /etc/kubernetes/kubelet.env
```

```bash
KUBELET_ARGS=" \
--container-runtime=remote \
--container-runtime-endpoint=unix:///run/containerd/containerd.sock \
"
```

Restart kubelet service and make sure it is running correctly. It may take a few minutes for Kubelet to fully boot up and error logs may print during this time. Ignore 'Not found $NODENAME' logs.

```bash
systemctl restart kubelet
systemctl status kubelet
```

After that we need to run the kubespray 2.17 cluster.yml again with the following changes and **\--limit=$NODENAME** parameter (Run this command for each master node)

Edit inventory/$CLUSTER\_NAME/group\_vars/k8s\_cluster/k8s-cluster.yml

* Change resolvconf\_mode: host\_resolvconf
    
* Change container\_manager: containerd
    

```bash
vim inventory/$CLUSTER_NAME/group_vars/k8s_cluster/k8s-cluster.yml
```

Edit inventory/`$CLUSTER_NAME`/group\_vars/etcd.yml

* Change etcd\_deployment\_type: host
    

```bash
vim inventory/$CLUSTER_NAME/group_vars/etcd.yml
```

Apply changes

```bash
ansible-playbook cluster.yml -b -i inventory/$CLUSTER_NAME/inventory.cfg --user root --limit=$NODENAME
```

Check etcd, docker, containerd and kubelet services

```bash
systemctl status etcd docker containerd kubelet
#docker should be stopped
```

Reboot, just to make sure everything restarts fresh before the node is uncordoned

```bash
reboot

systemctl status docker containerd kubelet etcd
# etcd service is exists on master nodes only
```

Uncordon the node

```bash
kubectl uncordon <node name>
```

---

### Upgrade Cluster to Kubernetes 1.22.8 Using Kubespray 2.18 (Optional)

Install kubespray release

```bash
mkdir -p /kubespray
cd /kubespray
wget https://github.com/kubernetes-sigs/kubespray/archive/release-2.18.zip
unzip release-2.18.zip
cd kubespray-release-2.18
```

Create virtualenv and install required python packages

```bash
mkvirtualenv -p python3 kspray_218
pip install -r requirements.txt

mkdir -p inventory/$CLUSTER_NAME
cp -rfp inventory/sample/* inventory/$CLUSTER_NAME/
cp ../kubespray-release-2.17/inventory/$CLUSTER_NAME/inventory.cfg inventory/$CLUSTER_NAME/inventory.cfg
```

Edit inventory/`$CLUSTER_NAME`/inventory.cfg if needed (e.g. add/remove nodes, etc)

```bash
vim inventory/$CLUSTER_NAME/inventory.cfg
```

Edit inventory/`$CLUSTER_NAME`/group\_vars/k8s\_cluster/k8s-cluster.yml

* Change kubernetes\_version to 1.22.8
    
* Uncomment and change kube\_token\_auth: true
    
* cluster\_name:
    
* kube\_network\_plugin: calico
    

```bash
vim inventory/$CLUSTER_NAME/group_vars/k8s_cluster/k8s-cluster.yml
```

Edit `inventory/$CLUSTER_NAME/group_vars/all/all.yml`

* kube\_read\_only\_port: 10255
    

```bash
vim inventory/$CLUSTER_NAME/group_vars/all/all.yml
```

Apply upgrade

```bash
ansible-playbook upgrade-cluster.yml -b -i inventory/$CLUSTER_NAME/inventory.cfg --user root
```

---

### After Upgrade

#### Check Ingress Controller Version

Versions smaller than Nginx ingress v1 are not supported in kubernetes version 1.22. Therefore, the ingress controller version needs to be updated. There are multiple ingress controllers used for this purpose. I will advance the example through the nginx ingress controller offered by kubernetes.

The following steps are experimental and will affect you negatively if you have made manual changes to the ingress controller. Proceed at your own risk.

Delete existing ingress controller and its resources; (Careful, this will delete all ingress resources)

```bash
kubectl delete namespace ingress-nginx
```

##### Install new ingress controller

* Development environment;
    

```bash
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/cloud/deploy.yaml
```

* Production environment;
    

```bash
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/baremetal/deploy.yaml
```

##### Edit Existing Ingress Objects

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: example-ingress
  namespace: app-namespace
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: example-service
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - example.com
    secretName: example-com-ssl-2022-2023
status:
  loadBalancer:
    ingress:
    - ip: X.X.X.X
```

Ingress must be include the following annotations:

```yaml
kubernetes.io/ingress.class: nginx
```

#### Check Log Forwarder Configuration (If Exists)

Docker and Containerd saves container logs in different formats. Tools like fluentd configured to collect logs from Docker need to be reconfigured for logs saved in containerd's format.

An example fluentd configuration for docker is as follows:

```xml
<parse>
  @type multi_format
  <pattern>
    # Get docker stdout-stderr json output logs file and parse it.
    format json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </pattern>
  <pattern>
    # Get the non json !docker! stdout or stderr logs and set message into the "log" key.
    format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
    time_format %Y-%m-%dT%H:%M:%S.%N%:z
  </pattern>
</parse>
```

An example fluentd configuration for containerd is as follows:

```xml
<parse>
  @type cri
  <parse>
    @type multi_format
    <pattern>
      format json
      keep_time_key true
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</parse>
```