Applies to SUSE Telco Cloud 3.6

60 Lifecycle actions #

This section covers the lifecycle management actions for clusters deployed via SUSE Telco Cloud.

60.1 Load Balancer Exclusion #

There are many lifecycle actions that require nodes to be drained. During the draining process, all pods will be moved to other nodes in the cluster. After the draining process is finished, the node does not host any services and therefore should not have any traffic routed to it. Load balancers, such as MetalLB, can be made aware of this by applying a label to the node:

node.kubernetes.io/exclude-from-external-load-balancers: "true"

For more details see: Kubernetes Documentation.

To see the labels on all your nodes in a cluster, you can run:

kubectl get nodes -o json | jq -r '.items[].metadata | .name, .labels'

In the case of upgrades of downstream clusters, this can be automated by annotating the RKE2ControlPlane on the management cluster:

rke2.controlplane.cluster.x-k8s.io/load-balancer-exclusion="true"

This immediately creates an annotation on all machine objects on the management cluster for that RKE2ControlPlane.

pre-drain.delete.hook.machine.cluster.x-k8s.io/rke2-lb-exclusion: ""

With this annotation on the machine objects, any node on the downstream cluster that is scheduled for draining will get the above node label attached prior to the start of the draining process. The label will be removed from the node once it is available and ready again.

60.2 Management cluster upgrades #

The upgrade of the management cluster is described in the Day 2 management cluster (Chapter 59, Management Cluster) documentation.

60.3 Downstream cluster upgrades #

Upgrading downstream clusters involves updating several components. The following sections cover the upgrade process for each of the components.

Upgrading the operating system

For this process, check the following reference (Chapter 50, Prepare downstream cluster image for connected scenarios) to build the new image with a new operating system version. With this new image generated by EIB, the next provision phase uses the new operating version provided. In the following step, the new image is used to upgrade the nodes.

Upgrading the RKE2 cluster

The changes required to upgrade the RKE2 cluster using the automated workflow are the following:

Change the block RKE2ControlPlane in the capi-provisioning-example.yaml shown in the following section (Chapter 52, Downstream cluster provisioning with Directed network provisioning (single-node)):
- Specify the desired rolloutStrategy.
- Change the version of the RKE2 cluster to the new version replacing ${RKE2_NEW_VERSION}.
- Decide if an ingress controller is to be deployed in the downstream cluster:
  - [Option 0]: Do not deploy any ingress controller
  - [Option 1]: Deploy only Traefik
  - [Option 2]: Deploy both Ingress-NGINX and Traefik (to be used for complex ingress migration scenarios)

Note

The Traefik ingress provider integrated into RKE2/K3s is the only ingress controller supported in SUSE Telco Cloud 3.6 release, being still possible to temporarily run Ingress-NGINX alongside Traefik in order to support complex ingress migration scenarios, but only after SUSE Telco Cloud Management and/or Downstream clusters have been upgraded to version 3.6 and for the time required to perform that migration. Since Traefik is not yet the default ingress controller in RKE2 (it will be from RKE2 v1.36 onwards), it must be explicitly "requested" from the RKE2 server configuration file.

RKE2 Ingress NGINX to Traefik Migration guide provides details on the ingress migration paths available once the Traefik ingress controller replaces the discontinued Ingress-NGINX.

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: RKE2ControlPlane
metadata:
  name: single-node-cluster
  namespace: default
spec:
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: Metal3MachineTemplate
    name: single-node-cluster-controlplane
  version: ${RKE2_NEW_VERSION}
  replicas: 1
  rolloutStrategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 0
  serverConfig:
    cni: cilium
  #===========================================================================
  # Uncomment the following lines if selecting [Option 0]: Do not deploy
  # any ingress controller
  #===========================================================================
    #disableComponents:
    #  pluginComponents:
    #  - "rke2-ingress-nginx"
  #---------------------------------------------------------------------------
  rolloutStrategy:
    rollingUpdate:
      maxSurge: 0
  registrationMethod: "control-plane-endpoint"
  agentConfig:
    format: ignition
    additionalUserData:
      config: |
        variant: fcos
        version: 1.4.0
        systemd:
          units:
          - name: rke2-preinstall.service
            enabled: true
            contents: |
              [Unit]
              Description=rke2-preinstall
              Wants=network-online.target
              Before=rke2-install.service
              ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
              [Service]
              Type=oneshot
              User=root
              ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
              ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"node-label:\" >> /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"  - metal3.io/uuid=$(jq -r .uuid /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
              ExecStartPost=/bin/sh -c "umount /mnt"
              [Install]
              WantedBy=multi-user.target
          # rke2-ingress-deployment.service unit
          - name: rke2-ingress-deployment.service
            enabled: true
            contents: |
              [Unit]
              Description=rke2-ingress-deployment
              Wants=rke2-preinstall.service
              Before=rke2-install.service
              ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
              [Service]
              Type=oneshot
              User=root
              #===============================================================================================================================
              # Leave one (and only one) of the two following ExecStart lines uncommented, depending on the desired ingress-controller(s):
              #   [Option 1]: Deploy only "Traefik"
              #   [Option 2]: Deploy both "Ingress-NGINX" and "Traefik"
              #
              # Keep both commented ONLY in case of seleting [Option 0]: "Do not deploy any ingress controller"
              #===============================================================================================================================
              #ExecStart=/bin/sh -c "echo \"ingress-controller: traefik\" >> /etc/rancher/rke2/config.yaml"                       # [Option 1]
              ExecStart=/bin/sh -c "echo -e \"ingress-controller:\n- ingress-nginx\n- traefik\" >> /etc/rancher/rke2/config.yaml" # [Option 2]
              #-------------------------------------------------------------------------------------------------------------------------------
              [Install]
              WantedBy=multi-user.target
        storage:
          directories:
          - path: /var/lib/rancher/rke2/server/manifests
            overwrite: true
          files:
          #############################################################################
          # if [Option 2]: "Deploy both `Ingress-NGINX` and `Traefik`" is selected
          #############################################################################
          - path: /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
            overwrite: true
            contents:
              inline: |
                apiVersion: helm.cattle.io/v1
                kind: HelmChartConfig
                metadata:
                  name: rke2-ingress-nginx
                  namespace: kube-system
                spec:
                  valuesContent: |-
                    controller:
                      hostPort:
                        enabled: false  # not needed when exposing through a type:LoadBalancer service
                      config:
                        use-forwarded-headers: "true"
                        enable-real-ip: "true"
                      publishService:
                        enabled: true
                      service:
                        enabled: true
                        type: LoadBalancer
                        externalTrafficPolicy: Local
            mode: 0644
            user:
              name: root
            group:
              name: root
          #############################################################################
          # if [Option 1]: "Deploy only `Traefik`" OR  [Option 2]: "Deploy both
          #`Ingress-NGINX` and `Traefik`" is selected
          #############################################################################
          - path: /var/lib/rancher/rke2/server/manifests/rke2-traefik-config.yaml
            overwrite: true
            contents:
              inline: |
                apiVersion: helm.cattle.io/v1
                kind: HelmChartConfig
                metadata:
                  name: rke2-traefik
                  namespace: kube-system
                spec:
                  valuesContent: |-
                    ingressClass:
                      isDefaultClass: false  # Assumes [Option 2]; set to true if [Option 1]: "only deploying `Traefik`"
                    ports:
                      web:
                        hostPort: null    # disallow hostPort
                        exposedPort: 80
                      websecure:
                        hostPort: null    # disallow hostPort
                        exposedPort: 443
                    service:
                      enabled: true
                      type: LoadBalancer
                      spec:
                        externalTrafficPolicy: Local
                        allocateLoadBalancerNodePorts: false  # k8s GA from 1.24; supported by MetalLB
                    providers:
                      kubernetesIngressNginx:  # this provider allows Traefik to "understand" most of the Ingress-NGINX annotations
                        enabled: true
                        ingressClass: "rke2-ingress-nginx-migration"
                        controllerClass: "rke2.cattle.io/ingress-nginx-migration"
            mode: 0644
            user:
              name: root
            group:
              name: root
    kubelet:
      extraArgs:
      - provider-id=metal3://BAREMETALHOST_UUID
    nodeName: "localhost.localdomain"

Change the block Metal3MachineTemplate in the capi-provisioning-example.yaml shown in the following section (Chapter 52, Downstream cluster provisioning with Directed network provisioning (single-node)):
- Change the image name and checksum to the new version generated in the previous step.
- Add the directive nodeReuse to true to avoid creating a new node.
- Add the directive automatedCleaningMode to metadata to enable the automated cleaning for the node.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: single-node-cluster-controlplane
  namespace: default
spec:
  nodeReuse: True
  template:
    spec:
      automatedCleaningMode: metadata
      dataTemplate:
        name: single-node-cluster-controlplane-template
      hostSelector:
        matchLabels:
          cluster-role: control-plane
      image:
        checksum: http://imagecache.local:8080/${NEW_IMAGE_GENERATED}.sha256
        checksumType: sha256
        format: raw
        url: http://imagecache.local:8080/${NEW_IMAGE_GENERATED}.raw

Before applying the capi-provisioning-example.yaml file, it is always a good practice to inform external load balancers (e.g. MetalLB) about nodes being drained so that they do not route traffic to nodes in this state. As mentioned in the Section 60.1, “Load Balancer Exclusion” section, you can automate this by annotating the RKE2ControlPlane on the management cluster. In this example, an RKE2ControlPlane object called multinode-cluster is annotated:

kubectl annotate  RKE2ControlPlane/multinode-cluster  rke2.controlplane.cluster.x-k8s.io/load-balancer-exclusion="true"

Verify that the machine objects have been annotated:

pre-drain.delete.hook.machine.cluster.x-k8s.io/rke2-lb-exclusion: ""

Fetch the annotations for all your machine objects:

kubectl get machines -o json | jq -r '.items[].metadata | .name, .annotations'

Note

Without these annotations users might experience longer response times for services as the load-balancers are unaware of drained nodes.

After making these changes, the capi-provisioning-example.yaml file can be applied to the cluster using the following command:

kubectl apply -f capi-provisioning-example.yaml

60.4 Longhorn Engine Image Cleanup #

This section describes how to implement automated cleanup of obsolete Longhorn engine images that accumulate after cluster upgrades.

60.4.1 Problem statement #

Longhorn does not automatically remove old engine images after upgrades. This is an intentional design decision to prevent accidental data loss — Longhorn retains previous engine image versions as a safety measure in case rollback is required.

However, in production environments with frequent upgrades, these unused engine images accumulate over time, resulting in:

Storage consumption: Each engine image consumes disk space on cluster nodes
Resource overhead: Kubernetes manages additional image resources unnecessarily
Operational complexity: Manual identification and cleanup of obsolete images

60.4.2 When to implement cleanup #

Engine image cleanup is recommended in the following scenarios:

Regular cluster upgrades: Environments that perform frequent Longhorn version upgrades
Storage-constrained deployments: Clusters with limited node storage capacity
Operational hygiene: As part of routine Day 2 maintenance procedures

Note

This cleanup procedure is classified as low risk because it only removes engine images with zero active references (refCount == 0). Images currently in use by volumes are protected and will not be deleted.

60.4.3 Solution architecture #

The cleanup solution uses a Kubernetes CronJob that executes periodic maintenance tasks:

Discovery: Identifies the latest Longhorn engine image version deployed in the cluster
Candidate selection: Locates all older engine image versions with no active volume references
Cleanup execution: Deletes obsolete engine images that meet safety criteria
Audit logging: Records cleanup operations for operational visibility

The implementation requires:

ServiceAccount: For authentication to the Kubernetes API
RBAC permissions: Read and delete access to engineimages.longhorn.io resources
CronJob: Scheduled execution container with cleanup logic

60.4.4 Implementation #

The complete cleanup solution consists of four Kubernetes manifests deployed to the longhorn-system namespace.

Create ServiceAccount

Create a dedicated ServiceAccount for the cleanup CronJob:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: longhorn-engineimage-cleanup
  namespace: longhorn-system
  labels:
    app: longhorn-engineimage-cleanup

Configure RBAC permissions

Define a Role with minimal required permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: longhorn-engineimage-cleanup
  namespace: longhorn-system
  labels:
    app: longhorn-engineimage-cleanup
rules:
- apiGroups: [longhorn.io]
  resources: [engineimages]
  verbs: [get, list, delete]

Bind the Role to the ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: longhorn-engineimage-cleanup
  namespace: longhorn-system
  labels:
    app: longhorn-engineimage-cleanup
subjects:
- kind: ServiceAccount
  name: longhorn-engineimage-cleanup
  namespace: longhorn-system
roleRef:
  kind: Role
  name: longhorn-engineimage-cleanup
  apiGroup: rbac.authorization.k8s.io

Deploy cleanup CronJob

Deploy the automated cleanup CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: longhorn-engineimage-cleanup
  namespace: longhorn-system
  labels:
    app: longhorn-engineimage-cleanup
  annotations:
    description: "Cleanup old Longhorn engine images left over after upgrades"
    source: "SUSE Edge Telco Cloud - Day 2 Operations"
spec:
  # Execute weekly on Sunday at 2:00 AM UTC
  schedule: "0 2 * * 0"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 600
      template:
        metadata:
          labels:
            app: longhorn-engineimage-cleanup
        spec:
          serviceAccountName: longhorn-engineimage-cleanup
          restartPolicy: Never
          automountServiceAccountToken: true
          containers:
          - name: cleanup
            image: registry.suse.com/bci/bci-base:15.6
            command:
            - sh
            - -c
            - |
              set -e

              # Install required tools
              zypper --non-interactive install curl jq

              # Install kubectl from official Kubernetes
              curl -LO "https://dl.k8s.io/release/v1.28.0/bin/linux/amd64/kubectl"
              chmod +x kubectl
              mv kubectl /usr/local/bin/

              echo "=== Longhorn Engine Image Cleanup ==="
              echo "Automated maintenance - Day 2 Operations"
              echo ""

              echo "Step 1: Discovering latest engine image version..."

              ENGINE_IMAGES=$(kubectl get engineimages.longhorn.io -n longhorn-system -o json)

              LATEST_VERSION=$(echo "$ENGINE_IMAGES" | jq -r '
                .items[]
                | select(.status.version != null)
                | .status.version' |
                sed 's/^v//' |
                sort -V |
                tail -1)

              if [ -z "$LATEST_VERSION" ]; then
                echo "No engine images found. Nothing to cleanup."
                exit 0
              fi

              echo "Latest version detected: v${LATEST_VERSION}"
              echo ""

              echo "Step 2: Identifying obsolete engine images..."

              TO_DELETE=$(echo "$ENGINE_IMAGES" | jq -r --arg latest "v${LATEST_VERSION}" '
                .items[]
                | select(.status.version != $latest)
                | select(.status.refCount == 0)
                | .metadata.name')

              if [ -z "$TO_DELETE" ]; then
                echo "No obsolete engine images found"
                echo "All engine images are either:"
                echo "  - Latest version (v${LATEST_VERSION})"
                echo "  - Still in use (refCount > 0)"
                exit 0
              fi

              DELETE_COUNT=$(echo "$TO_DELETE" | wc -l | tr -d ' ')
              echo "Found $DELETE_COUNT obsolete engine image(s) for deletion:"
              echo "$TO_DELETE"
              echo ""

              COUNT=0
              FAILED=0
              for IMAGE_NAME in $TO_DELETE; do
                VERSION=$(echo "$ENGINE_IMAGES" | jq -r --arg name "$IMAGE_NAME" \
                  '.items[] | select(.metadata.name == $name) | .status.version')
                REFCOUNT=$(echo "$ENGINE_IMAGES" | jq -r --arg name "$IMAGE_NAME" \
                  '.items[] | select(.metadata.name == $name) | .status.refCount')

                echo "Deleting: $IMAGE_NAME (version: $VERSION, refCount: $REFCOUNT)"

                if kubectl delete engineimages.longhorn.io "$IMAGE_NAME" -n longhorn-system; then
                  COUNT=$((COUNT + 1))
                  echo "  ✓ Deleted successfully"
                else
                  FAILED=$((FAILED + 1))
                  echo "  ✗ Failed to delete"
                fi
              done

              echo ""
              echo "=== Cleanup Summary ==="
              echo "Latest version: v${LATEST_VERSION}"
              echo "Successfully deleted: $COUNT obsolete engine images"
              echo "Failed deletions: $FAILED engine images"

              if [ $FAILED -gt 0 ]; then
                exit 1
              fi

              echo ""
              echo "Cleanup completed successfully"

Apply the configuration

Save all four manifests to a single file (e.g. longhorn-engine-cleanup.yaml), then deploy them to the cluster:
```
kubectl apply -f longhorn-engine-cleanup.yaml
```

Verify the CronJob was created:

kubectl get cronjob -n longhorn-system longhorn-engineimage-cleanup

Expected output:

NAME                            SCHEDULE     SUSPEND   ACTIVE   LAST SCHEDULE   AGE
longhorn-engineimage-cleanup    0 2 * * 0    False     0        <none>          5s

60.4.5 Operational procedures #

This section describes the day-to-day operational tasks for managing the Longhorn engine image cleanup CronJob, including manual execution, schedule customization, monitoring, and verification.

60.4.5.1 Manual execution #

To execute cleanup immediately without waiting for the scheduled time, run the following command:

kubectl create job --from=cronjob/longhorn-engineimage-cleanup \
  longhorn-engineimage-cleanup-manual-$(date +%Y%m%d-%H%M%S) \
  -n longhorn-system

Monitor manual execution:

Watch job status:
```
kubectl get jobs -n longhorn-system -w
```

View job logs:

kubectl logs -n longhorn-system -l app=longhorn-engineimage-cleanup --tail=100

60.4.5.2 Schedule configuration #

The default schedule executes cleanup weekly on Sunday at 2:00 AM UTC:

schedule: "0 2 * * 0"

Common schedule alternatives:

Daily at 3:00 AM: "0 3 * * *"
Monthly on the 1st at 4:00 AM: "0 4 1 * *"
Every 6 hours: "0 */6 * * *"

Modify the schedule:

kubectl patch cronjob longhorn-engineimage-cleanup \
  -n longhorn-system \
  --type='merge' \
  -p '{"spec":{"schedule":"0 3 * * *"}}'

60.4.5.3 Monitoring and logs #

View cleanup history:

List completed jobs:

kubectl get jobs -n longhorn-system -l app=longhorn-engineimage-cleanup

View logs from last successful run:

kubectl logs -n longhorn-system \
  -l app=longhorn-engineimage-cleanup \
  --tail=200 \
  | grep -A 50 "Cleanup Summary"

Inspect specific job execution:

# Get job name
JOB_NAME=$(kubectl get jobs -n longhorn-system \
  -l app=longhorn-engineimage-cleanup \
  --sort-by=.metadata.creationTimestamp \
  -o jsonpath='{.items[-1].metadata.name}')

# View job details
kubectl describe job $JOB_NAME -n longhorn-system

# View job logs
kubectl logs job/$JOB_NAME -n longhorn-system

60.4.5.4 Verification #

After cleanup execution, verify that only current engine images remain:

List all engine images:

kubectl get engineimages.longhorn.io -n longhorn-system

Check reference counts:

kubectl get engineimages.longhorn.io -n longhorn-system \
  -o custom-columns=NAME:.metadata.name,VERSION:.status.version,REFCOUNT:.status.refCount

Expected output showing only the latest version with active references:

NAME                                VERSION     REFCOUNT
ei-acba2c08                         v1.11.1     3