19 System Upgrade Controller #

See the System Upgrade Controller documentation.

The System Upgrade Controller (SUC) aims to provide a general-purpose, Kubernetes-native upgrade controller (for nodes). It introduces a new CRD, the Plan, for defining any and all of your upgrade policies/requirements. A Plan is an outstanding intent to mutate nodes in your cluster.

19.1 How does SUSE Edge use System Upgrade Controller? #

SUC is used to assist in the various Day 2 operations that need to be executed in order to upgrade management/downstream clusters from one Edge platform version to another. Day 2 operations are defined in the form of SUC Plans. Based on the these plans, SUC deploys workloads on each node that executes the respective Day 2 operations.

19.2 Installing the System Upgrade Controller #

We recommend that you install SUC through Fleet (Chapter 6, Fleet) located in the suse-edge/fleet-examples repository.

Note

The resources offered by the suse-edge/fleet-examples repository must always be used from a valid fleet-examples release. To determine which release you need to use, refer to the Release Notes (Section 36.1, “Abstract”).

If you are unable to use Fleet (Chapter 6, Fleet) for the installation of SUC, you can install it through Rancher’s Helm chart repository, or incorporate the Rancher’s Helm chart in your own third-party GitOps workflow.

This section covers:

Fleet installation (Section 19.2.1, “System Upgrade Controller Fleet installation”)
Helm installation (Section 19.2.2, “System Upgrade Controller Helm installation”)

19.2.1 System Upgrade Controller Fleet installation #

Using Fleet there are two possible resources that can be used to deploy SUC:

GitRepo resource - for use-cases where an external/local Git server is available. For installation instructions, see System Upgrade Controller installation - GitRepo (Section 19.2.1.1, “System Upgrade Controller installation - GitRepo”).
Bundle resource - for air-gapped use-cases that do not support a local Git server option. For installation instructions, see System Upgrade Controller installation - Bundle (Section 19.2.1.2, “System Upgrade Controller installation - Bundle”).

19.2.1.1 System Upgrade Controller installation - GitRepo #

Note

This process can also be done through the Rancher UI, if such is available. For more information, see Accessing Fleet in the Rancher UI.

In your management cluster:

Determine on which clusters you want to deploy SUC. This is done by deploying the SUC GitRepo in the correct Fleet workspace inside your management cluster. By default, Fleet has two workspaces:
- fleet-local - for resources that need to be deployed on the management cluster.
- fleet-default - for resources that need to be deployed on downstream clusters.
  For more information on Fleet workspaces, see the upstream documentation.

Deploy the GitRepo resource:

To deploy SUC on your management cluster:

kubectl apply -n fleet-local -f - <<EOF
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: system-upgrade-controller
spec:
  revision: release-3.1.0
  paths:
  - fleets/day2/system-upgrade-controller
  repo: https://github.com/suse-edge/fleet-examples.git
EOF

To deploy SUC on your downstream clusters:

Note

Before deploying the resource below, you must provide a valid targets configuration, so that Fleet knows on which downstream clusters to deploy your resource. For information on how to map to downstream clusters, see Mapping to Downstream Clusters.

kubectl apply -n fleet-default -f - <<EOF
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: system-upgrade-controller
spec:
  revision: release-3.1.0
  paths:
  - fleets/day2/system-upgrade-controller
  repo: https://github.com/suse-edge/fleet-examples.git
  targets:
  - clusterSelector: CHANGEME
  # Example matching all clusters:
  # targets:
  # - clusterSelector: {}
EOF

Validate that the GitRepo is deployed:

# Namespace will vary based on where you want to deploy SUC
kubectl get gitrepo system-upgrade-controller -n <fleet-local/fleet-default>

NAME                        REPO                                              COMMIT          BUNDLEDEPLOYMENTS-READY   STATUS
system-upgrade-controller   https://github.com/suse-edge/fleet-examples.git   release-3.1.0   1/1

Validate the System Upgrade Controller deployment:

kubectl get deployment system-upgrade-controller -n cattle-system
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
system-upgrade-controller   1/1     1            1           2m20s

19.2.1.2 System Upgrade Controller installation - Bundle #

This section illustrates how to build and deploy a Bundle resource from a standard Fleet configuration using the fleet-cli.

On a machine with network access download the fleet-cli:
Note
Make sure that the version of the fleet-cli you download matches the version of Fleet that has been deployed on your cluster.
- For Mac users there is a fleet-cli Homebrew Formulae.
- For Linux and Windows users the binaries are present as assets to each Fleet release.
  - Linux AMD:
    curl -L -o fleet-cli https://github.com/rancher/fleet/releases/download/<FLEET_VERSION>/fleet-linux-amd64
  - Linux ARM:
    curl -L -o fleet-cli https://github.com/rancher/fleet/releases/download/<FLEET_VERSION>/fleet-linux-arm64
Make fleet-cli executable:
```
chmod +x fleet-cli
```

Clone the suse-edge/fleet-examples release that you wish to use:

git clone -b release-3.1.0 https://github.com/suse-edge/fleet-examples.git

Navigate to the SUC fleet, located in the fleet-examples repo:
```
cd fleet-examples/fleets/day2/system-upgrade-controller
```
Determine on which clusters you want to deploy SUC. This is done by deploying the SUC Bundle in the correct Fleet workspace inside your management cluster. By default, Fleet has two workspaces:
- fleet-local - for resources that need to be deployed on the management cluster.
- fleet-default - for resources that need to be deployed on downstream clusters.
  For more information on Fleet workspaces, see the upstream documentation.
If you intend to deploy SUC only on downstream clusters, create a targets.yaml file that matches the specific clusters:
```
cat > targets.yaml <<EOF
targets:
- clusterSelector: CHANGEME
EOF
```
For information on how to map to downstream clusters, see Mapping to Downstream Clusters
Proceed to building the Bundle:
Note
Make sure you did not download the fleet-cli in the fleet-examples/fleets/day2/system-upgrade-controller directory, otherwise it will be packaged with the Bundle, which is not advised.
- To deploy SUC on your management cluster, execute:
```
fleet-cli apply --compress -n fleet-local -o - system-upgrade-controller . > system-upgrade-controller-bundle.yaml
```
- To deploy SUC on your downstream clusters, execute:
```
fleet-cli apply --compress --targets-file=targets.yaml -n fleet-default -o - system-upgrade-controller . > system-upgrade-controller-bundle.yaml
```
  For more information about this process, see Convert a Helm Chart into a Bundle.
  For more information about the fleet-cli apply command, see fleet apply.
Transfer the system-upgrade-controller-bundle.yaml bundle to your management cluster machine:
```
scp system-upgrade-controller-bundle.yaml <machine-address>:<filesystem-path>
```
On your management cluster, deploy the system-upgrade-controller-bundle.yaml Bundle:
```
kubectl apply -f system-upgrade-controller-bundle.yaml
```

On your management cluster, validate that the Bundle is deployed:

# Namespace will vary based on where you want to deploy SUC
kubectl get bundle system-upgrade-controller -n <fleet-local/fleet-default>

NAME                        BUNDLEDEPLOYMENTS-READY   STATUS
system-upgrade-controller   1/1

Based on the Fleet workspace that you deployed your Bundle to, navigate to the cluster and validate the SUC deployment:

Note

SUC is always deployed in the cattle-system namespace.

kubectl get deployment system-upgrade-controller -n cattle-system
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
system-upgrade-controller   1/1     1            1           111s

19.2.2 System Upgrade Controller Helm installation #

Add the Rancher chart repository:

helm repo add rancher-charts https://charts.rancher.io/

Deploy the SUC chart:

helm install system-upgrade-controller rancher-charts/system-upgrade-controller --version 104.0.0+up0.7.0 --set global.cattle.psp.enabled=false -n cattle-system --create-namespace

This will install SUC 0.13.4 version which is needed by the Edge 3.1 platform.

Validate the SUC deployment:

kubectl get deployment system-upgrade-controller -n cattle-system
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
system-upgrade-controller   1/1     1            1           37s

19.3 Monitoring System Upgrade Controller Plans #

SUC Plans can be viewed in the following ways:

Through the Rancher UI (Section 19.3.1, “Monitoring System Upgrade Controller Plans - Rancher UI”).
Through manual monitoring (Section 19.3.2, “Monitoring System Upgrade Controller Plans - Manual”) inside of the cluster.

Important

Pods deployed for SUC Plans are kept alive 15 minutes after a successful execution. After that they are removed by the corresponding Job that created them. To have access to the Pod’s logs after this time period, you should enable logging for your cluster. For information on how to do this in Rancher, see Rancher Integration with Logging Services.

19.3.1 Monitoring System Upgrade Controller Plans - Rancher UI #

To check Pod logs for the specific SUC plan:

In the upper left corner, ☰ → <your-cluster-name>
Select Workloads → Pods
Select the Only User Namespaces drop down menu and add the cattle-system namespace
In the Pod filter bar, write the name for your SUC Plan Pod. The name will be in the following template format: apply-<plan_name>-on-<node_name>
Note
There may be both Completed and Unknown Pods for a specific SUC Plan. This is expected and happens due to the nature of some of the upgrades.
Select the pod that you want to review the logs of and navigate to ⋮ → View Logs

19.3.2 Monitoring System Upgrade Controller Plans - Manual #

Note

The below steps assume that kubectl has been configured to connect to the cluster where the SUC Plans have been deployed to.

List deployed SUC Plans:
```
kubectl get plans -n cattle-system
```
Get Pod for SUC Plan:
```
kubectl get pods -l upgrade.cattle.io/plan=<plan_name> -n cattle-system
```
Note
There may be both Completed and Unknown Pods for a specific SUC Plan. This is expected and happens due to the nature of some of the upgrades.

Get logs for the Pod:

kubectl logs <pod_name> -n cattle-system