1 BMC automated deployments with Metal³ #

Metal³ is a CNCF project which provides bare-metal infrastructure management capabilities for Kubernetes.

Metal³ provides Kubernetes-native resources to manage the lifecycle of bare-metal servers which support management via out-of-band protocols such as Redfish.

It also has mature support for Cluster API (CAPI) which enables management of infrastructure resources across multiple infrastructure providers via broadly adopted vendor-neutral APIs.

1.1 Why use this method #

This method is useful for scenarios where the target hardware supports out-of-band management, and a fully automated infrastructure management flow is desired.

A management cluster is configured to provide declarative APIs that enable inventory and state management of downstream cluster bare-metal servers, including automated inspection, cleaning and provisioning/deprovisioning.

1.2 High-level architecture #

1.3 Prerequisites #

There are some specific constraints related to the downstream cluster server hardware and networking:

Management cluster
- Must have network connectivity to the target server management/BMC API
- Must have network connectivity to the target server control plane network
- For multi-node management clusters, an additional reserved IP address is required
Hosts to be controlled
- Must support out-of-band management via Redfish, iDRAC or iLO interfaces
- Must support deployment via virtual media (PXE is not currently supported)
- Must have network connectivity to the management cluster for access to the Metal³ provisioning APIs

Some tools are required, these can be installed either on the management cluster, or on a host which can access it.

Kubectl, Helm and Clusterctl
A container runtime such as Podman or Rancher Desktop

The SLE-Micro.x86_64-5.5.0-Default-GM.raw.xz OS image file must be downloaded from the SUSE Customer Center or the SUSE Download page.

1.3.1 Setup Management Cluster #

The basic steps to install a management cluster and use Metal³ are:

Install an RKE2 management cluster
Install Rancher
Install a storage provider
Install the Metal³ dependencies
Install CAPI dependencies
Build a SLEMicro OS image for downstream cluster hosts
Register BareMetalHost CRs to define the bare-metal inventory
Create a downstream cluster by defining CAPI resources

This guide assumes an existing RKE2 cluster and Rancher (including cert-manager) has been installed, for example by using Edge Image Builder (Chapter 9, Edge Image Builder).

Tip

The steps here can also be fully automated as described in the ATIP management cluster documentation (Chapter 29, Setting up the management cluster).

1.3.2 Installing Metal³ dependencies #

If not already installed as part of the Rancher installation, cert-manager must be installed and running.

A persistent storage provider must be installed. Longhorn is recommended but local-path can also be used for dev/PoC environments. The instructions below assume a StorageClass has been marked as default, otherwise additional configuration for the Metal³ chart is required.

An additional IP is required, which is managed by MetalLB to provide a consistent endpoint for the Metal³ management services. This IP must be part of the control plane subnet and reserved for static configuration (not part of any DHCP pool).

Tip

If the management cluster is a single node, the requirement for an additional floating IP managed via MetalLB can be avoided, see Single-node configuration (Section 1.6.1, “Single-node configuration”)

First, we install MetalLB:

helm install \
  metallb oci://registry.suse.com/edge/metallb-chart \
  --namespace metallb-system \
  --create-namespace

Then we define an IPAddressPool and L2Advertisment using the reserved IP, defined as STATIC_IRONIC_IP below:

export STATIC_IRONIC_IP=<STATIC_IRONIC_IP>

cat <<-EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ironic-ip-pool
  namespace: metallb-system
spec:
  addresses:
  - ${STATIC_IRONIC_IP}/32
  serviceAllocation:
    priority: 100
    serviceSelectors:
    - matchExpressions:
      - {key: app.kubernetes.io/name, operator: In, values: [metal3-ironic]}
EOF

cat <<-EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ironic-ip-pool-l2-adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - ironic-ip-pool
EOF

Now Metal³ can be installed:

helm install \
  metal3 oci://registry.suse.com/edge/metal3-chart \
  --namespace metal3-system \
  --create-namespace \
  --set global.ironicIP="${STATIC_IRONIC_IP}"

It can take around two minutes for the initContainer to run on this deployment, so ensure the pods are all running before proceeding:

kubectl get pods -n metal3-system
NAME                                                    READY   STATUS    RESTARTS   AGE
baremetal-operator-controller-manager-85756794b-fz98d   2/2     Running   0          15m
metal3-metal3-ironic-677bc5c8cc-55shd                   4/4     Running   0          15m
metal3-metal3-mariadb-7c7d6fdbd8-64c7l                  1/1     Running   0          15m

Warning

Do not proceed to the following steps until all pods in the metal3-system namespace are running

1.3.3 Installing cluster API dependencies #

First, we need to disable the Rancher-embedded CAPI controller:

cat <<-EOF | kubectl apply -f -
apiVersion: management.cattle.io/v3
kind: Feature
metadata:
  name: embedded-cluster-api
spec:
  value: false
EOF

kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration
kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration
kubectl wait --for=delete namespace/cattle-provisioning-capi-system --timeout=300s

Then, to use the SUSE images, a configuration file is needed:

mkdir ~/.cluster-api
cat >  ~/.cluster-api/clusterctl.yaml <<EOF
images:
  all:
    repository: registry.suse.com/edge
EOF

Install clusterctl 1.6.x, after which we will install the core, infrastructure, bootstrap and control plane providers as follows:

clusterctl init --core "cluster-api:v1.6.2" --infrastructure "metal3:v1.6.0" --bootstrap "rke2:v0.2.6" --control-plane "rke2:v0.2.6"

After some time, the controller pods should be running in the capi-system, capm3-system, rke2-bootstrap-system and rke2-control-plane-system namespaces.

1.3.4 Prepare downstream cluster image #

Edge Image Builder (Chapter 9, Edge Image Builder) is used to prepare a modified SLEMicro base image which is provisioned on downstream cluster hosts.

Much of the configuration via Edge Image Builder is possible, but in this guide, we cover the minimal configurations necessary to set up the downstream cluster.

1.3.4.1 Image configuration #

When running Edge Image Builder, a directory is mounted from the host, so it is necessary to create a directory structure to to store the configuration files used to define the target image.

downstream-cluster-config.yaml is the image definition file, see Chapter 3, Standalone clusters with Edge Image Builder for more details.
The base image when downloaded is xz compressed, which must be uncompressed with unxz and copied/moved under the base-images folder.
The network folder is optional, see Section 1.3.5.1.1, “Additional script for static network configuration” for more details.
The custom/scripts directory contains scripts to be run on first-boot; currently a growfs.sh script is required to resize the OS root partition on deployment

├── downstream-cluster-config.yaml
├── base-images/
│   └ SLE-Micro.x86_64-5.5.0-Default-GM.raw
├── network/
|   └ configure-network.sh
└── custom/
    └ scripts/
        └ growfs.sh

1.3.4.1.1 Downstream cluster image definition file #

The downstream-cluster-config.yaml file is the main configuration file for the downstream cluster image. The following is a minimal example for deployment via Metal³:

apiVersion: 1.0
image:
  imageType: RAW
  arch: x86_64
  baseImage: SLE-Micro.x86_64-5.5.0-Default-GM.raw
  outputImageName: SLE-Micro-eib-output.raw
operatingSystem:
  kernelArgs:
    - ignition.platform.id=openstack
    - net.ifnames=1
  systemd:
    disable:
      - rebootmgr
  users:
    - username: root
      encryptedPassword: ${ROOT_PASSWORD}
      sshKeys:
      - ${USERKEY1}

${ROOT_PASSWORD} is the encrypted password for the root user, which can be useful for test/debugging. It can be generated with the openssl passwd -6 PASSWORD command

For the production environments, it is recommended to use the SSH keys that can be added to the users block replacing the ${USERKEY1} with the real SSH keys.

Note

net.ifnames=1 enables Predictable Network Interface Naming

This matches the default configuration for the metal3 chart, but the setting must match the configured chart predictableNicNames value.

Also note ignition.platform.id=openstack is mandatory, without this argument SLEMicro configuration via ignition will fail in the Metal³ automated flow.

1.3.4.1.2 Growfs script #

Currently is a custom script (custom/scripts/growfs.sh) which is required to grow the file system to the match the disk size on first-boot after provisioning. The growfs.sh script contains the following information:

#!/bin/bash
growfs() {
  mnt="$1"
  dev="$(findmnt --fstab --target ${mnt} --evaluate --real --output SOURCE --noheadings)"
  # /dev/sda3 -> /dev/sda, /dev/nvme0n1p3 -> /dev/nvme0n1
  parent_dev="/dev/$(lsblk --nodeps -rno PKNAME "${dev}")"
  # Last number in the device name: /dev/nvme0n1p42 -> 42
  partnum="$(echo "${dev}" | sed 's/^.*[^0-9]\([0-9]\+\)$/\1/')"
  ret=0
  growpart "$parent_dev" "$partnum" || ret=$?
  [ $ret -eq 0 ] || [ $ret -eq 1 ] || exit 1
  /usr/lib/systemd/systemd-growfs "$mnt"
}
growfs /

Note

Add your own custom scripts to be executed during the provisioning process using the same approach. For more information, see Chapter 3, Standalone clusters with Edge Image Builder.

The bug related to this workaround is https://bugzilla.suse.com/show_bug.cgi?id=1217430

1.3.4.2 Image creation #

Once the directory structure is prepared following the previous sections, run the following command to build the image:

podman run --rm --privileged -it -v $PWD:/eib \
 registry.suse.com/edge/edge-image-builder:1.0.2 \
 build --definition-file downstream-cluster-config.yaml

This creates the output image file named SLE-Micro-eib-output.raw, based on the definition described above.

The output image must then be made available via a webserver, either the media-server container enabled via the Metal3 chart (Note) or some other locally accessible server. In the examples below, we refer to this server as imagecache.local:8080

1.3.5 Adding BareMetalHost inventory #

Registering bare-metal servers for automated deployment requires creating two resources: a Secret storing BMC access credentials and a Metal³ BareMetalHost resource defining the BMC connection and other details:

apiVersion: v1
kind: Secret
metadata:
  name: controlplane-0-credentials
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQ=
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: controlplane-0
  labels:
    cluster-role: control-plane
spec:
  online: true
  bootMACAddress: "00:f3:65:8a:a3:b0"
  bmc:
    address: redfish-virtualmedia://192.168.125.1:8000/redfish/v1/Systems/68bd0fb6-d124-4d17-a904-cdf33efe83ab
    disableCertificateVerification: true
    credentialsName: controlplane-0-credentials

Note the following:

The Secret username/password must be base64 encoded. Note this should not include any trailing newlines (for example, use echo -n, not just echo!)
The cluster-role label may be set now or later on cluster creation. In the example below, we expect control-plane or worker
bootMACAddress must be a valid MAC that matches the control plane NIC of the host
The bmc address is the connection to the BMC management API, the following are supported:
- redfish-virtualmedia://<IP ADDRESS>/redfish/v1/Systems/<SYSTEM ID>: Redfish virtual media, for example, SuperMicro
- idrac-virtualmedia://<IP ADDRESS>/redfish/v1/Systems/System.Embedded.1: Dell iDRAC
See the Upstream API docs for more details on the BareMetalHost API

1.3.5.1 Configuring Static IPs #

The BareMetalHost example above assumes DHCP provides the controlplane network configuration, but for scenarios where manual configuration is needed such as static IPs it is possible to provide additional configuration, as described below.

1.3.5.1.1 Additional script for static network configuration #

When creating the base image with Edge Image Builder, in the network folder, create the following configure-network.sh file.

This consumes configuration drive data on first-boot, and configures the host networking using the NM Configurator tool.

#!/bin/bash

set -eux

# Attempt to statically configure a NIC in the case where we find a network_data.json
# In a configuration drive

CONFIG_DRIVE=$(blkid --label config-2 || true)
if [ -z "${CONFIG_DRIVE}" ]; then
  echo "No config-2 device found, skipping network configuration"
  exit 0
fi

mount -o ro $CONFIG_DRIVE /mnt

NETWORK_DATA_FILE="/mnt/openstack/latest/network_data.json"

if [ ! -f "${NETWORK_DATA_FILE}" ]; then
  umount /mnt
  echo "No network_data.json found, skipping network configuration"
  exit 0
fi

DESIRED_HOSTNAME=$(cat /mnt/openstack/latest/meta_data.json | tr ',{}' '\n' | grep '\"metal3-name\"' | sed 's/.*\"metal3-name\": \"\(.*\)\"/\1/')

mkdir -p /tmp/nmc/{desired,generated}
cp ${NETWORK_DATA_FILE} /tmp/nmc/desired/${DESIRED_HOSTNAME}.yaml
umount /mnt

./nmc generate --config-dir /tmp/nmc/desired --output-dir /tmp/nmc/generated
./nmc apply --config-dir /tmp/nmc/generated

1.3.5.1.2 Additional secret with host network configuration #

An additional secret containing data in the nmstate format supported by NM Configurator (Chapter 10, Edge Networking) can be defined for each host.

The secret is then referenced in the BareMetalHost resource via the preprovisioningNetworkDataName spec field.

apiVersion: v1
kind: Secret
metadata:
  name: controlplane-0-networkdata
type: Opaque
stringData:
  networkData: |
    interfaces:
    - name: enp1s0
      type: ethernet
      state: up
      mac-address: "00:f3:65:8a:a3:b0"
      ipv4:
        address:
        - ip:  192.168.125.200
          prefix-length: 24
        enabled: true
        dhcp: false
    dns-resolver:
      config:
        server:
        - 192.168.125.1
    routes:
      config:
      - destination: 0.0.0.0/0
        next-hop-address: 192.168.125.1
        next-hop-interface: enp1s0
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: controlplane-0
  labels:
    cluster-role: control-plane
spec:
  preprovisioningNetworkDataName: controlplane-0-networkdata
# Remaining content as in previous example

Note

although optional in the nmstate API, the mac-address is mandatory for configuration via NM Configurator and must be provided.

1.3.5.2 BareMetalHost preparation #

After creating the BareMetalHost resource and associated secrets as described above, a host preparation workflow is triggered:

A ramdisk image is booted by virtualmedia attachment to the target host BMC
The ramdisk inspects hardware details, and prepares the host for provisioning (for example by cleaning disks of previous data)
On completion of this process, hardware details in the BareMetalHost status.hardware field are updated and can be verified

This process can take several minutes, but when completed you should see the BareMetalHost state become available:

% kubectl get baremetalhost
NAME             STATE       CONSUMER   ONLINE   ERROR   AGE
controlplane-0   available              true             9m44s
worker-0         available              true             9m44s

1.3.6 Creating downstream clusters #

We now create Cluster API resources which define the downstream cluster, and Machine resources which will cause the BareMetalHost resources to be provisioned, then bootstrapped to form an RKE2 cluster.

1.3.7 Control plane deployment #

To deploy the controlplane we define a yaml manifest similar to the one below, which contains the following resources:

Cluster resource defines the cluster name, networks, and type of controlplane/infrastructure provider (in this case RKE2/Metal3)
Metal3Cluster defines the controlplane endpoint (host IP for single-node, LoadBalancer endpoint for multi-node, this example assumes single-node)
RKE2ControlPlane defines the RKE2 version and any additional configuration needed during cluster bootstrapping
Metal3MachineTemplate defines the OS Image to be applied to the BareMetalHost resources, and the hostSelector defines which BareMetalHosts to consume
Metal3DataTemplate defines additional metaData to be passed to the BareMetalHost (note networkData is not currently supported in the Edge solution)

Note for simplicity this example assumes a single-node controlplane, where the BareMetalHost is configured with an IP of 192.168.125.200 - for more advanced multi-node examples please see the ATIP documentation (Chapter 31, Fully automated directed network provisioning)

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: sample-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 192.168.0.0/18
    services:
      cidrBlocks:
        - 10.96.0.0/12
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: RKE2ControlPlane
    name: sample-cluster
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: Metal3Cluster
    name: sample-cluster
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3Cluster
metadata:
  name: sample-cluster
  namespace: default
spec:
  controlPlaneEndpoint:
    host: 192.168.125.200
    port: 6443
  noCloudProvider: true
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: RKE2ControlPlane
metadata:
  name: sample-cluster
  namespace: default
spec:
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: Metal3MachineTemplate
    name: sample-cluster-controlplane
  replicas: 1
  agentConfig:
    format: ignition
    kubelet:
      extraArgs:
        - provider-id=metal3://BAREMETALHOST_UUID
    additionalUserData:
      config: |
        variant: fcos
        version: 1.4.0
        systemd:
          units:
            - name: rke2-preinstall.service
              enabled: true
              contents: |
                [Unit]
                Description=rke2-preinstall
                Wants=network-online.target
                Before=rke2-install.service
                ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
                [Service]
                Type=oneshot
                User=root
                ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
                ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
                ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
                ExecStartPost=/bin/sh -c "umount /mnt"
                [Install]
                WantedBy=multi-user.target
    version: v1.28.9+rke2r1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: sample-cluster-controlplane
  namespace: default
spec:
  template:
    spec:
      dataTemplate:
        name: sample-cluster-controlplane-template
      hostSelector:
        matchLabels:
          cluster-role: control-plane
      image:
        checksum: http://imagecache.local:8080/SLE-Micro-eib-output.raw.sha256
        checksumType: sha256
        format: raw
        url: http://imagecache.local:8080/SLE-Micro-eib-output.raw
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: sample-cluster-controlplane-template
  namespace: default
spec:
  clusterName: sample-cluster
  metaData:
    objectNames:
      - key: name
        object: machine
      - key: local-hostname
        object: machine
      - key: local_hostname
        object: machine

When the example above has been copied and adapted to suit your environment, it can be applied via kubectl then the cluster status can be monitored with clusterctl

% kubectl apply -f rke2-control-plane.yaml

# Wait for the cluster to be provisioned - status can be checked via clusterctl
% clusterctl describe cluster sample-cluster
NAME                                                    READY  SEVERITY  REASON  SINCE  MESSAGE
Cluster/sample-cluster                                  True                     22m
├─ClusterInfrastructure - Metal3Cluster/sample-cluster  True                     27m
├─ControlPlane - RKE2ControlPlane/sample-cluster        True                     22m
│ └─Machine/sample-cluster-chflc                        True                     23m

1.3.8 Worker/Compute deployment #

Similar to the controlplane we define a yaml manifest, which contains the following resources:

MachineDeployment defines the number of replicas (hosts) and the bootstrap/infrastructure provider (in this case RKE2/Metal3)
RKE2ConfigTemplate describes the RKE2 version and first-boot configuration for agent host bootstrapping
Metal3MachineTemplate defines the OS Image to be applied to the BareMetalHost resources, and the hostSelector defines which BareMetalHosts to consume
Metal3DataTemplate defines additional metaData to be passed to the BareMetalHost (note networkData is not currently supported in the Edge solution)

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: sample-cluster
  name: sample-cluster
  namespace: default
spec:
  clusterName: sample-cluster
  replicas: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: sample-cluster
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: sample-cluster
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
          kind: RKE2ConfigTemplate
          name: sample-cluster-workers
      clusterName: sample-cluster
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: Metal3MachineTemplate
        name: sample-cluster-workers
      nodeDrainTimeout: 0s
      version: v1.28.9+rke2r1
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: RKE2ConfigTemplate
metadata:
  name: sample-cluster-workers
  namespace: default
spec:
  template:
    spec:
      agentConfig:
        format: ignition
        version: v1.28.9+rke2r1
        kubelet:
          extraArgs:
            - provider-id=metal3://BAREMETALHOST_UUID
        additionalUserData:
          config: |
            variant: fcos
            version: 1.4.0
            systemd:
              units:
                - name: rke2-preinstall.service
                  enabled: true
                  contents: |
                    [Unit]
                    Description=rke2-preinstall
                    Wants=network-online.target
                    Before=rke2-install.service
                    ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
                    [Service]
                    Type=oneshot
                    User=root
                    ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
                    ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
                    ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
                    ExecStartPost=/bin/sh -c "umount /mnt"
                    [Install]
                    WantedBy=multi-user.target
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: sample-cluster-workers
  namespace: default
spec:
  template:
    spec:
      dataTemplate:
        name: sample-cluster-workers-template
      hostSelector:
        matchLabels:
          cluster-role: worker
      image:
        checksum: http://imagecache.local:8080/SLE-Micro-eib-output.raw.sha256
        checksumType: sha256
        format: raw
        url: http://imagecache.local:8080/SLE-Micro-eib-output.raw
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: sample-cluster-workers-template
  namespace: default
spec:
  clusterName: sample-cluster
  metaData:
    objectNames:
      - key: name
        object: machine
      - key: local-hostname
        object: machine
      - key: local_hostname
        object: machine

When the example above has been copied and adapted to suit your environment, it can be applied via kubectl then the cluster status can be monitored with clusterctl

% kubectl apply -f rke2-agent.yaml

# Wait some time for the compute/agent hosts to be provisioned
% clusterctl describe cluster sample-cluster
NAME                                                    READY  SEVERITY  REASON  SINCE  MESSAGE
Cluster/sample-cluster                                  True                     25m
├─ClusterInfrastructure - Metal3Cluster/sample-cluster  True                     30m
├─ControlPlane - RKE2ControlPlane/sample-cluster        True                     25m
│ └─Machine/sample-cluster-chflc                        True                     27m
└─Workers
  └─MachineDeployment/sample-cluster                    True                     22m
    └─Machine/sample-cluster-56df5b4499-zfljj           True                     23m

1.3.9 Cluster deprovisioning #

The downstream cluster may be deprovisioned by deleting the resources applied in the creation steps above:

% kubectl delete -f rke2-agent.yaml
% kubectl delete -f rke2-control-plane.yaml

This triggers deprovisioning of the BareMetalHost resources, which may take several minutes, after which they should be in available state again:

% kubectl get bmh
NAME             STATE            CONSUMER                            ONLINE   ERROR   AGE
controlplane-0   deprovisioning   sample-cluster-controlplane-vlrt6   false            10m
worker-0         deprovisioning   sample-cluster-workers-785x5        false            10m

...

% kubectl get bmh
NAME             STATE       CONSUMER   ONLINE   ERROR   AGE
controlplane-0   available              false            15m
worker-0         available              false            15m

1.4 Known issues #

The upstream IP Address Management controller is currently not supported, because it’s not yet compatible with our choice of network configuration tooling and first-boot toolchain in SLEMicro.
Relatedly, the IPAM resources and Metal3DataTemplate networkData fields are not currently supported.
Only deployment via redfish-virtualmedia is currently supported.
Deployed clusters are not currently imported into Rancher
Due to disabling the Rancher embedded CAPI controller, a management cluster configured for Metal³ as described above cannot also be used for other cluster provisioning methods such as Elemental (Chapter 11, Elemental)

1.5 Planned changes #

Deployed clusters imported into Rancher, this is planned via Rancher Turtles in future
Aligning with Rancher Turtles is also expected to remove the requirement to disable the Rancher embedded CAPI, so other cluster methods should be possible via the management cluster.
Enable support of the IPAM resources and configuration via networkData fields

1.6 Additional resources #

The ATIP Documentation (Chapter 26, SUSE Adaptive Telco Infrastructure Platform (ATIP)) has examples of more advanced usage of Metal³ for telco use-cases.

1.6.1 Single-node configuration #

For test/PoC environments where the management cluster is a single node, it is possible to avoid the requirement for an additional floating IP managed via MetalLB.

In this mode, the endpoint for the management cluster APIs is the IP of the management cluster, therefore it should be reserved when using DHCP or statically configured to ensure the management cluster IP does not change - referred to as <MANAGEMENT_CLUSTER_IP> below.

To enable this scenario the metal3 chart values required are as follows:

global:
  ironicIP: <MANAGEMENT_CLUSTER_IP>
metal3-ironic:
  service:
    type: NodePort

1.6.2 Disabling TLS for virtualmedia ISO attachment #

Some server vendors verify the SSL connection when attaching virtual-media ISO images to the BMC, which can cause a problem because the generated certificates for the Metal3 deployment are self-signed, to work around this issue it’s possible to disable TLS only for the virtualmedia disk attachment with metal3 chart values as follows:

global:
  enable_vmedia_tls: false

An alternative solution is to configure the BMCs with the CA cert - in this case you can read the certificates from the cluster using kubectl:

kubectl get secret -n metal3-system ironic-vmedia-cert -o yaml

The certificate can then be configured on the server BMC console, although the process for that is vendor specific (and not possible for all vendors, in which case the enable_vmedia_tls flag may be required).

1 BMC automated deployments with Metal3 #

1.1 Why use this method #

1.2 High-level architecture #

1.3 Prerequisites #

1.3.1 Setup Management Cluster #

1.3.2 Installing Metal3 dependencies #

1.3.3 Installing cluster API dependencies #

1.3.4 Prepare downstream cluster image #

1.3.4.1 Image configuration #

1.3.4.1.1 Downstream cluster image definition file #

1.3.4.1.2 Growfs script #

1.3.4.2 Image creation #

1.3.5 Adding BareMetalHost inventory #

1.3.5.1 Configuring Static IPs #

1.3.5.1.1 Additional script for static network configuration #

1.3.5.1.2 Additional secret with host network configuration #

1.3.5.2 BareMetalHost preparation #

1.3.6 Creating downstream clusters #

1.3.7 Control plane deployment #

1.3.8 Worker/Compute deployment #

1.3.9 Cluster deprovisioning #

1.4 Known issues #

1.5 Planned changes #

1.6 Additional resources #

1.6.1 Single-node configuration #

1.6.2 Disabling TLS for virtualmedia ISO attachment #

1 BMC automated deployments with Metal³ #

1.3.2 Installing Metal³ dependencies #