Applies to SUSE Telco Cloud 3.6

55 Telco features (DPDK, SR-IOV, CPU isolation, huge pages, NUMA, etc.) #

The directed network provisioning workflow allows to automate the Telco features to be used in the downstream clusters to run Telco workloads on top of those servers.

Requirements

The image generated using EIB, as described in the previous section (Chapter 50, Prepare downstream cluster image for connected scenarios), has to be located in the management cluster exactly on the path you configured on this section (Note).
The image generated using EIB has to include the specific Telco packages following this section (Section 50.2.5, “Additional configuration for Telco workloads”).
The management server created and available to be used on the following sections. For more information, refer to the Management Cluster section: Part V, “Setting up the management cluster”.

Configuration

Use the following two sections as the base to enroll and provision the hosts:

Downstream cluster provisioning with Directed network provisioning (single-node) (Chapter 52, Downstream cluster provisioning with Directed network provisioning (single-node))
Downstream cluster provisioning with Directed network provisioning (multi-node) (Chapter 53, Downstream cluster provisioning with Directed network provisioning (multi-node))

The Telco features covered in this section are the following:

DPDK and VFs creation
SR-IOV and VFs allocation to be used by the workloads
CPU isolation and performance tuning
Huge pages configuration
Kernel parameters tuning

Note

For more information about the Telco features, see Part VI, “Telco features configuration”.

The changes required to enable the Telco features shown above are all inside the RKE2ControlPlane block in the provision file capi-provisioning-example.yaml. The rest of the information inside the file capi-provisioning-example.yaml is the same as the information provided in the provisioning section (Chapter 52, Downstream cluster provisioning with Directed network provisioning (single-node)).

To make the process clear, the changes required on that block (RKE2ControlPlane) to enable the Telco features are the following:

The ignition file /var/lib/rancher/rke2/server/manifests/configmap-sriov-custom-auto.yaml to be used to define the interfaces, drivers and the number of VFs to be created and exposed to the workloads.
- The values inside the config map sriov-custom-auto-config are the only values to be replaced with real values.
  - ${RESOURCE_NAME1} — The resource name to be used for the first PF interface (for example, sriov-resource-du1). It is added to the prefix rancher.io to be used as a label to be used by the workloads (for example, rancher.io/sriov-resource-du1).
  - ${SRIOV-NIC-NAME1} — The name of the first PF interface to be used (for example, eth0).
  - ${PF_NAME1} — The name of the first physical function PF to be used. Generate more complex filters using this (for example, eth0#2-5).
  - ${DRIVER_NAME1} — The driver name to be used for the first VF interface (for example, vfio-pci).
  - ${NUM_VFS1} — The number of VFs to be created for the first PF interface (for example, 8).
The /var/sriov-auto-filler.sh to be used as a translator between the high-level config map sriov-custom-auto-config and the sriovnetworknodepolicy which contains the low-level hardware information. This script has been created to abstract the user from the complexity to know in advance the hardware information. No changes are required in this file, but it should be present if we need to enable sr-iov and create VFs.
The kernel arguments to be used to enable the following features:

Parameter	Value	Description
isolcpus	domain,nohz,managed_irq,1-30,33-62	Isolate the cores 1-30 and 33-62.
skew_tick	1	Allows the kernel to skew the timer interrupts across the isolated CPUs.
nohz	on	Allows the kernel to run the timer tick on a single CPU when the system is idle.
nohz_full	1-30,33-62	kernel boot parameter is the current main interface to configure full dynticks along with CPU Isolation.
rcu_nocbs	1-30,33-62	Allows the kernel to run the RCU callbacks on a single CPU when the system is idle.
irqaffinity	0,31,32,63	Allows the kernel to run the interrupts on a single CPU when the system is idle.
idle	poll	Minimizes the latency of exiting the idle state.
iommu	pt	Allows to use vfio for the dpdk interfaces.
intel_iommu	on	Enables the use of vfio for VFs.
hugepagesz	1G	Allows to set the size of huge pages to 1 G.
hugepages	40	Number of huge pages defined before.
default_hugepagesz	1G	Default value to enable huge pages.
nowatchdog		Disables the watchdog.
nmi_watchdog	0	Disables the NMI watchdog.

The following systemd services are used to enable the following:
- rke2-preinstall.service to replace automatically the BAREMETALHOST_UUID and node-name during the provisioning process using the Ironic information.
- cpu-partitioning.service to enable the isolation cores of the CPU (for example, 1-30,33-62).
- performance-settings.service to enable the CPU performance tuning.
- sriov-custom-auto-vfs.service to install the sriov Helm chart, wait until custom resources are created and run the /var/sriov-auto-filler.sh to replace the values in the config map sriov-custom-auto-config and create the sriovnetworknodepolicy to be used by the workloads.
The ${RKE2_VERSION} is the version of RKE2 to be used replacing this value (for example, v1.35.3+rke2r3).

With all these changes mentioned, the RKE2ControlPlane block in the capi-provisioning-example.yaml will look like the following:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: RKE2ControlPlane
metadata:
  name: single-node-cluster
  namespace: default
spec:
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: Metal3MachineTemplate
    name: single-node-cluster-controlplane
  replicas: 1
  version: ${RKE2_VERSION}
  rolloutStrategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 0
  serverConfig:
    cni: calico
    cniMultusEnable: true
  agentConfig:
    format: ignition
    additionalUserData:
      config: |
        variant: fcos
        version: 1.4.0
        storage:
          files:
          - path: /var/lib/rancher/rke2/server/manifests/configmap-sriov-custom-auto.yaml
            overwrite: true
            contents:
              inline: |
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  name: sriov-custom-auto-config
                  namespace: kube-system
                data:
                  config.json: |
                    [
                        {
                          "resourceName": "${RESOURCE_NAME1}",
                          "interface": "${SRIOV-NIC-NAME1}",
                          "pfname": "${PF_NAME1}",
                          "driver": "${DRIVER_NAME1}",
                          "numVFsToCreate": ${NUM_VFS1}
                        },
                        {
                          "resourceName": "${RESOURCE_NAME2}",
                          "interface": "${SRIOV-NIC-NAME2}",
                          "pfname": "${PF_NAME2}",
                          "driver": "${DRIVER_NAME2}",
                          "numVFsToCreate": ${NUM_VFS2}
                        }
                    ]
            mode: 0644
            user:
              name: root
            group:
              name: root
          - path: /var/lib/rancher/rke2/server/manifests/sriov-crd.yaml
            overwrite: true
            contents:
              inline: |
                apiVersion: helm.cattle.io/v1
                kind: HelmChart
                metadata:
                  name: sriov-crd
                  namespace: kube-system
                spec:
                  chart: oci://registry.suse.com/edge/charts/sriov-crd
                  targetNamespace: sriov-network-operator
                  version: 306.0.4+up1.6.0
                  createNamespace: true
          - path: /var/lib/rancher/rke2/server/manifests/sriov-network-operator.yaml
            overwrite: true
            contents:
              inline: |
                apiVersion: helm.cattle.io/v1
                kind: HelmChart
                metadata:
                  name: sriov-network-operator
                  namespace: kube-system
                spec:
                  chart: oci://registry.suse.com/edge/charts/sriov-network-operator
                  targetNamespace: sriov-network-operator
                  version: 306.0.4+up1.6.0
                  createNamespace: true
        kernel_arguments:
          should_exist:
            - intel_iommu=on
            - iommu=pt
            - idle=poll
            - mce=off
            - hugepagesz=1G hugepages=40
            - hugepagesz=2M hugepages=0
            - default_hugepagesz=1G
            - irqaffinity=${NON-ISOLATED_CPU_CORES}
            - isolcpus=domain,nohz,managed_irq,${ISOLATED_CPU_CORES}
            - nohz_full=${ISOLATED_CPU_CORES}
            - rcu_nocbs=${ISOLATED_CPU_CORES}
            - rcu_nocb_poll
            - nosoftlockup
            - nowatchdog
            - nohz=on
            - nmi_watchdog=0
            - skew_tick=1
            - quiet
        systemd:
          units:
          - name: rke2-preinstall.service
            enabled: true
            contents: |
              [Unit]
              Description=rke2-preinstall
              Wants=network-online.target
              Before=rke2-install.service
              ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
              [Service]
              Type=oneshot
              User=root
              ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
              ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"node-label:\" >> /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"  - metal3.io/uuid=$(jq -r .uuid /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
              ExecStartPost=/bin/sh -c "umount /mnt"
              [Install]
              WantedBy=multi-user.target
          # rke2-traefik-deployment.service unit to be removed once "traefik" being the default ingress controller (starting with RKE2 v1.36)
          - name: rke2-traefik-deployment.service
            enabled: true
            contents: |
              [Unit]
              Description=rke2-traefik-deployment
              Wants=rke2-preinstall.service
              Before=rke2-install.service
              ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
              [Service]
              Type=oneshot
              User=root
              ExecStart=/bin/sh -c "echo \"ingress-controller: traefik\" >> /etc/rancher/rke2/config.yaml"
              [Install]
              WantedBy=multi-user.target
          - name: cpu-partitioning.service
            enabled: true
            contents: |
              [Unit]
              Description=cpu-partitioning
              Wants=network-online.target
              After=network.target network-online.target
              [Service]
              Type=oneshot
              User=root
              ExecStart=/bin/sh -c "echo isolated_cores=${ISOLATED_CPU_CORES} > /etc/tuned/cpu-partitioning-variables.conf"
              ExecStartPost=/bin/sh -c "tuned-adm profile cpu-partitioning"
              ExecStartPost=/bin/sh -c "systemctl enable tuned.service"
              [Install]
              WantedBy=multi-user.target
          - name: performance-settings.service
            enabled: true
            contents: |
              [Unit]
              Description=performance-settings
              Wants=network-online.target
              After=network.target network-online.target cpu-partitioning.service
              [Service]
              Type=oneshot
              User=root
              ExecStart=/bin/sh -c "/opt/performance-settings/performance-settings.sh"
              [Install]
              WantedBy=multi-user.target
          - name: sriov-custom-auto-vfs.service
            enabled: true
            contents: |
              [Unit]
              Description=SRIOV Custom Auto VF Creation
              Wants=network-online.target  rke2-server.target
              After=network.target network-online.target rke2-server.target
              [Service]
              User=root
              Type=forking
              TimeoutStartSec=900
              ExecStart=/bin/sh -c "while ! /var/lib/rancher/rke2/bin/kubectl --kubeconfig=/etc/rancher/rke2/rke2.yaml wait --for condition=ready nodes --all ; do sleep 2 ; done"
              ExecStartPost=/bin/sh -c "while [ $(/var/lib/rancher/rke2/bin/kubectl --kubeconfig=/etc/rancher/rke2/rke2.yaml get sriovnetworknodestates.sriovnetwork.openshift.io --ignore-not-found --no-headers -A | wc -l) -eq 0 ]; do sleep 1; done"
              ExecStartPost=/bin/sh -c "/opt/sriov/sriov-auto-filler.sh"
              RemainAfterExit=yes
              KillMode=process
              [Install]
              WantedBy=multi-user.target
    kubelet:
      extraArgs:
      - provider-id=metal3://BAREMETALHOST_UUID
    nodeName: "localhost.localdomain"

Once the file is created by joining the previous blocks, the following command must be executed in the management cluster to start provisioning the new downstream cluster using the Telco features:

$ kubectl apply -f capi-provisioning-example.yaml

55.1 Precision Time Protocol (PTP) #

PTP synchronization is enabled at provisioning time in two steps: enabling the daemons in the EIB image and deploying the configuration files through Cluster API. Two provisioning models are documented, depending on the target platform:

Standard scenarios (Section 55.1.1, “Standard scenarios (single ptp4l + phc2sys)”): a single ptp4l instance and, optionally, phc2sys. Covers Scenario 1 (Section 47.3, “Scenario 1 - Ordinary Clock (PTP-only)”) through Scenario 3 (Section 47.5, “Scenario 3 - Boundary Clock (PTP-only) with Multiport Single NIC including propagation to Ordinary Clock server”).
GNR-D multi-NIC scenarios (Section 55.1.2, “GNR-D multi-NIC scenarios (DPLL)”): several ptp4l instances, ts2phc and on-board DPLL setup, which require additional files and services. Covers Scenario 4 (Section 47.6, “Scenario 4 - Boundary Clock (PTP-only) Multi-NIC with internal timing module (DPLL GNR-D based)”) (and the SyncE-enhanced Scenario 5).

55.1.1 Standard scenarios (single ptp4l + phc2sys) #

Since only the configuration files change between these PTP scenarios, this section provides a single generic template followed by the specific values for each scenario.

Enable the daemons in the EIB image

The linuxptp package is already part of the Telco package list (see Additional configuration for Telco workloads (Section 50.2.5, “Additional configuration for Telco workloads”)). However, the ptp4l and phc2sys daemons are not enabled by default, so they must be added to the systemd.enable list of the same EIB image definition file:

operatingSystem:
  systemd:
    enable:
      - ptp4l
      - phc2sys
  packages:
    packageList:
      - linuxptp      # already present in the Telco package list
    sccRegistrationCode: $SCC_REGISTRATION_CODE

With the daemons enabled at image-build time, both services start automatically once their configuration files are present on the host.

Deploy the configuration files through Cluster API

The PTP configuration is deployed through a modified RKE2ControlPlane object that writes the ptp4l profile file and the two /etc/sysconfig files that control the ptp4l and phc2sys daemons. The rest of the object is the same as the one described in the provisioning section (Chapter 52, Downstream cluster provisioning with Directed network provisioning (single-node)); complete it with the other Cluster API objects and replace the placeholders with the values of the chosen scenario:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: RKE2ControlPlane
metadata:
  name: single-node-cluster
  namespace: default
spec:
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: Metal3MachineTemplate
    name: single-node-cluster-controlplane
  replicas: 1
  version: ${RKE2_VERSION}
  rolloutStrategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 0
  registrationMethod: "control-plane-endpoint"
  serverConfig:
    cni: canal
  agentConfig:
    format: ignition
    cisProfile: cis
    additionalUserData:
      config: |
        variant: fcos
        version: 1.4.0
        systemd:
          units:
            - name: rke2-preinstall.service
              enabled: true
              contents: |
                [Unit]
                Description=rke2-preinstall
                Wants=network-online.target
                Before=rke2-install.service
                ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
                [Service]
                Type=oneshot
                User=root
                ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
                ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
                ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
                ExecStart=/bin/sh -c "echo \"node-label:\" >> /etc/rancher/rke2/config.yaml"
                ExecStart=/bin/sh -c "echo \"  - metal3.io/uuid=$(jq -r .uuid /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
                ExecStartPost=/bin/sh -c "umount /mnt"
                [Install]
                WantedBy=multi-user.target
        storage:
          files:
            - path: ${PTP4L_CONF_PATH}
              overwrite: true
              contents:
                inline: |
                  ## Replace ${PTP4L_CONF_CONTENTS} with the ptp4l configuration content of the PTP scenario to deploy
                  ${PTP4L_CONF_CONTENTS}
              mode: 0644
              user:
                name: root
              group:
                name: root
            - path: /etc/sysconfig/ptp4l
              overwrite: true
              contents:
                inline: |
                  ## Path:           Network/LinuxPTP
                  ## Description:    Precision Time Protocol (PTP): ptp4l settings
                  ## Replace ${PTP4L_OPTIONS} with the ptp4l OPTIONS value of the PTP scenario to deploy
                  OPTIONS="${PTP4L_OPTIONS}"
              mode: 0644
              user:
                name: root
              group:
                name: root
            - path: /etc/sysconfig/phc2sys
              overwrite: true
              contents:
                inline: |
                  ## Path:           Network/LinuxPTP
                  ## Description:    Precision Time Protocol (PTP): phc2sys settings
                  ## Replace ${PHC2SYS_OPTIONS} with the phc2sys OPTIONS value of the PTP scenario to deploy
                  OPTIONS="${PHC2SYS_OPTIONS}"
              mode: 0644
              user:
                name: root
              group:
                name: root
    kubelet:
      extraArgs:
        - provider-id=metal3://BAREMETALHOST_UUID
    nodeName: "localhost.localdomain"

The placeholders are provided by each scenario:

Placeholder	Description
`${PTP4L_CONF_PATH}`	Path of the `ptp4l` profile file, for example `/etc/ptp4l-G.8275.1.conf`.
`${PTP4L_CONF_CONTENTS}`	Contents of that `ptp4l` file, taken from the chosen scenario.
`${PTP4L_OPTIONS}`	Value of the `OPTIONS=` line in `/etc/sysconfig/ptp4l`.
`${PHC2SYS_OPTIONS}`	Value of the `OPTIONS=` line in `/etc/sysconfig/phc2sys`.

For the specific values to substitute in each case, see the CAPI automation of the corresponding scenario in the PTP scenarios (Section 47.1, “Introduction”) of the PTP documentation.

Note

This approach is convenient only if the hardware in the cluster is uniform and the same PTP configuration is needed on all hosts, interface name included. The Boundary Clock scenarios that rely on ts2phc or on-board DPLL routing require additional files and services in addition to the base example, which are described in the next section.

55.1.2 GNR-D multi-NIC scenarios (DPLL) #

The GNR-D Boundary Clock (Scenario 4 (Section 47.6, “Scenario 4 - Boundary Clock (PTP-only) Multi-NIC with internal timing module (DPLL GNR-D based)”)) cannot be described with a single ptp4l profile: it runs one ptp4l instance per PHC, aligns the add-in cards with ts2phc, wires the on-board DPLL at boot and forwards the Grandmaster parameters periodically. The provisioning therefore extends the base example with several configuration files, scripts and systemd units, all deployed through the same RKE2ControlPlane Ignition block. The contents of every file and unit are the ones listed in Scenario 4 (Section 47.6, “Scenario 4 - Boundary Clock (PTP-only) Multi-NIC with internal timing module (DPLL GNR-D based)”).

Enable the daemons in the EIB image

The GNR-D units require a custom setup: a DPLL service, one templated ptp4l instance per PHC, and a Grandmaster forwarding timer. Consequently, you must deploy and enable them via the Ignition block below instead of using the EIB systemd.enable list. The EIB image only needs the linuxptp package, which provides the ptp4l, ts2phc and pmc binaries:

operatingSystem:
  packages:
    packageList:
      - linuxptp      # already present in the Telco package list
    sccRegistrationCode: $SCC_REGISTRATION_CODE

Deploy the files and units through Cluster API

The RKE2ControlPlane object below deploys the GNR-D files and units. It follows the same approach as the standard example: the file contents are provided as placeholders and taken from Scenario 4 (Section 47.6, “Scenario 4 - Boundary Clock (PTP-only) Multi-NIC with internal timing module (DPLL GNR-D based)”), while the systemd units (constant for this scenario) are shown inline. Complete it with the other Cluster API objects as in the standard example, and replace the placeholders with the values of the scenario:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: RKE2ControlPlane
metadata:
  name: single-node-cluster
  namespace: default
spec:
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: Metal3MachineTemplate
    name: single-node-cluster-controlplane
  replicas: 1
  version: ${RKE2_VERSION}
  rolloutStrategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 0
  registrationMethod: "control-plane-endpoint"
  serverConfig:
    cni: canal
  agentConfig:
    format: ignition
    cisProfile: cis
    additionalUserData:
      config: |
        variant: fcos
        version: 1.4.0
        systemd:
          units:
            - name: rke2-preinstall.service
              enabled: true
              contents: |
                [Unit]
                Description=rke2-preinstall
                Wants=network-online.target
                Before=rke2-install.service
                ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
                [Service]
                Type=oneshot
                User=root
                ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
                ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
                ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
                ExecStart=/bin/sh -c "echo \"node-label:\" >> /etc/rancher/rke2/config.yaml"
                ExecStart=/bin/sh -c "echo \"  - metal3.io/uuid=$(jq -r .uuid /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
                ExecStartPost=/bin/sh -c "umount /mnt"
                [Install]
                WantedBy=multi-user.target
            - name: gnrd-ptp-dpll-setup.service
              enabled: true
              contents: |
                [Unit]
                Description=GNR-D PTP DPLL and internal signal setup
                Before=ts2phc.service ptp4l@nac.service ptp4l@aic1.service ptp4l@aic2.service
                [Service]
                Type=oneshot
                RemainAfterExit=yes
                ExecStart=/usr/local/bin/01-gnrd-startup-setup.sh
                [Install]
                WantedBy=multi-user.target
            - name: ptp4l@.service
              contents: |
                [Unit]
                Description=PTP Boundary Clock instance %i
                After=gnrd-ptp-dpll-setup.service
                Requires=gnrd-ptp-dpll-setup.service
                [Service]
                ExecStart=/usr/sbin/ptp4l -f /etc/ptp4l-%i.conf
                [Install]
                WantedBy=multi-user.target
            - name: ptp4l@nac.service
              enabled: true
            - name: ptp4l@aic1.service
              enabled: true
            - name: ptp4l@aic2.service
              enabled: true
            - name: ts2phc.service
              enabled: true
            - name: ptp-forward-gm.service
              contents: |
                [Unit]
                Description=Forward Grandmaster parameters to the add-in ptp4l instances
                After=ptp4l@nac.service ptp4l@aic1.service ptp4l@aic2.service
                [Service]
                Type=oneshot
                ExecStart=/usr/local/bin/04-ptp-forward-GM.sh
            - name: ptp-forward-gm.timer
              enabled: true
              contents: |
                [Unit]
                Description=Refresh forwarded Grandmaster parameters
                [Timer]
                OnBootSec=2min
                OnUnitActiveSec=5min
                [Install]
                WantedBy=timers.target
        storage:
          files:
            - path: /usr/local/bin/01-gnrd-startup-setup.sh
              mode: 0755
              overwrite: true
              contents:
                inline: |
                  ${GNRD_DPLL_SETUP_CONTENTS}
            - path: /usr/local/bin/04-ptp-forward-GM.sh
              mode: 0755
              overwrite: true
              contents:
                inline: |
                  ${GNRD_FORWARD_GM_CONTENTS}
            - path: /etc/ts2phc-cf-all.cfg
              mode: 0644
              overwrite: true
              contents:
                inline: |
                  ${TS2PHC_CONF_CONTENTS}
            - path: /etc/sysconfig/ts2phc
              mode: 0644
              overwrite: true
              contents:
                inline: |
                  OPTIONS="${TS2PHC_OPTIONS}"
            - path: /etc/ptp4l-nac.conf
              mode: 0644
              overwrite: true
              contents:
                inline: |
                  ${PTP4L_NAC_CONTENTS}
            - path: /etc/ptp4l-aic1.conf
              mode: 0644
              overwrite: true
              contents:
                inline: |
                  ${PTP4L_AIC1_CONTENTS}
            - path: /etc/ptp4l-aic2.conf
              mode: 0644
              overwrite: true
              contents:
                inline: |
                  ${PTP4L_AIC2_CONTENTS}
    kubelet:
      extraArgs:
        - provider-id=metal3://BAREMETALHOST_UUID
    nodeName: "localhost.localdomain"

The placeholders are provided by the GNR-D scenario:

Placeholder	Description
`${GNRD_DPLL_SETUP_CONTENTS}`	Contents of the DPLL boot-time setup script (`01-gnrd-startup-setup.sh`).
`${GNRD_FORWARD_GM_CONTENTS}`	Contents of the Grandmaster forwarding script (`04-ptp-forward-GM.sh`).
`${TS2PHC_CONF_CONTENTS}`	Contents of the `ts2phc` configuration file (`/etc/ts2phc-cf-all.cfg`).
`${TS2PHC_OPTIONS}`	Value of the `OPTIONS=` line in `/etc/sysconfig/ts2phc`.
`${PTP4L_NAC_CONTENTS}`	Contents of the integrated-controller `ptp4l` file (`/etc/ptp4l-nac.conf`).
`${PTP4L_AIC1_CONTENTS}`	Contents of the add-in card 1 `ptp4l` file (`/etc/ptp4l-aic1.conf`).
`${PTP4L_AIC2_CONTENTS}`	Contents of the add-in card 2 `ptp4l` file (`/etc/ptp4l-aic2.conf`).

For the specific values to substitute, see the CAPI automation of Scenario 4 (Section 47.6, “Scenario 4 - Boundary Clock (PTP-only) Multi-NIC with internal timing module (DPLL GNR-D based)”).

Note

The ptp4l@.service template is deployed without enabled; the instances are created by enabling ptp4l@nac, ptp4l@aic1 and ptp4l@aic2. ptp-forward-gm.service is triggered by its timer, so only the timer is enabled. ts2phc.service is shipped by the linuxptp package and reads /etc/sysconfig/ts2phc; the boot ordering is enforced by the Before= directive of gnrd-ptp-dpll-setup.service. Adjust the device names (em5, ptpN, p1p1, p2p1) and the number of add-in cards to match the target hardware.