20 MetalLB in front of the Kubernetes API server #

This guide demonstrates using a MetalLB service to expose the K3s API externally on an HA K3s cluster with three control-plane nodes. To achieve this, a Kubernetes Service of type LoadBalancer and Endpoints will be manually created. The Endpoints keep the IPs of all control plane nodes available in the cluster. For the Endpoint to be continuously synchronized with the events occurring in the cluster (adding/removing a node or a node goes offline), the Endpoint Copier Operator will be deployed. The operator monitors the events happening in the default kubernetes Endpoint and updates the managed one automatically to keep them in sync. Since the managed Service is of type LoadBalancer, MetalLB assigns it a static ExternalIP. This ExternalIP will be used to communicate with the API Server.

20.1 Prerequisites #

Three hosts to deploy K3s on top.
- Ensure the hosts have different host names.
- For testing, these could be virtual machines
At least 2 available IPs in the network (one for the Traefik and one for the managed service).
Helm

20.2 Installing K3s #

Note

If you do not want a fresh cluster but want to use an existing one, skip this step and proceed to the next one.

First, a free IP in the network must be reserved that will be used later for ExternalIP of the managed Service.

SSH to the first host and install K3s in cluster mode as:

# Export the free IP mentioned above
export VIP_SERVICE_IP=<ip>
export INSTALL_K3S_SKIP_START=false

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --cluster-init \
 --disable=servicelb --write-kubeconfig-mode=644 --tls-san=${VIP_SERVICE_IP} \
 --tls-san=https://${VIP_SERVICE_IP}.sslip.io" K3S_TOKEN=foobar sh -

Note

Make sure that --disable=servicelb flag is provided in the k3s server command.

Important

From now on, the commands should be run on the local machine.

To access the API server from outside, the IP of the K3s VM will be used.

# Replace <node-ip> with the actual IP of the machine
export NODE_IP=<node-ip>
scp ${NODE_IP}:/etc/rancher/k3s/k3s.yaml ~/.kube/config && sed \
 -i '' "s/127.0.0.1/${NODE_IP}/g" ~/.kube/config && chmod 600 ~/.kube/config

20.3 Configuring an existing K3s cluster #

Note

This step is valid only if you intend to use an existing K3s cluster.

To use an existing K3s cluster, the servicelb LB should be disabled and also tls-san flags modified.

To change the K3s flags, /etc/systemd/system/k3s.service should be modified on all the VMs in the cluster.

The flags should be inserted in the ExecStart. For example:

# Replace the <vip-service-ip> with the actual ip
ExecStart=/usr/local/bin/k3s \
    server \
        '--cluster-init' \
        '--write-kubeconfig-mode=644' \
        '--disable=servicelb' \
        '--tls-san=<vip-service-ip>' \
        '--tls-san=https://<vip-service-ip>.sslip.io' \

Then the following commands should be executed for K3s to load the new configurations:

systemctl daemon-reload
systemctl restart k3s

20.4 Installing MetalLB #

To deploy MetalLB, the MetalLB on K3s guide can be used.

NOTE: Ensure that the IP addresses of the ip-pool IPAddressPool do not overlap with the IP addresses previously selected for the LoadBalancer service.

Create a separate IpAddressPool that will be used only for the managed Service.

# Export the VIP_SERVICE_IP on the local machine
# Replace with the actual IP
export VIP_SERVICE_IP=<ip>

cat <<-EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: kubernetes-vip-ip-pool
  namespace: metallb-system
spec:
  addresses:
  - ${VIP_SERVICE_IP}/32
  serviceAllocation:
    priority: 100
    namespaces:
      - default
EOF

cat <<-EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ip-pool-l2-adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - ip-pool
  - kubernetes-vip-ip-pool
EOF

20.5 Installing the Endpoint Copier Operator #

helm install \
endpoint-copier-operator oci://registry.suse.com/edge/endpoint-copier-operator-chart \
--namespace endpoint-copier-operator \
--create-namespace

The command above will deploy three different resources in the cluster:

The endpoint-copier-operator operator Deployment with two replicas. One will be the leader and the other will take over the leader role if needed.
A Kubernetes service called kubernetes-vip in the default namespace that will be a copy of the kubernetes Service but from type LoadBalancer.
An Endpoint resource called kubernetes-vip in the default namespace that will be a copy of the kubernetes Endpoint.

Verify that the kubernetes-vip Service has the correct IP address:

kubectl get service kubernetes-vip -n default \
 -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'

Ensure that the kubernetes-vip and kubernetes Endpoints resources in the default namespace point to the same IPs.

kubectl get endpoints kubernetes kubernetes-vip

If everything is correct, the last thing left is to use the VIP_SERVICE_IP in our Kubeconfig.

sed -i '' "s/${NODE_IP}/${VIP_SERVICE_IP}/g" ~/.kube/config

From now on, all the kubectl will go through the kubernetes-vip service.

20.6 Adding control-plane nodes #

To monitor the entire process, two more terminal tabs can be opened.

First terminal:

watch kubectl get nodes

Second terminal:

watch kubectl get endpoints

Now execute the commands below on the second and third nodes.

# Export the VIP_SERVICE_IP in the VM
# Replace with the actual IP
export VIP_SERVICE_IP=<ip>
export INSTALL_K3S_SKIP_START=false

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server \
 --server https://${VIP_SERVICE_IP}:6443 --disable=servicelb \
 --write-kubeconfig-mode=644" K3S_TOKEN=foobar sh -