Certification error: "Error occurred while fetching machine certifcates..."

Symptom

You recieve the following certification error on your vCenter server in the vSphere UI: “Certificate Status”

In Certificate Mangement you see: “Error occurred while fetching machine certifcates: com.vmware.vcenter.certificate_management.vcenter.tls”

Apparently there is a bad certificate somewhere which vCenter doesn’t like.

Procedure

Before you start be sure to take a snapshot of your appliance so that you can revert if needed.

Use the following cmd to generate a list of certificates:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

In this case it was the data-encipherment cert that was causing this certification error as it was out of date, as shown in the below output:

[*] Store : data-encipherment
Alias : data-encipherment
            Not After : Aug 23 11:19:47 2023 GMT

Solution

Step 1. Take a backup of the old certificate and private key:

root@vcsa1 [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store data-encipherment --alias data-encipherment --output /tmp/old-data-encipherment.crt
root@vcsa1 [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store data-encipherment --alias data-encipherment --output /tmp/old-data-encipherment.key

Step 2. Delete the existing certificate from the VECS store:

root@vcsa1 [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store data-encipherment --alias data-encipherment
Deleted entry with alias [data-encipherment] in store [data-encipherment] successfully

Step 3. List the VECS store and confirm the list is Empty:

root@vcsa1 [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store data-encipherment
Number of entries in store :    0

Step 4. Generate a new certificate using the existing private key, and add it to the VECS store:

Note: –genCIScert switch in certool will automatically add the new certificate to the Certificate Store

/usr/lib/vmware-vmca/bin/certool --server=<PSC_FQDN> --genCIScert --dataencipherment --privkey=/tmp/old-data-encipherment.key --cert=/tmp/tmp-data-encipherment.crt --Name=data-encipherment --FQDN=<VC_FQDN>

root@vcsa1 [ ~ ]# /usr/lib/vmware-vmca/bin/certool --server=vcenter01.local --genCIScert --dataencipherment --privkey=/tmp/old-data-encipherment.key --cert=/tmp/tmp-data-encipherment.crt --Name=data-encipherment --FQDN=vcenter01.local
Status : Success

Note:
<PSC_FQDN>
For vCenter Server 6.7 – Replace this value with the FQDN of Platform Service Controller if vCenter is running with External PSC OR use the FQDN of vCenter Server if it is embedded PSC.
For vCenter Server 7.0 – Replace this value with the VC FQDN or with “localhost”

<VC_FQDN>
Replace this value with FQDN/PNID of vCenter Server

Step 5. Verify the new certificate using below command:

root@vcsa1 [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store data-encipherment --text | egrep 'Alias|Serial Number:|Subject:|Not Before|Not After'
Alias : data-encipherment
        Serial Number:
            Not Before: Oct  6 07:06:59 2023 GMT
            Not After : Sep  9 09:38:02 2032 GMT
        Subject: CN=data-encipherment, DC=vsphere, DC=test, C=US, OU=mID-16fb5820-4d56-476c-9d10-9c9ce5ce8add

Step 6. Restart the VPXD service for changes to take effect, otherwise Guest OS Customizations might fail:

root@vcsa1 [ ~ ]# service-control --stop vpxd
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service vpxd...
Successfully stopped service vpxd
root@vcsa1 [ ~ ]# service-control --start vpxd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...
Successfully started service vpxd

The next time you restart the vCSA appliance the certificate error should be gone.

Certification error: “Error occurred while fetching machine certifcates…”