phatqYou see that the VMware ESX Agent Manager is not running.
When starting the service you get the following error:
Troubleshooting:
- List all certificates to look for possible out of date certifications
- Renewed the certificates through the vCSA’s Certificate Manager (explanation below)
- Manual replacement of the eam.properties file > kb 2148582
- Ran the lsdoctor tool > kb 80469
List certificates
root@vcsa1 [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;
MACHINE_SSL_CERT
Not After : Sep 14 11:21:40 2024 GMT
TRUSTED_ROOTS
Not After : Aug 18 11:26:49 2031 GMT
Not After : Aug 18 08:36:16 2031 GMT
Not After : Sep 8 11:52:23 2032 GMT
TRUSTED_ROOT_CRLS
machine
Not After : Oct 23 11:37:19 2025 GMT
vsphere-webclient
Not After : Oct 23 11:37:21 2025 GMT
vpxd
Not After : Oct 23 11:37:22 2025 GMT
vpxd-extension
Not After : Oct 23 11:37:22 2025 GMT
APPLMGMT_PASSWORD
data-encipherment
Not After : Sep 9 09:38:02 2032 GMT
SMS
Not After : Aug 23 11:31:39 2031 GMT
hvc
Not After : Oct 23 11:37:25 2025 GMT
wcp
Not After : Oct 23 11:37:25 2025 GMT
BACKUP_STORE
Not After : Sep 14 11:21:40 2024 GMT
Not After : Oct 23 11:13:01 2025 GMT
Not After : Oct 23 11:13:02 2025 GMT
Certificate Manager > Option 6
root@vcsa1 [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 7.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| NOTE: Solution user certs will be deprecated in a future |
| release of vCenter. Refer to release notes for more details.|
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 6
Manual replacement of the eam.properties file > kb 2148582
Copy the file > cp eam.properties eam.properties.old
root@vcsa1 [ /etc/vmware-eam ]# cp eam.properties eam.properties.old
Determine the hostid > cat /etc/vmware/install-defaults/sca.hostid:
root@vcsa1 [ /etc/vmware-eam ]# cat /etc/vmware/install-defaults/sca.hostid
18fb8399-4c31-389c-2d30-3c3ce6ce9addlsdoctor script > kb 80469
Upload the lsdoctor.zip file through winscp and put it in the tmp dir, from here unzip the file and enter the lsdoctor-230919 dir to execute the script with the command: python lsdoctor.py -l
root@vcsa1 [ /tmp ]# cd lsdoctor-230919
root@vcsa1 [ /tmp/lsdoctor-230919 ]# ls
CHANGELOG config_log.ini lib lsdoctor.py README templates
root@vcsa1 [ /tmp/lsdoctor-230919 ]# python lsdoctor.py -l
Solution
When analyzing the log files with VMware support we looked at the vmon.log file, and my Colleague noticed the following error:
Stderr = error: cannot stat /etc/logrotate.d/vmware-eam.lr: No such file or directory
Which means that the eam service is trying to start a pre-task, named vmware-eam.lr, but it can’t because it isn’t present at it’s expected location. So we looked in the directory > /etc/logrotate.d
And the file(s) were indeed missing.
Comparing it to the server that had no issues we copied the entire directory including the vmware-eam.lr
root@vcsa1 [ /etc/logrotate.d ]# ls -l
total 84
cap-instant-br.lr
cap-workflow-engine.lr
cloudvm_ram_size.lr
hcl.lr
patch-runner.lr
procstate.lr
syslog
vami-lighttpd
vmware-applmgmt.lr
vmware-auditd.lr
vmware-autodeploy
vmware-eam.lr
vmware_postgres_archiver
vmware_postgres_cron
vmware_postgres_log
vmware_postgres_telemetry
vmware-psc.lr
vmware-sshlog.lr
vmware-syslog.lr
vmware-vmafd.lr
vmware-vsphere-ui.lr
After copying the files to the /etc/logrotate.d directory we were able to start the EAM service successfully!