NAME
eam-start-fail - work through a failing ESX Agent Manager service, from expired certificates to lsdoctor and eam.properties
DESCRIPTION
You see that the VMware ESX Agent Manager is not running.
When starting the service you get the following error:
Troubleshooting:
- List all certificates to look for possible out of date certifications
- Renewed the certificates through the vCSA’s Certificate Manager (explanation below)
- Manual replacement of the eam.properties file > kb 2148582
- Ran the lsdoctor tool > kb 80469
List certificates
root@vcsa1 [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;
MACHINE_SSL_CERT
Not After : Sep 14 11:21:40 2024 GMT
TRUSTED_ROOTS
Not After : Aug 18 11:26:49 2031 GMT
Not After : Aug 18 08:36:16 2031 GMT
Not After : Sep 8 11:52:23 2032 GMT
TRUSTED_ROOT_CRLS
machine
Not After : Oct 23 11:37:19 2025 GMT
vsphere-webclient
Not After : Oct 23 11:37:21 2025 GMT
vpxd
Not After : Oct 23 11:37:22 2025 GMT
vpxd-extension
Not After : Oct 23 11:37:22 2025 GMT
APPLMGMT_PASSWORD
data-encipherment
Not After : Sep 9 09:38:02 2032 GMT
SMS
Not After : Aug 23 11:31:39 2031 GMT
hvc
Not After : Oct 23 11:37:25 2025 GMT
wcp
Not After : Oct 23 11:37:25 2025 GMT
BACKUP_STORE
Not After : Sep 14 11:21:40 2024 GMT
Not After : Oct 23 11:13:01 2025 GMT
Not After : Oct 23 11:13:02 2025 GMT
Certificate Manager > Option 6
root@vcsa1 [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 7.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| NOTE: Solution user certs will be deprecated in a future |
| release of vCenter. Refer to release notes for more details.|
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 6
Manual replacement of the eam.properties file > kb 2148582
Copy the file > cp eam.properties eam.properties.old
root@vcsa1 [ /etc/vmware-eam ]# cp eam.properties eam.properties.old
Determine the hostid > cat /etc/vmware/install-defaults/sca.hostid:
root@vcsa1 [ /etc/vmware-eam ]# cat /etc/vmware/install-defaults/sca.hostid
18fb8399-4c31-389c-2d30-3c3ce6ce9addlsdoctor script > kb 80469
Upload the lsdoctor.zip file through winscp and put it in the tmp dir, from here unzip the file and enter the lsdoctor-230919 dir to execute the script with the command: python lsdoctor.py -l
root@vcsa1 [ /tmp ]# cd lsdoctor-230919 root@vcsa1 [ /tmp/lsdoctor-230919 ]# ls CHANGELOG config_log.ini lib lsdoctor.py README templates root@vcsa1 [ /tmp/lsdoctor-230919 ]# python lsdoctor.py -l
Solution
When analyzing the log files with VMware support we looked at the vmon.log file, and my Colleague noticed the following error:
Stderr = error: cannot stat /etc/logrotate.d/vmware-eam.lr: No such file or directory
Which means that the eam service is trying to start a pre-task, named vmware-eam.lr, but it can’t because it isn’t present at it’s expected location. So we looked in the directory > /etc/logrotate.d
And the file(s) were indeed missing.
Comparing it to the server that had no issues we copied the entire directory including the vmware-eam.lr
root@vcsa1 [ /etc/logrotate.d ]# ls -l total 84 cap-instant-br.lr cap-workflow-engine.lr cloudvm_ram_size.lr hcl.lr patch-runner.lr procstate.lr syslog vami-lighttpd vmware-applmgmt.lr vmware-auditd.lr vmware-autodeploy vmware-eam.lr vmware_postgres_archiver vmware_postgres_cron vmware_postgres_log vmware_postgres_telemetry vmware-psc.lr vmware-sshlog.lr vmware-syslog.lr vmware-vmafd.lr vmware-vsphere-ui.lr
After copying the files to the /etc/logrotate.d directory we were able to start the EAM service successfully!
SEE ALSO
hostprofile-vs-config(8), vmdk-lock(8), vcf-study-guide(8), vmware(7)