phatqWhen logging in through the vSphere Web Client you recieve the following error message:
[400] An error occurred while processing the authentication response from the vCenter Single Sign-On server. Details: HTTP error code: 403, status: BadResponse, sub status: Issuer not trusted:
It happens with multiple domain users as well as the local administrator@vsphere.local account independent of the used pc and/or browser, IE / Chrome / Firefox (JAVA & HTML5 client).
Environment Characteristics:
- A (non Windows) vCenter appliance with external, loadbalance’d psc’s.
- No linked mode.
- No use of View, SRM or Replication
Troubleshooting methods:
- In Administration > Single Sign on > Configuration > Identity Sources > select the option to set the Active Directory as Default
- Review several log files: vmware-sts-idmd.log / vsphere_client_virgo.log
- Reset vsphere.local account & machine account
Look in the “websso.log” log file if there are any LDAP error messages:
root@yourPSC.ssodomain [ /var/log/vmware/sso ]#
[2020-03-05T21:33:32.263+01:00 tomcat-http--30 vsphere.local 3599e352-04db-4b5a-99de-bfbbe7e5a31a WARN com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: 49
.....
[2020-03-05T21:33:32.278+01:00 tomcat-http--30 vsphere.local 3599e352-04db-4b5a-99de-bfbbe7e5a31a ERROR com.vmware.identity.samlservice.impl.CasIdmAccessor] Caught exception.
com.vmware.identity.idm.IDMLoginException: Login failed
Search for bind authentication errors in the “vmdird” dir, in our case the below snippet stood out stating an error regarding the machine account:
root@yourPSC [ /var/log/vmware/vmdird ]# grep -i "bind" vmdird-syslog.log
2020-03-09T20:08:51.305415+01:00 err vmdird t@14013105069: VmDirSendLdapResult: Request (Bind), Error (49), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
2020-03-09T20:08:51.305609+01:00 err vmdird t@14013105069: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "account@ssodomain,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL
2020-03-09T20:08:59.973767+01:00 err vmdird t@14013187275: VmDirSafeLDAPBind to (ldap://account@ssodomain:389) failed. SRP(9234)
This issue occurs when the Inventory service loses its trust due to a password mismatch in the vmdird for the account listed in the vmdird-syslog.log file. The solution here is to reset the password for this account.
Before you begin, take snapshots of your vCenter & PSC(‘s).
Reset the password for the account which was listed in the vmdird-syslog.log file
Connect to your PSC to perform the password reset through the vdcadmintool: /usr/lib/vmware-vmdir/bin/vdcadmintool
root@yourPSC [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcadmintool
==================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
6. Get vmdir state
7. Get vmdir log level and mask
==================
3
Please enter account UPN : account@ssodomain
New password is -
xxxxxxxxxxxxxxxxx
Now ssh to your vCenter appliance and perform the following steps:
Get in to “lwregshell” mode:
root@yourvCenter [ ~ ]# /opt/likewise/bin/lwregshell
Once in the shell, cd your way to “vmdir”
\> cd HKEY_THIS_MACHINE\Services\vmdir\
List the content of “vmdir” and look for the “dcAccountPassword”, this is where the current password is listed (I have taken out the other properties which I think is not necessary for this example)
HKEY_THIS_MACHINE\Services\vmdir> ls
[\Services\vmdir\]
+ "dcAccount" REG_SZ "account@ssodomain"
+ "dcAccountDN" REG_SZ "cn=account@ssodomain,ou=Domain Controllers,dc=vsphere,dc=local"
+ "dcAccountOldPassword" REG_SZ "Old Password"
+ "dcAccountPassword" REG_SZ "Current Password"
Now set the “New Password” for “dcAccountPassword” with the following cmd
HKEY_THIS_MACHINE\Services\vmdir> set_value dcAccountPassword "New Password"
Check to make sure the new password is there:
HKEY_THIS_MACHINE\Services\vmdir> ls
[\Services\vmdir\]
+ "dcAccount" REG_SZ "account@ssodomain"
+ "dcAccountDN" REG_SZ "cn=account@ssodomain,ou=Domain Controllers,dc=vsphere,dc=local"
+ "dcAccountOldPassword" REG_SZ "Old Password"
+ "dcAccountPassword" REG_SZ "New Password"
To sum it up:
root@yourPSC [ ~ ]# /opt/likewise/bin/lwregshell
\> cd HKEY_THIS_MACHINE\Services\vmdir\
HKEY_THIS_MACHINE\Services\vmdir> ls
[\Services\vmdir\]
+ "dcAccount" REG_SZ "account@ssodomain"
+ "dcAccountDN" REG_SZ "cn=account@ssodomain,ou=Domain Controllers,dc=vsphere,dc=local"
+ "dcAccountOldPassword" REG_SZ "Old Password"
+ "dcAccountPassword" REG_SZ "Current Password"
[HKEY_THIS_MACHINE\Services\vmdir\Parameters]
HKEY_THIS_MACHINE\Services\vmdir> set_value dcAccountPassword "New Password"
HKEY_THIS_MACHINE\Services\vmdir> ls
[\Services\vmdir\]
+ "dcAccount" REG_SZ "account@ssodomain"
+ "dcAccountDN" REG_SZ "cn=account@ssodomain,ou=Domain Controllers,dc=vsphere,dc=local"
+ "dcAccountOldPassword" REG_SZ "Old Password"
+ "dcAccountPassword" REG_SZ "New Password"
[HKEY_THIS_MACHINE\Services\vmdir\Parameters]
HKEY_THIS_MACHINE\Services\vmdir> quit
Restart the Services:
root@yourPSC [ ~ ]# service-control --stop --all
Operation not cancellable. Please wait for it to finish...
Performing stop operation on profile: ALL...
Successfully stopped service vmware-vmon
Successfully stopped profile: ALL.
Performing stop operation on service vmdnsd...
Successfully stopped service vmdnsd
Performing stop operation on service vmware-stsd...
Successfully stopped service vmware-stsd
Performing stop operation on service vmware-sts-idmd...
Successfully stopped service vmware-sts-idmd
Performing stop operation on service vmcad...
Successfully stopped service vmcad
Performing stop operation on service vmdird...
Successfully stopped service vmdird
Performing stop operation on service vmafdd...
Successfully stopped service vmafdd
Performing stop operation on service lwsmd...
Successfully stopped service lwsmd
root@yourPSC [ ~ ]# service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
Successfully started service vmdnsd
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Successfully started profile: ALL.
Source: VMware KB Arcticle
But unfortunately, all of this didn’t solve the problem. The login proces was still an issue, dependent on which PSC you got loadbalanced to, the login was succesful or not.
The thing that stood out was that this problem occurred after the upgrade to 6.7, coming from version 6.5
It turns out that after you upgrade your PSC’s you should run a SSO update script because the previous configuration has been overwritten during the upgrade proces. See the following KB Article voor more information about SSL termination from 6.5 to 6.7. The script updateLsEndpoint.py may not be necessary as the upgrade does not change the certificate.
Further more, have a look at this KB Article about the upgrade proces of your PSC’s.
After we ran the script updateSSOConfig.py we were able to log in successfully to vCenter through the (before troubled) PSC, the 2nd PSC was turned off after which we did the same test with the 2nd PSC, successfully.
Many thanks to VMware for their support in this case.