Before upgrading VMware Cloud Foundation (VCF) to a newer release, you should first run an upgrade pre-check to ensure there are no issues that could arise during the upgrade. Recently, I helped one of our customers plan and execute the VCF upgrade from VCF 5.2.2 to 9.0.2, and we completed the Upgrade Precheck to ensure everything was in order! There are some expected errors, such as updating the vSAN HCL and the VCF LCM Manifest file, which are straightforward to resolve.

Besides the usual errors, we also faced “ERROR vSphere SHA-1 validation failed” and without addressing this issue, we shouldn’t continue the upgrade. This issue came up because the SHA-1 hashing algorithm was used to sign vCenter Server SSL certificates. VCF requires stronger algorithms, like SHA-256, for certificate signing.

To verify if the vCenter certificates are using SHA-1, you can run the following command on the vCenter server. First, SSH to the vCenter, log in with root, run the shell command and run the following command:

echo | /usr/bin/openssl s_client -connect localhost:443 2>/dev/null | /usr/bin/openssl x509 -noout -text | grep "Signature Algorithm"

The output shows whether the certificates are signed with SHA-1 or SHA-256, and if SHA-1 is used, we need to refresh the certificates on vCenter Servers. We have a KB article that explains how to run the refresh process. Basically, take an offline snapshot from the vCenter Server, connect to vCenter through SSH, and then run the command below.

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

In our case, after successfully running the command, we verified that SHA-256 is being used for certificate signing.

Re-running the SDDC Manager upgrade pre-check disappointed us because it was still there! We waited a bit and then re-ran the check and finally restarted both vCenter and SDDC Manager, but the issue was still there!

If you face the same situation, the final step in troubleshooting is to connect to the SDDC Manager via SSH, then from the SDDC Manager, connect to the vCenter Server! Sounds a bit odd, but when you connect to the vCenter Server directly from SDDC Manager, the certificates with SHA-256 signing are updated on SDDC Manager, and it bypasses any stale or cached certificates! 😀

Please make sure you have a backup and a cold snapshot of both SDDC Manager and vCenter server before executing any of the above commands!

If you ran into the same issue and could solve it with the steps above, or if you had to take any extra steps, please share them in the comments so other readers of this post can have fun troubleshooting this upgrade precheck error! 😀

Sadaf also wrote a series of blog posts explaining how to upgrade VCF from 9.0.0 to 9.0.1, which you can read here!