由于长时间没有启动 VCF 情况，如今在启动 SDDC Manager 组件后，UI 一直处于如下图所示的“初始化”状态。当时第一直觉就认为肯定是 VCF 情况组件的用户密码过期了，之前在管理 VCF 情况中组件的用户密码和密码计谋文章中了解过，VCF 情况中组件的用户密码统一由 SDDC Manager 来管理，好比执行轮换、更新以及修复等，而默认情况下，VCF 组件的用户密码具有有效期，因为很长时间没有进行密码手动/自动轮换，而我也没有实时进行处置惩罚，所以猜测如今组件的密码应该已经到期了。
最终，在等候很长一段时间后，网页出现如下所示的报错，或者 UI 提示服务器启动失败。- {"message":"Identity Internal Server Error","code":"IDENTITY_INTERNAL_SERVER_ERROR","status":500}
- Server failed to start.Fail to init PSC and/or Postgres.Check the SDDC Manager UI Server logs for more details.
假如你实验登录 VCF 管理域 vCenter Server VAMI 管理界面，并实验使用 root 用户登录，效果表现不能认证用户。
假如你实验登录 VCF 管理域 NSX Manager VIP 管理界面，并实验使用 admin 用户登录，效果表现您的密码已过期。
之前都是在正常情况下，再去管理 VCF 组件用户密码，如今恰好趁这个时机，看看当 VCF 组件密码已经出现过期后，该如何进行办理。
根据上面的情况可以大概判定方向，应该就是 VCF 组件的密码发生过期，进而导致了 SDDC Manager 服务无法启动。所以，我可以通过 SSH 并使用 vcf 用户连接到 SDDC Manager，然后使用 SoS 实用程序来直接检查 VCF 组件的密码健康状态，固然，假如还不知道方向，直接进行所有检查（-- health-check）也行。从下面的输出效果可以发现，vCenter Server 组件的用户密码状态无法获取，问题可能出如今该组件上。- vcf@vcf-mgmt01-sddc01 [ ~ ]$ sudo /opt/vmware/sddc-support/sos --password-health --force
- Welcome to Supportability and Serviceability(SoS) utility!
- Locks info:
- [{'status': 'ACTIVE', 'lockId': '363d9097-4938-4872-9fad-d50a05c4360a', 'createdTimeStamp': 1, 'vcfClientContext': {'serviceIdentifier': 'Password Manager', 'pollingInterval': 0}, 'resourceType': 'DEPLOYMENT', 'description': 'Password management operation in progress. Please wait for completion'}]
- Found active workflow for Password Manager, Description: Password management operation in progress. Please wait for completion
- User passed --force flag to continue SoS operations although notified about running workflows. Please expect failures with SoS operations.
- Performing SoS operation for vcf-mgmt01 domain components
- Health Check : /var/log/vmware/vcf/sddc-support/healthcheck-2025-03-17-04-06-48-10241
- Health Check log : /var/log/vmware/vcf/sddc-support/healthcheck-2025-03-17-04-06-48-10241/sos.log
- NOTE : The Health check operation was invoked without --skip-known-host-check, additional identity checks will be included for Connectivity Health, Password Health and Certificate Health Checks because of security reasons.
- SDDC Manager : vcf-mgmt01-sddc01.mulab.local
- +-------------------------+-----------+
- | Stage | Status |
- +-------------------------+-----------+
- | Bringup | Completed |
- | Management Domain State | Completed |
- +-------------------------+-----------+
- +--------------------+---------------+
- | Component | Identity |
- +--------------------+---------------+
- | SDDC-Manager | 192.168.32.70 |
- | Number of Servers | 4 |
- +--------------------+---------------+
- Password Expiry Status : RED
- +-----+-----------------------------------------+---------------------------+-------------------+--------------+-----------------+-----------------------+
- | SL# | Component | User | Last Changed Date | Expiry Date | Expires in Days | State |
- +-----+-----------------------------------------+---------------------------+-------------------+--------------+-----------------+-----------------------+
- | 1 | ESXI : vcf-mgmt01-esxi01.mulab.local | svc-vcf-vcf-mgmt01-esxi01 | Mar 14, 2025 | Never | Never | GREEN |
- | | | root | Mar 14, 2025 | Never | Never | GREEN |
- | 2 | ESXI : vcf-mgmt01-esxi02.mulab.local | svc-vcf-vcf-mgmt01-esxi02 | Mar 14, 2025 | Never | Never | GREEN |
- | | | root | Mar 14, 2025 | Never | Never | GREEN |
- | 3 | ESXI : vcf-mgmt01-esxi03.mulab.local | svc-vcf-vcf-mgmt01-esxi03 | Mar 14, 2025 | Never | Never | GREEN |
- | | | root | Mar 14, 2025 | Never | Never | GREEN |
- | 4 | ESXI : vcf-mgmt01-esxi04.mulab.local | svc-vcf-vcf-mgmt01-esxi04 | Mar 14, 2025 | Never | Never | GREEN |
- | | | root | Mar 14, 2025 | Never | Never | GREEN |
- | 5 | SDDC : vcf-mgmt01-sddc01.mulab.local | vcf | Mar 17, 2025 | Apr 21, 2026 | 400 days | GREEN |
- | | | backup | Dec 07, 2024 | Jan 11, 2026 | 300 days | GREEN |
- | | | root | Dec 07, 2024 | Jan 11, 2026 | 300 days | GREEN |
- | 6 | vCenter : vcf-mgmt01-vcsa01.mulab.local | root | - | - | - | Failed to get details |
- +-----+-----------------------------------------+---------------------------+-------------------+--------------+-----------------+-----------------------+
- Legend:
- GREEN - No attention required, health status is NORMAL
- YELLOW - May require attention, health status is WARNING
- RED - Requires immediate attention, health status is CRITICAL
- Health Check completed successfully for : [VCF-SUMMARY]
- Operation failed for : [PASSWORD-CHECK(1/12 Failed)]
- For detailed report please refer : /var/log/vmware/vcf/sddc-support/healthcheck-2025-03-17-04-06-48-10241/report.json
- vcf@vcf-mgmt01-sddc01 [ ~ ]$
- 2025-03-17T03:54:54.221+0000 ERROR [vcf_dm,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VsphereClient,ForkJoinPool.commonPool-worker-4] Failed to connect to https://vcf-mgmt01-vcsa01.mulab.local:443/sdk
- java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
- faultCause = null,
- faultMessage = null
- }
- at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:121)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.connect(VcManagerBase.java:514)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:495)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:468)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.getVcManagerBase(VcManagerFactory.java:436)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:52)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:157)
- at com.vmware.vcf.configreconciler.config.checks.ClusterHaSettingsConfigDriftCheck.isConfigurationRealized(ClusterHaSettingsConfigDriftCheck.java:44)
- at com.vmware.vcf.configreconciler.service.ResourceCacheService.performIsConfigRealizedCheck(ResourceCacheService.java:1014)
- at com.vmware.vcf.configreconciler.service.ResourceCacheService.refreshFailedConfigRealizedCheck(ResourceCacheService.java:1055)
- at com.vmware.vcf.configreconciler.service.ResourceCacheService.refreshConfigApplicability(ResourceCacheService.java:935)
- at com.vmware.vcf.configreconciler.service.ResourceCacheService.lambda$refreshLocalInventoryWithConfigApplicability$0(ResourceCacheService.java:264)
- at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
- at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3573)
- at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
- at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
- at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
- at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
- at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
- at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
- at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
- at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
- Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
- at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
- at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
- at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
- at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
- at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
- at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
- at java.base/java.lang.Class.newInstance(Class.java:645)
- at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)
- at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)
- at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)
- at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167)
- at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105)
- at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92)
- at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86)
- at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
- at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127)
- at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70)
- at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:288)
- at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:243)
- at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:267)
- at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:56)
- at com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan.runWithinSpanContext(NoopTracer.java:120)
- at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:17)
- at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:54)
- at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229)
- at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:119)
- at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
- at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
- at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
- at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
- at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
- at jdk.proxy2/jdk.proxy2.$Proxy287.login(Unknown Source)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:120)
- ... 21 common frames omitted
- 2025-03-17T03:54:54.235+0000 WARN [vcf_dm,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VsphereClient,ForkJoinPool.commonPool-worker-4] Error logging out of session
- com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
- 2025-03-17T08:52:47.868+0000 ERROR [vcf_om,a20e030c092d465d,8a4d] [c.v.e.s.c.c.v.vsphere.VcManagerBase,http-nio-127.0.0.1-7300-exec-1] Cannot complete login due to incorrect credentials: vcf-mgmt01-vcsa01.mulab.local, svc-vcf-mgmt01-sddc01-vcf-mgmt01-vcsa01@vsphere.local.
- 2025-03-17T08:52:47.875+0000 DEBUG [vcf_om,a20e030c092d465d,8a4d] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7300-exec-1] Fetching Management vCenter data from inventory
- 2025-03-17T08:52:47.875+0000 DEBUG [vcf_om,a20e030c092d465d,8a4d] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7300-exec-1] Fetching vCenters data from inventory
- 2025-03-17T08:52:47.897+0000 ERROR [vcf_om,a20e030c092d465d,8a4d] [c.v.v.r.s.PluginInfoServiceImpl,http-nio-127.0.0.1-7300-exec-1] Exception when calling get extension: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cannot connect to vCenter Server vcf-mgmt01-vcsa01.mulab.local
- at com.vmware.vcf.remoteplugin.service.PluginRegistrationServiceImpl.get(PluginRegistrationServiceImpl.java:174)
- at com.vmware.vcf.remoteplugin.service.PluginInfoServiceImpl.initSddcManagerPlugin(PluginInfoServiceImpl.java:75)
- at com.vmware.vcf.remoteplugin.controller.PluginRegistrationController.initSddcManagerPlugin(PluginRegistrationController.java:53)
- at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
- at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
- at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
- at java.base/java.lang.reflect.Method.invoke(Method.java:569)
- at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
- at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
- at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
- at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
- at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:798)
- at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
- 2025-03-17T03:43:16.119+0000 ERROR [vcf_om,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-scheduler-2] Cannot complete login due to incorrect credentials: vcf-mgmt01-vcsa01.mulab.local, svc-vcf-mgmt01-sddc01-vcf-mgmt01-vcsa01@vsphere.local.
- 2025-03-17T03:43:16.119+0000 WARN [vcf_om,0000000000000000,0000] [c.v.v.t.services.TaskPublisher,om-scheduler-2] Failed to check for leftover tasks
- java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
- faultCause = null,
- faultMessage = null
- }
- at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:121)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.connect(VcManagerBase.java:514)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:495)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:468)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.getVcManagerBase(VcManagerFactory.java:436)
- at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:52)
- at com.vmware.vcf.taskpublisher.services.TaskPublisher.checkForLeftoverTasks(TaskPublisher.java:343)
- at com.vmware.vcf.taskpublisher.services.TaskPublisher.cloneTasks(TaskPublisher.java:102)
- at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
- at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96)
- at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
- at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
- at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
- at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
所有的信息都指向那个 SVC 用户，这到底是一个什么用户？在 VCF 情况它被称为服务账户（Service Account），服务账户类型（SERVICE）由 VMware Cloud Foundation 自动创建，用于产品组件之间的交互。服务账户（Service Account）通常不需要我们手动干预，但是一些特别情况除外。
综合以上信息，可以大概判定是 SVC 用户的根据不正确导致无法连接到 SDDC Manager 或者 SDDC Manager 无法连接到 vCenter Server 进行认证，这种情况有可能是 SDDC Manager 自动轮换了 SVC 用户的密码，但是只更新了自身的数据库，vCenter Server 由于恒久没有启动并且用户密码已经过期，所以导致无法进行连接 vCenter Server 并对 SVC 密码进行更新。
如今有两个办法，一是在 vCenter Server 上找到这个 SVC 用户的密码，然后去 SDDC Manager 更新这个 SVC 用户的密码；另一个就是反过来。很显然，第一种方法不太可能找到，但可幸的是官方提供了第二种方法。
参考 KB 327195 知识库，可以通过 SDDC Manager 获取服务账户（Service Account）的根据密码，通过 SSH 以 vcf 用户登录到 SDDC Manager 并使用“su - ”切换到 root 用户，然后使用以下命令获取到 SDDC Manager API 的连接 token 根据。- TOKEN=$(curl -d '{"username" : administrator@vsphere.local, "password" : "Vcf521@password"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
- curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | less
通过运行以上命令，你应该可以找到对应服务账户的根据密码，“username”为服务账户，“secret”为根据密码。
然后登录到 vCenter Server（vSphere Client），在系统管理中找到用户管理，在用户中选择 SSO 域（vsphere.local），找到服务账户并点击“编辑”，将密码修改为上面获取到的密码。
通过 SSH 以 vcf 用户登录到 SDDC Manager 并使用“su - ”切换到 root 用户，运行以下命令重启所有服务，或者也可以直接重启 SDDC Manager 电源。- /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
假如统统顺利，应该能正常登录到 SDDC Manager UI。导航到安全->密码管理，可以发现 vCenter Server 和 NSX Manager 组件的用户因为密码过期都已经从 SDDC Manager 断开连接。
vCenter Server 组件的账户可以直接使用密码轮换（Rotate Password）功能，这样可以直接更新并重新进行连接。
NSX Manager 组件的账户不能使用密码轮换，只能手动更改账户的密码后再使用密码修复（Remediate Password）功能。手动修改 NSX Manager 账户的密码支持多种方法（GUI、CLI、API 等），好比可以参考 KB 314657 知识库中的方法，使用 CLI 方式进行修改。
假如不知道 NSX Manager 组件账户的密码，可以在 SDDC Manager 上通过 lookup_passwords 命令查看密码。
通过 SSH 并使用 root 用户登录到 NSX Manager CLI 管理界面，然后依次执行以下操纵即可。注意，假如 NSX Manager 还未启动 SSH 服务，可以通过 vCenter Server 打开 NSX Manager 虚拟机的控制台，然后输入管理员用户和密码登录，运行“start service ssh”启动服务，运行“set service ssh start-on-boot”设置开机自启动，运行“set ssh root-login”允许 root 登录，运行“get service ssh”获取服务状态。- echo "" >/etc/security/opasswd
- /etc/init.d/nsx-mp-api-server stop
- passwd
- passwd admin
- passwd audit
- touch /var/vmware/nsx/reset_cluster_credentials
- /etc/init.d/nsx-mp-api-server start
- /usr/sbin/faillock --user root --reset
- /usr/sbin/faillock --user admin --reset
- /usr/sbin/faillock --user audit --reset
依次完成所有账户的修复，如今统统规复正常。
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。 |
