Skip to content
Lately, I encountered some issues related to VMware vSAN in my Lab environment. The error message that was popping up all the time was “PBM error occurred during PreCloneCheckCallback“.
So how did the problem occur? First, we start with some background information. My Lab environment is powered-on when needed and powered-off when not needed. This is, of course, a little bit different than a production 24×7 environment that you have in your datacenters worldwide.
The environment was booted successfully at first glance. We are talking about Domain Controllers, vCenter Server, VMware NSX-V, nested ESXi Hosts, and vRealize Automation. When I started deploying virtual machines with a vRealize Automation (vRA) based on blueprints with vSphere Templates issues started to occur.
vRealize Automation was failing on the provisioning task and was cleaning up the deployment because of the failed state (default behavior). So it was time to dig into the underlying infrastructure.
Environment
When the issue occurred the following software versions were used in my lab environment:
- VMware vCenter 6.5 Update 2B
- VMware vRealize Automation 7.3.1
- VMware ESXi 6.5 Update 2
- VMware vSAN 6.6
Error message(s)
Here is all the information that can be found in various locations surrounding the issue. Lets start with the screenshots. The first one is from VMware vCenter and the second one is from vRealize Automation. As you can see there is clearly a problem.
VMware vCenter – PBM error occurred during PreCloneCheckCallback
vRealize Automation – Service Unavailable (503)
And here is an overview of the error message(s). Here is the vRealize Automation log entry related to the VMware vSAN issue:
Error in Execute DynamicOps.Common.Client.HtmlResponseException: Service Unavailable (503)Here is the VMware vCenter log entry related to the VMware vSAN issue:
A general system error occurred - PBM error occurred during PreCloneCheckCallback (2118557)Solution
The solution is quick but is more like a quick fix because it comes back every time I re-start my lab environment (cold boot).
Procedure:
- Open a web browser.
- Navigate to your vCenter Server URL (https://%vc%/vsphere-client).
- Login with a user that has administrator credentials (administrator@vsphere.local).
- Navigate to Hosts & Clusters > Select the vCenter Object.
- Click on the Configure tab.
- Click on the Storage Providers.
- Click on the following two buttons:
- Synchronizes all Storage Providers with the current state of the environment.
- Rescan the storage provider for new storage systems and storage capabilities.
- After pressing the buttons, you don’t see any tasks running on the vCenter Server (expected behavior). After 5 seconds everything should be working and provisioning should be possible.
VMware vCenter – PBM error occurred during PreCloneCheckCallback – Solution
Wrap-up
Thanks for reading this blog post. If you have any comments, please respond in the comment section below!
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
Accept and close Read More
Storage vMotion is a feature used to change the Virtual Machine Location without a down time and this PBM Error occurred while I was trying to Storage vMotion one of my Virtual Machines. I tried with a different Virtual Machine and it was not possible and, I was realized none of the Virtual Machines were not possible to perform the Storage vMotion and SDRS recommendations were failing.
Manual Storage vMotion wizard had a SDRS Faults error message at the “Ready to complete” stage as below.
Once I clicked on the error message, it displayed the full error message as below
A general system error occurred: PBM error occurred during QueryAllAssociatedProfiles: Connection refused
I was found that this was due to the service failure of the “vmware-sps” service. I checked the service status, started and restarted with the below commands
#Check Service Status
service vmware-sps status
#Start the Service
service vmware-sps start
#Re-start the service
service vmware-sps restartService status was masked and inactive (dead)
If you need to check all the list unit files and the status use below command, but this for your information purpose only and it’s nothing to do with the fix here
systemctl list-unit-files | grep vmwareHere is the similar output
Simple service starts or restart doesn’t work here and I unmasked and removed the symlink of the service and tried to start the service. Unfortunately, service start or restart didn’t work as I expected.
Here is the command to unmask and remove the symlink of the service
systemctl unmask vmware-spsHere is the output of my attempts
Finally, I performed the service control start and it started all my services and my vmware-sps service started to run
Here is the command
service-control --start Here is the output of my command
I hope this article is helpful for someone having the similar issues with the SDRS and Storage vMotion.
Click to rate this post!
[Total: 6 Average: 5]
Aruna Lakmal
Associate Technical Specialist, Sri Lanka. Technology junky, enthusiast, a VMware vExpert and a blogger with more than 8 years of Experience in Virtualization and Cloud Native technologies.
Get Updates Directly To Your Inbox!
Hi All ,
I came across this issue today, vCenter server started to perform very slowly and the Inventory service was taking a very long time to start.
There was no specific issues , and it all started so suddenly to my surpise.
It’s a vCenter 5.5 , were vCenter and IS are installed on the same machine, with vCenter connecting to external DB.
After further digging on logs , got below messages
ds.log
RROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl] Store eror processing atom feed:
com.vmware.vim.query.server.store.exception.StoreException: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
at com.vmware.vim.query.server.store.impl.SessionPoolImpl.getSession(SessionPoolImpl.java:165)
at com.vmware.vim.query.server.store.impl.StoreImpl.updateVmomiPullAtomFeed(StoreImpl.java:2105)
Caused by: com.xhive.error.XhiveException: SERVER_TERMINATED: The server has terminated, Original message: Not enough storage is available to process this command
at xDB_9_0_11.uB.zd(xdb:366)
vpxd.log
2015-04-14T21:43:36.221-04:00 [10176 error ‘VmProv’ opID=task-internal-1-69b2e9b1-c-1d-3-af] [CallbackManager] Got exception while invoking precheck on PbmCallBack: Invalid response code: 503 Service Unavailable
–> backtrace[00] rip 000000018018b6aa
–> backtrace[01] rip 0000000180105288
–> backtrace[02] rip 000000018010668e
–> backtrace[03] rip 00000001800900f8
–> backtrace[04] rip 00000000005910d5
Also during the same time the vmotions were failing with pbmfault errors
There were no issues with drive space or DB at all.
Checked the Heap memory (wrapper.conf) for the services and it was looking fine as well.
Thus I tried the below steps , as it was failing with pbmfault and I though it must be Profile driven storage space.
Did the below steps :
1. Stop Profile driven storage service.
2. Goto C:Program FilesVMwareInfrastructureProfile-Driven Storageconflog4j.properties
change the values of MaxFileSize and MaxBackupIndex as below.
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=50
3. Delete all the logs for PRofile Driven Storage, or take a backup (we took a backup on destop)
4. Restart the Profile-Driven Storage service.
5. Restart the Inventory service
Wow , everything started looking normal and it started working fine.
It seems like somehow the log file disk only for Profile driven storage was getting full.
Happy Troubleshooting 🙂