Windows api call waitformultipleobjects returned error code 575

Please note: this solution is considered to be obsolete

Errors:

Error occurred during LocalDB instance startup: SQL Server process failed to start 

Veeam.Backup.Common.CSqlException: Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
Please wait, and try again later. 

Preparation:

1. Before you go, check:
— Your antivirus is not blocking the installation process.
— Microsoft SQL Server 2012 Express LocalDB is installed. If you have any installation issues — the best option is to install the version included in the setup
— SYSTEM accout has full access permissions to systemprofile folder C:WindowsSystem32configsystemprofile

2. Download, install and open PS Tools:
— Download PSTools: http://download.sysinternals.com/files/PSTools.zip
— Install it, for ease of use it could be unpacked to C:PSTools
— Go to Run > cmd as administrator
— cd C:PSTools (or the folder you used for during unpacking)
— Run PsExec.exe –i –s cmd.exe (accept the user agreement message may pop up )
3. Uninstall Veeam Endpoint Backup BETA via Control Panel > Programs and Features

Resolutions:

Case A: Instance was corrupted during setup and must be deleted:

1. Download, install and open PS Tools like described in preparation step 2
2. Delete corrupted instance by typing the following commands in PS Tools command prompt:

whoami 
//Response nt authoritysystem
 “C:Program FilesMicrosoft SQL Server110ToolsBinnSqlLocalDB.exe” delete VeeamEndPoint 
//Response: “LocalBD instance “VeeamEndPoint” deleted”

3. Reboot and to install Veeam Endpoint Backup.

Case B: The Case A solution did not help. Instance creation continues to fail during Endpoint setup.

1. Install the following components (reboot if required for each component):
— Microsoft System CLR Types for SQL Server 2012 (is included in Endpoint setup)
— Microsoft SQL Server 2012 Management Objects (is included in Endpoint setup)
— Microsoft SQL Server 2012 Express LocalBD (is included in Endpoint setup)
— Microsoft SQL Server 2012 Native Client
— Microsoft SQL Server 2012 Command Line Utilities

2. Add the following string values for SQL instance login/password in the registry via regedit.exe

 HKEY_LOCAL_MACHINESOFTWAREVeeamVeeam Endpoint Backup
//the following string values generated by QA
"SqlLogin"="VeeamLogin"
"SqlPassword"="75f00cdb-d8db-45ab-8024-8cc576df15b1"

3. Download, install and open PS Tools like described in preparation step 2
4. Re create the instance by typing the following commands in PS Tools command prompt:

 whoami 
//Response nt authoritysystem
 “C:Program FilesMicrosoft SQL Server110ToolsBinnSqlLocalDB.exe” delete VeeamEndPoint 
//Response: “LocalBD instance “VeeamEndPoint” deleted”
SqlLocalDB create VeeamEndPoint 
//Response: “LocalBD instance “VeeamEndPoint” created with version 11.0.”
SqlLocalDB start VeeamEndPoint 
//Response: “LocalBD instance “VeeamEndPoint” started”
SqlLocalDB info VeeamEndPoint 
//Response example (12345678 is an example – you will have other output): “Instance pipe name: np:\.pipeLOCALDB#12345678tsqlquery”
//Copy np:\.pipeLOCALDB#12345678tsqlquery from the previous query
sqlcmd -S np:\.pipeLOCALDB#12345678tsqlquery 
// Command prompt cursor will change to 1>
// Type in each line and hit enter one by one
USE [master]
GO
CREATE LOGIN [VeeamLogin] WITH PASSWORD=N'75f00cdb-d8db-45ab-8024-8cc576df15b1', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
GO
EXEC master..sp_addsrvrolemember @loginame = N'VeeamLogin', @rolename = N'sysadmin'
GO
EXIT
// Command prompt cursor revert back to normal 
SqlLocalDB stop VeeamEndPoint 
// Response “LocalBD instance “VeeamEndPoint” stopped”

Additionally:
— Microsoft SQL Server 2012 Native Client and Microsoft SQL Server 2012 Command Line Utilities can be found here: http://www.microsoft.com/en-US/download … x?id=29065

If the LocalDB and sqlcmd fail:
During LocalDB and sqlcmd installation path variables are automatically pushed into the system. If SqlLocalDB or sqlcmd commands fail, you may need to type a full path instead:
— SqlLocalDB should be replaced by “%Program Files%Microsoft SQL Server110ToolsBinnSqlLocalDB.exe” (with quotes)
— sqlcmd should be replaced by “%Program Files%Microsoft SQL Server110ToolsBinnSqlCmd.exe” (with quotes)

Still having installation issues?
Collect and share:
— The full setup logs from C:ProgramDataVeeam
— Windows Event logs from windows event viewer (Custom View > Administrative events)
— LocalDB error log from %systemroot%system32configsystemprofileAppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesVeeamEndPoint (it should look like error.log, error1.log, error2.log etc)
— Pack and send it via any available cloud storage (Dropbox/Google Drive/OneDrive etc)
— Ping me via PM or posting to this thread

I have a WPF application which is connecting to the SQL Server 2014 Express LocalDB (i.e. a local .mdf database file).

The application uses the EntityFramework.Core library version 7.0.0-rc1-final.

This is the connection string that I am using:
data source=(LocalDb)InstanceLocalDB; Initial Catalog = MyAppDB; Integrated Security=True; MultipleActiveResultSets=True; Connection Timeout = 10

I do the installation of the SQL Server LocalDB during the application’s setup. The same goes for the creation and starting of the SqlLocalDB Instance, which is performed with these command lines:
SqlLocalDB.exe create «SqlLocalDBInstance»
SqlLocalDB.exe start «SqlLocalDBInstance»

The app runs successfully on approximately 50 PCs, but fails on two as follows:

At first, the application successfully communicates with the database, but after awhile, if there is no activity, the instance is stopping automatically (after about 10 minutes), and from then it fails to reconnect again, which result in these following error:

«your sql server installation is either corrupt or has been tampered with (unknown package id).»

On starting the LocalDB instance manually using the same command line mentioned above (SqlLocalDB.exe start «SqlLocalDBInstance») the application successfully communicates with the DB, until the next time the LocalDB instance is stopping, and so on.
 

Windows logs gives this following error:

«Windows API call WaitForMultipleObjects returned error code: 575. Windows system error message is: {Application Error}
The application was unable to start correctly (0x%lx). Click OK to close the application.»

SQL LocalDB log files provided this notification message:

«The RANU instance is terminating in response to its internal time out. This is an informational message only. No user action is required.»

I have installed and re-installed all over again (both Microsoft SQL Server 2014 Express as well as deleted the .mdf and .ldf files), and keep getting the same error.

I noticed that in both problematic machines the UAC was turned off; so I tried to reproduce the issue on test machines with the UAC turned off — there wasn’t any reproduction of the problem.    

The 2 problematic machines runs under Windows 10 pro 64bit.

Comments

Use a minimal amount of commands for MSTest to try and narrow down what causes #7.

Restore use of the custom .runsettings file, but comment-out the Code Coverage data collector as it causes the test run to fail as described by #7.

Issue #7 appears to be caused by issues with SQL LocalDB itself when running in a context where the Code Coverage profiler is enabled. Workaround this for this repo by using the <Include> element to add the assembly under test to code coverage, rather than using <Exclude> to remove the test and AppVeyor assemblies. This means that SQL LocalDB itself isn't covered, resolving the inability to start the SQL Server process.
Further investigation is needed to determine exactly what module(s) cause the start-up issue and then excluding them so anyone else experiencing the same problem can exclude those module(s).

Issue #7 appears to be caused by cryptbase.dll being instrumented when code coverage is enabled.

***** EDIT 3/29/2022 *****As per /u/WorstTimeline, version 2.1.1.0 has corrected this issue. Recommend you test before deploying in production. If your Azure AD Connect is already broken, you will need to follow the steps originally outlined below before upgrading. Additional information here —

https://docs.microsoft.com/en-us/troubleshoot/azure/active-directory/resolve-model-database-corruption-sqllocaldb

*** EDIT 4/6/2022 *** Have been running 2.1.1.0 in a few environments for a week. All quiet. Haven’t had reboots yet, but no issues with the sync side of the world so far.

*** ***

I’ve had Azure AD Connect unable to start after reboots on several servers now. All since the upgrade to v2, continuing through version 2.0.89. Completely at random — sometimes reboots are fine, sometimes not.

Event log shows a SQL error —

Windows API call WaitForMultipleObjects returned error code: 575. Windows system error message is: {Application Error} The application was unable to start correctly (0x%lx). Click OK to close the application. Reported at line: 3714.

Only way I could find to fix was to reinstall AADC. Was finally able to get Microsoft rep on this off-hours when it happened. Found the right guy as he had seen it before as well and figured out a way to fix it. If you run in to this, do the following —

Copy the MODEL db and transaction log files from C:Program FilesMicrosoft SQL Server150LocalDBBinnTemplates to either

C:UsersADSyncxxxxx$AppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesADSync2019.

—or—

C:WindowsServiceProfilesADSyncAppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesADSync2019 (Thanks TinyBerry2)

Overwrite the existing files. Service will then start.

You may find the service is stuck starting over and over. Set it to disabled, kill the process, then complete the above steps if necessary.

As for an RCA —

«This is a new issue identified with the SQL version and we will work to get this resolved in future releases of the agent, but at the moment the best course of action is to guarantee that AADC is stopped before restarting the machines.»

IMPORTANT UPDATE: Microsoft released Azure AD Connect 2.1.1.0 on March 24th, 2022 which fixes the issue described in this blog post). You can read about it here Azure AD Connect: Version release history | Microsoft Docs The fun thing is they wrote a doc about how to fix it on March 25th, 2022. The best option is to upgrade to AD Connect 2.1.1.0 or higher.

IMPORTANT UPDATE 2: Upgrade to version 2.1.15.0 (or higher) as that version also addresses LocalDB corruption issues!
Introduction

On Windows Server 2019 and Windows Server 2022 running AD Connect v2, I have been seeing an issue since October/November 2021 where Microsoft Azure AD Sync service fails to start – event id 528. It does not happen in every environment, but it does not seem to go away when it does. It manifests clearly by the Microsoft Azure AD Sync service failing to start after a reboot. If you do application-consistent backups or snapshots, you will notice errors related to the SQL Server VSS writer even before the reboot leaves the Microsoft Azure AD Sync service in a bad state. All this made backups a candidate for the cause. But that does not seem to be the case.

In the application event log, you’ll find Event ID 528 from SQLLocalDB 15.0 with the below content.

Windows API call WaitForMultipleObjects returned error code: 575. Windows system error message is: {Application Error}
The application was unable to start correctly (0x%lx). Click OK to close the application.
Reported at line: 3714.

So, what does one do? Well, a Veeam Vanguard turns to Veeam and restores the VM from a restore point that a recent known good AD Connect installation.

But then the issue comes back

But then it comes back. Even worse, the AD Connect staging server suffers the same fate. So, again, we restore from backups. And guess what, a couple of weeks later, it happens again. So, you rebuild clean AD Connect VMs, and it happens again. We upgraded to every new version of AD Connect but no joy. You could think it was caused by failed updates or such, but no.

The most dangerous time is when the AD Connect service restarts. Usually that is during a reboot, often after monthly patching.

Our backup reports a failure with the application consistent backup of the AD Connect Server, often before Azure does so. The backup notices the issues with LocalDB before the AD Sync Service fails to start due to the problems.

However, if you reboot enough, you can sometimes trigger the error. No backups are involved, it seems. That means it is not related to Veeam or any other application consistent backup. The backup process just stumbles over the LocalDB issue. It does not cause it. The error returns if we turn off application-consistent backups in Veeam any way. We also have SAN snapshots running, but these do not seem to cause the issue.

We did try all the tricks from an issue a few years back with backing up AD Connect servers. See https://www.veeam.com/kb2911 but even with the trick to prevent the unloading of the user profile
COM+ application stops working when users logs off – Windows Server | Microsoft Docs we could not get rid of the issue.

So backups, VSS, it seems there is a correlation but not causation.

What goes wrong with LocalDB

After a while, and by digging through the event and error logs of a server with the issue, we find that somehow, the model.mdf and model.ldf are toast for some inexplicable reason on a pseudo regular basis. Below you see a screenshot from the C:WindowsServiceProfilesADSyncAppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesADSync2019Error.log. Remember your path might differ.

You’ll find entries like “The log scan number (37:218:29) passed to log scan in database ‘model’ is not valid. This error may indicate data corruption or that the log file (.ldf) does not match the data file (.mdf).”

Bar restoring from backup, the fastest way to recover is to replace the corrupt model DB files with good ones. I will explain the process here because I am sure some of you don’t have a recent, good know backup.

Sure, you can always deploy new AD Connect servers, but that is a bit more involved, and as things are going, they might get corrupted as well. Again, this is not due to cosmic radiation on a one-off server. Now we see it happen sometime three weeks to a month apart, sometimes only a few days apart.

Manual fix by replacing the corrupt model dd files

Once you see the  SQLLocalDB event ID 528 entries in the application logs when your Microsoft Azure AD Sync service fails to start, you can do the following. First, check the logs for corruption issues with model DB. You’ll find them. To fix the problem, do the following.

Disable the Microsoft Azure AD Sync service. To stop the service that will hang in “starting” you will need to reboot the host. You can also try and force kill ADSync.exe via its PID

Depending on what user account the AD Sync Service runs under, you need to navigate to a different path. If you run under NT SERVICEADSync you need to navigate to

C:WindowsServiceProfilesADSyncAppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesADSync2019

If you don’t use the default account but another one, you need to go to C:Users YOURADSyncUSERAppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesADSync2019

Open a second explorer Windows and navigate to C:Program FilesMicrosoft SQL Server150LocalDBBinnTemplates. From there, you copy the model.mdf and modellog.ldf files and paste those in the folder you opened above, overwriting the existing, corrupt model.mdf and model.ldf files.

You can now change the Microsoft Azure AD Sync service back to start automatically and start the service.

If all goes well, the Microsoft Azure AD Sync service is running, and you can synchronize to your heart’s content.

Conclusion

If this doesn’t get resolved soon, I will automate the process. Just shut down or kill the ADSync process and replace the model.mdf and model.ldf files from a known good copy.

Here is an example script, which needs more error handling but wich you can run manually or trigger by monitoring for event id 528 or levering Task Scheduler. As always run this script in the lab first. Test it, make sure you understand what it does. You are the only one responsible for what you run on your server! Once you are done testing replace Write-Host with write-output or turn it into a function and use cmdletbinding and param to gain write-verbose if you don’t want all the output/feedback. Bothe those options are more automation friendly.

cls
$SQLServerTemplates = "C:Program FilesMicrosoft SQL Server150LocalDBBinnTemplates"
$ADConnectLocalDB = "C:WindowsServiceProfilesADSyncAppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesADSync2019"

Write-Host -ForegroundColor Yellow "Setting ADSync startup type to disabled ..."
Set-Service ADSync -StartupType Disabled

Write-Host -ForegroundColor Yellow "Stopping ADSync service  ..."
Stop-Service ADSync -force

$ADSyncStatus = Get-Service ADSync

if ($ADSyncStatus.Status -eq 'Stopped'){
    Write-Host -ForegroundColor Cyan "The ADSync service has been stopped  ..."
}
else {
    if ($ADSyncStatus.Status -eq 'Stopping' -or $ADSyncStatus.Status -eq 'Starting'){
        
        Write-Host -ForegroundColor Yellow "Setting ADSync startup type to disabled ..."
        Set-Service ADSync -StartupType Disabled

        Write-Host -ForegroundColor Red "ADSync service was not stopped but stuck in stoping or starting ..."
        $ADSyncService = Get-CimInstance -class win32_service | Where-Object name -eq 'ADSync'
        $ADSyncProcess = Get-Process | Where-Object ID -eq $ADSyncService.processid

        #Kill the ADSync process if need be ...
        Write-Host -ForegroundColor red "Killing ADSync service processs forcfully ..."
        Stop-Process $ADSyncProcess -Force

        #Kill the sqlserver process if need be ... (in order to be able to overwrite the corrupt model db files)
        Write-Host -ForegroundColor red "Killing sqlserver process forcfully ..."
         $SqlServerProcess = Get-Process -name "sqlservr" -ErrorAction SilentlyContinue
         if($SqlServerProcess){
        Stop-Process $SqlServerProcess -Force}

        }
    }

$ADSyncStatus = Get-Service ADSync
if ($ADSyncStatus.Status -eq 'Stopped'){

    Write-Host -ForegroundColor magenta "Copy known good copies of model DB database to AD Connect LocaclDB path file ..."
    Copy-Item "$SQLServerTemplatesmodel.mdf" $ADConnectLocalDB

    Write-Host -ForegroundColor magenta "Copy known good copy of model DB log file to AD Connect LocaclDB path ..."
    Copy-Item "$SQLServerTemplatesmodellog.ldf" $ADConnectLocalDB


    Write-Host -ForegroundColor magenta "Setting ADSync startup type to automatic ..."
    Set-Service ADSync -StartupType Automatic

    Write-Host -ForegroundColor magenta "Starting ADSync service ..."
    Start-Service ADSync
}

$ADSyncStatus = Get-Service ADSync
if ($ADSyncStatus.Status -eq 'Running' -and $ADSyncStatus.StartType -eq 'Automatic'){
    Write-Host -ForegroundColor green "The ADSync service is running ..."
}
else {
    Write-Host -ForegroundColor Red "ADSync service is not running, something went wrong! You must trouble shoot this"
}

That fixes this cause for when Microsoft Azure AD Sync service fails to start – event id 528. For now, we keep an eye on it and get alerts from the AD Connect health service in Azure when things break or when event id occurs on the AD Connect servers. Let’s see if Microsoft comes up with anything.

IMPORTANT UPDATE: Microsoft released Azure AD Connect 2.1.1.0 on March 24th 2022 which fixes the issue described in this blog post). You can read about it here Azure AD Connect: Version release history | Microsoft Docs The fun thing is the wrote a doc about how to fix it on March 25th 2022. The best option is top upgrade to AD Connect 2.1.1.0 or higher.

PS: I am not the only one seeing this issue Azure AD Sync Connect keeps getting corrupted – Spiceworks