Veeam backup error log

General Troubleshooting: Veeam Backup Job Errors

Below are common host and backup job error messages you may encounter and what they mean.

This article is organized by Alarm Type:

1. Backup Agent Job State Errors
2. Job Session State Errors
3. Job Status Errors
4. Computer/VM Not Backed Up
5. Miscellaneous

For Additional Errors Not Listed Below, Consult the Veeam Knowledge Base and/or Forums:

• Veeam Knowledge Base –  https://www.veeam.com/kb_search_results.html
• Veeam Community Forums – https://forums.veeam.com/ 

1. Backup Agent Job State Errors

Error: write: An existing connection was forcibly closed by the remote host Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

What This Means

• The last backup job was unsuccessful because the remote host (machine backing up) closed the connection to eSilo while the backup was running.

Possible Causes

• The machine was powered down during the backup. Upon shutdown, the active snapshot taken at the start of the backup is lost.

Troubleshooting Steps

• No user intervention is typically required.  When the computer is powered back on, a new snapshot will be taken at the next scheduled backup time, and a new backup job will begin.  (Note: If the machine is put to Sleep or Standby during a backup, it should resume when it wakes up. It is only when the machine is completely powered down that this error happens).

Error:  Task failed unexpectedly

What This Means

• The job was terminated suddenly, resulting in an incomplete backup and restore point.

Possible Causes

• Host rebooted during the backup, resulting in the loss of the active snapshot.

Troubleshooting Steps

• Check host’s Event Viewer Windows System logs to see if there was a reboot.
• Rerun the backup job.

Error:  Root element missing

What This Means

• The VBM file associated with the backup job has become corrupted.

Possible Causes

• This may be caused by a connection issue during the backup job or the Veeam repository had an error.

Troubleshooting Steps

• Locate the VBM file associated with the backup job and rename the file, such as by adding “.old” to end of the file name.
• Start a new backup job to generate a new VBM file.
• After the new VBM file gets created, the old file can be safely deleted without waiting for the running job to complete.

Unable to allocate processing resources. Error: Job session with id [STRING] does not exist

What This Means

• Host requested but was not assigned processing resources by eSilo cloud connect.  This happens because the maximum number of concurrent backup jobs was already reached.

Possible Causes

• eSilo limits the number of concurrent backup jobs from a single Company when their bandwidth is lower than recommended thresholds.  This max limit ensures hosts queue their backup jobs sequentially (vs. all machines at once) and avoids network slowdowns for the customer.
• When the first few machines’ backup jobs run long, they can cause subsequent hosts’ jobs to time out while waiting for their turn to backup.

Troubleshooting Steps

• Hosts should automatically retry backups at their next scheduled interval. No user intervention is typically required.
• If this is a persistent issue, or if your hosts are connecting from multiple locations (and thus there is little concern that multiple backups may saturate a site’s connection), contact eSilo Support to request the concurrent task limit be increased for your Company.

Failed to start a backup job. Failed to perform the operation. Invalid job configuration: Connection over network is blocked by network throttling rules

What This Means

• Backup job was unable to complete due to network restrictions.

Possible Causes

• The host is on a metered connection (ex: hot spot), or on a WiFi connection that has “Set as a metered connection” toggled on in the properties for the currently active WiFi connection.  By default, eSilo Backup Jobs are set to “Disable backups over a metered connection”.
• Temporary network congestion may also cause this error.
• Firewall rules that limit or severely restrict certain types of traffic.

Troubleshooting Steps

• To check if the current WiFi connection is flagged as “metered” by Windows, the user can navigate to the Properties of their WiFi Network, and scroll to “Metered Connection” to verify if this is toggled ON. The preferred setting is to turn this OFF.  Alternatively, if you don’t have remote access to the machine, uncheck the “Disable backups over metered connection” setting in the backup job and rerun it.
  • 

• Wait until the next scheduled backup run to see if the issue persists or was temporary.
• Contact the network or IT administrator for the site to investigate if there were recent firewall rule changes or upgrades that may have introduced new blocking or throttling settings.

Unable to allocate processing resources. Error: Authentication failed because the remote party has closed the transport stream.

What This Means

• The backup job failed due to an authentication error between the client machine backing up and the eSilo infrastructure.

Possible Causes

• Client machine is behind on Windows Updates which include authentication and security enhancements, or they have not checked the box in Windows Update Settings to include updates for other Microsoft Products, such as .NET Framework.

Troubleshooting Steps

• Apply latest Windows Security updates and .NET Framework updates
• Check if at the time of the above error on the tenant side, the eSilo Svc.VeeamCloudConnect.log log file displays the following error: “A call to SSPI failed, see inner exception“.  If so, the issue may be related to a Windows Update enforcing a new .Net Framework security check. This check does not allow the client to establish a secure connection between their Veeam backup servers or agents and the eSilo Cloud Connect service, if there is a weak Diffie-Hellman Ephemeral (DHE) key. See this help article from Veeam on the steps needed to confirm and resolve.  https://www.veeam.com/kb3208
• Subsequent job reattempts may complete successfully without user intervention, although this error may still sporadically cause jobs to fail.

Error: Failed to connect to the port [DNS_Name:Port].

What This Means

• Host unable to connect to eSilo cloud connect at the specified gateway and port address.

Possible Causes

• If this issue is occurring for only one or two machines (most common), and not all machines connecting to the eSilo cloud connect infrastructure, it may be indicative of a network issue on the client side.

Troubleshooting Steps

• The host should reattempt the job at the next scheduled interval.  You can also manually start a backup to reattempt the job.  No other user intervention is typically required.

Error: Insufficient quota to complete the requested service. Asynchronous read operation failed Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Insufficient quota to complete the requested service. Asynchronous read operation failed Unable to retrieve next block transmission command. Number of already processed blocks: [#]. Failed to download disk ‘[LONG_ID]’.

What This Means

• The error description may be misleading.  We’ve observed this error previously, and it was unrelated to the Tenant’s Quota, which was well within limits.

Possible Causes

• When this was observed on an internal testing VM, the cause was that the source VM in the backup job was very low on memory (RAM) resources and was unresponsive.  Restarting the VM brought it back online, and the next automatic job try was successful.

Troubleshooting Steps

• Verify the source machine is online and responsive. Ensure resource levels look good.
• Host will reattempt job at the next scheduled interval. No user intervention is typically required.

Error: Invalid backup cache synchronization state.

What This Means

• Host is currently saving backups to Local Backup Cache.  An attempt to sync cached backups to the eSilo cloud repository failed,  due to a mismatch between what the Veeam Agent for Windows had in its local database for expected restore points and what was actually found in the repository and/or local backup cache.

Possible Causes

• If an in-progress backup is abruptly stopped, for example due to power failure, Veeam will discard any partially written restore points.  However, if all references to those now discarded restore points are not cleared from the database (which should happen automatically), this can cause a job error on the next run, which highlights a mismatch between restore points expected on disk and what was found.

Troubleshooting Steps

• In most cases where this has been observed, the job will complete successfully the next time it is run, without any user intervention.
• If this error persists more than once, contact eSilo Support for assistance troubleshooting Backup Cache issues.

Job session for “[JOB_NAME]” finished with error. Job [JOB_NAME] cannot be started. SessionId: [ID], Timeout: [XX sec]

What This Means

• The Veeam job could not start due to too many active sessions or jobs running on the host consuming all available memory.

Possible Causes

• Too many running sessions causes Veeam services to be impacted by the host’s Desktop Heap limitation. This is because the Desktop Heap size for services is much smaller than that for applications.

Troubleshooting Steps

• To resolve this issue the Desktop Heap size must be increased via a registry modification.
• See this Veeam Knowledge Base article for detailed resolution steps:  KB1909

Error: The system cannot find the file specified. Failed to open I/O device Failed to open emulated disk. Failed to open disk for read. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}

What This Means

• An expected disk on the client machine (machine to backup) was unable to be opened and read during the backup job.

Possible Causes

• Not known at this time.

Troubleshooting Steps

• The host will reattempt job at the next scheduled interval. No user intervention is typically required.

Error: Oib is complete and cannot be continued

What This Means

• The “oib” stands for “objects in backup” and is a unique identifier used by Veeam.
• The error indicates there is a discrepancy; a job is attempting to write to an oib that is already complete or finalized.

Possible Causes

• This may occur on a backup cache sync job that was interrupted right at very end, during the finalization stage.  The error indicates that the oib was finalized by the previous job (“oib is complete”), however the current job is trying to modify or append to it, which is not allowed.

Troubleshooting Steps

• This error should resolve itself on next job run.  If not, contact eSilo support.

Job session for “[JOB_NAME]” finished with error. Backup cache size has been exceeded

What This Means

• The size of the local backup cache has exceeded the maximum allowed size as configured in the backup job, preventing new restore points from being saved to the cache.

Possible Causes

• Persistent network disruptions may be preventing cached restore points from syncing, or uploading, to the eSilo Backup Cloud and thus they are not rotated out of the local backup cache.  Restore points accumulate until the backup cache location is full.
• Service provider resources are busy or unable to be allocated to this host, preventing the syncing of cached restore points.

Troubleshooting Steps

• The Backup Cache Sync should resume in time, once the originating network or resource issues are resolved.
• The maximum Backup Cache size can be increased in the job settings, so long as there is sufficient local space.
• For more detail on advanced resolution steps, see this eSilo KB article: How to Resolve Backup Cache Size Exceeded Error

Error: Failed to create snapshot: Backup job failed. Cannot create a shadow copy of the volumes containing writer’s data. A VSS critical writer has failed. Writer name: [NAME]. Class ID: [ID]. Instance ID: [ID]. Writer’s state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].

What This Means

• There is an issue with the built-in Windows VSS (Volume Shadow Copy service) on the host machine. Specifically, the VSS writer mentioned was not available at the time of the backup.
• eSilo Backups powered by Veeam use VSS writers to backup files that may be in-use, open or locked at the time of backup.  This is particularly useful for databases, allowing backups to complete without downtime.  If a writer is not in the proper state and functioning as expected, the backup snapshot will fail.  VSS writer issues must be resolved on the host, and can usually be corrected by restarting the associated service.

Possible Causes

• The VSS service and/or the VSS Providers service is disabled
• The VSS writer is not in the Stable state, indicating it is ready and waiting to perform a backup.  Below are alternative states:
  • Failed or Unstable – the Writer encountered a problem, and must be reset .
  • In-Progress or Waiting for Completion – the Writer is currently in use by a backup process. When the backup is finished, the Writer will revert to back to Stable state.  However, if you see this state when no backups are running, the Writer needs to be reset.

Troubleshooting Options

• Verify the Volume Shadow Copy and Microsoft Software Shadow Copy Provider services are not disabled in services.msc.
• Check the state of VSS Writers using the following syntax in an admin command prompt.  Also check the Windows Event Viewer for additional error information.

vssadmin list writers​

• • For the specified Writer in the error message, verify it is in a Stable state.  If not, restart the respective Service related to that writer as mentioned in the table here.  Then run the above command a second time to ensure the writer has returned to a stable state.
  • Note that Services often have dependencies on one another.  When one service is reset it may require others to be reset as well.  Restarting a service will momentarily disrupt any application services that rely on it.  For example, while resetting the MEIS service (Microsoft Exchange Information Store), MS Exchange will be unable to send and receive emails.
• A system reboot can also resolve most VSS writer problems, although it requires downtime.
• This Veeam KB article can also be useful in troubleshooting VSS issues for servers.

Job Session for [JOB_NAME] finished with error.  Job [JOB_NAME] cannot be started. SessionID: [ID], Timeout: [VALUE]

What This Means:

• The job could not start, due to timeout waiting for required Veeam resources

Possible Causes:

• The Concurrent Task limit set at the Company level is too low for the number of hosts and disks schedule to be backup within a defined backup window.

Troubleshooting Options:

• Increase the number of concurrent tasks (e.g. disks that can be processed at once).  This setting can be found in the eSilo Backup Portal under Companies >> Edit >> Bandwidth >> Max Concurrent Tasks.  The minimum value should be 2, but greater numbers may be needed based on the timing and staggering of host backups.

Job session for “[JOB_NAME]” finished with error.
Error: Service provider side storage commander failed to perform an operation: CreateStorage

What this Means:

• eSilo was not able to allocate repository storage for the backup job.

Possible Causes:

• The assigned repository quota for this Tenant has been exceeded, thus preventing new backups from initiating.

Troubleshooting Options: 

• Increase the Company’s repository quota.
• Remove existing backup chains or reduce the retention period to free space.

Job Status Warning: Unable to truncate SQL server transaction logs. Details: Failed to truncate SQL server transaction logs for instances: [MSSQLSERVER].

What this Means:

• Veeam was unable to truncate SQL server logs as specified in the job settings.

Possible Causes:

• This most commonly due to a permissions issue.

Troubleshooting Options: 

• This Veeam Helpcenter article discusses the Log Truncation settings.
• You can confirm if this is a permissions issue by reviewing the Backup Job log for Warning items. Ex:  Description = The server principal “[HOST][ACCOUNT]” is not able to access the database “[HOSTNAME]” under the current security context.
•  Grant necessary permissions and rerun the job.
• Alternatively, you can modify the Backup Job settings to not truncate SQL Logs.
  • Edit the Backup Job
  • Under Guest Processing, click to “Customize application handling options for individual applications…”
  • On the SQL tab, select the option for “Do not truncate logs”

2. Job Session State Errors (for VMs)

Host [LOCAL_IP] is not available. Error: Cannot complete login due to an incorrect user name or password. Virtual Machine [NAME] is unavailable and will be skipped from processing. Nothing to process. [#] machines were excluded from task list.

What This Means

• Veeam Backup and Replication was unable to access the Virtual Machine (VM) to perform the backup.

Possible Causes

• Incorrect user name or password specified to access the source VM.  The password may have expired or the account credentials or permissions may have been changed.

Troubleshooting Steps

• Contact the IT Administrator for the VM to troubleshoot the credentials saved in the Backup Job.

3. Job Status Errors

SQL VSS Writer is missing: databases will be backed up in crash-consistent state and transaction log processing will be skipped

What This Means

• The SQL Writer for the Windows Volume Shadow Copy Service (VSS) is not available on the host machine, or is not configured with adequate permissions.  This issue is related to the setup of the SQL database, and not specific to eSilo provided software or the backup itself.

Steps to Confirm the Issue

• Running ‘vssadmin list writers’ in an Administrator Command Prompt shows that SqlServerWriter is not in the list, or is in a State other than ‘Stable’.

Possible Causes

• The SQL instance has at least one database with name starting or ending in a space character
• The account under which SQL VSS Writer service is running doesn’t have sysadmin role on a SQL server – most frequently encountered 
• SQL VSS Writer service is stuck in an invalid state, e.g. other than ‘Stable’

Troubleshooting Steps

• Depending on a particular cause:
• Rename the database to a new name (without a space in it).  To check if your database has space in the name you can run the following query:

  select name from sys.databases where name like '% '

  If you notice any spaces in the database names, then you will need to remove the spaces from the database names.

• Grant the SQL VSS Writer service user a sysadmin role (Instructions in KB here: https://www.veeam.com/kb1978)
• Restart SQL VSS Writer service (Instructions in KB here: https://www.veeam.com/kb2041)
• In the case of SBS machines that are also Domain Controllers, ensure that the SQL Writer is running as a domain administrator and not local system.
• Allow the SQL Writer service account access to the Volume Shadow Copy service via the registry:
  HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesVSSVssAccessControl
  • If the DWORD value “NT SERVICESQLWriter” is present in this key, it must be set to 1.
  • If the Volume Shadow Copy service is running, stop it after changing this registry value. Do not disable it.

For More Information

See this Veeam Knowledgebase article: https://www.veeam.com/kb2095

4. Computer/VM  Not Backed Up Errors

Backup Agent ‘[HOSTNAME]’ has fallen out of the configured RPO interval ([#]days). Last backup: [#] days, [#] hours ago.

What This Means

• The host’s most recent eSilo cloud backup is greater than the specified RPO (Recovery Point Objective) interval.

Possible Causes

• Host has been powered off or offline.
• Backups are being saved to the host’s Local Backup Cache and all restore points in the cache have not yet synced to the eSilo Cloud Repository (e.g. eSilo has not yet received the backups).
• Veeam Backup Agent Service or Veeam Management Agent Service is not running on host.
• The backups and/or backup schedule have been manually disabled (uncommon).

Troubleshooting Steps

• Verify host is online and connected to network.
• Check Backup Job to determine if Backup Cache is enabled.  View cache folder on host to see if populated with recent restore points (default location: C:VeeamCache).
• Verify ‘VeeamManagementAgentSvc’ service is running on host.  Status should be ‘Running’ and Startup Type should be ‘Automatic (Delayed Start)’.  If the service is ‘Stopped’, check the Event Viewer for possible error details.  See this article for more troubleshooting steps.
• Verify ‘VeeamEndpointBackupSvc’ service is running on host.  Status should be ‘Running’ and Startup Type should be ‘Automatic’.  Restart or reinstall if necessary.
• Check Backup Job to verify schedule and ensure not ‘Disabled’

Other Errors – Full Details Coming Soon

Error: Reconnectable protocol device was closed. Agent failed to process method {FileBackup.SyncDirs}. Exception from server: Reconnectable protocol device was closed.

Troubleshooting Steps

• Host will reattempt job at the next scheduled interval. No user intervention is typically required.

Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Troubleshooting Steps

• Host will reattempt job at the next scheduled interval. No user intervention is typically required.

[In Backup Job Log] Error: [CStorageLinksHelper] Link Id=[LONG_ID] doesn’t exist for storage [JOBNAME_SUBTENANTNAME]yyy-mm-ddTxxxxxx.vib

What This Means

• While the restore point exists on the repository, the Link ID to that restore point in the metadata (.vbm) file is missing.
• Note: If the Backup Job is configured to use Backup Cache, this error does not by itself trigger a Backup Job Failure, since the restore points are successfully written to local cache. You will notice it however, because the eSilo Backup Portal will warn that a new restore point has not been uploaded in X days (according to the RPO alarm thresholds set for this tenant).

Possible Causes

• This can be a symptom of a network drop, where a handshake was missed in the final stages of job completion for the last restore point uploaded to eSilo. The local Veeam Agent database (on the subtenant’s machine) saw the restore point created, but the finalization step didn’t update the metadata file on the repository side with eSilo.

Troubleshooting Steps

• In the repository, we will force the job to recreate the metadata file by editing the existing metadata file to append “.old” at the end. At the next job run, this will force a recheck of all restore points in the backup chain and recreate the metadata file from that chain.

[In Svc.VeeamEndpointBackup.log] Error: Warning  [CertificateError] Validation complete with warnings, AND/OR Warning Remote certificate chain errors, AND/OR Warning   WarningRevocationStatusUnknown (The revocation function was unable to check revocation for the certificate.

What This Means

• The subtenant was unable to validate the eSilo Cloud Connect server’s certificate.

Possible Causes

• If this is happening for only one tenant, as opposed to all tenants, it suggests an issue with how this specific subtenant is connecting to eSilo.
• If no firewall or other changes have been made recently, you can recheck the credentials used by the subtenant in the backup job.
• This Veeam KB article is also helpful for investigating common causes of certificate errors: https://www.veeam.com/kb2323

Troubleshooting Steps

• Verify the Management Agent status shows as Connected.  You can force a reconnect by changing a property in the dialog box, then changing it back and clicking “Apply“.
• Pause and unpause sync of Backup Cache files by right-clicking on the Veeam Backup Agent icon in the taskbar.
• Edit the Backup Job to specify the correct sub-tenant login credentials.  Save and rerun the job. Upon the next job run, you should see “Uploading cached restore points” when you hover over the Veeam Backup Agent icon in the taskbar.

Was this article helpful?