Veeam error unhandled exception was thrown during licensing process

Veeam R&D Forums

Technical discussions about Veeam products and related data center technologies

Sizing and task number

Sizing and task number

Post by matteu » Oct 16, 2021 7:54 am this post

I would like to know how I can size correctly my customer infrastructure.

Total VM : arround 400

There is 2 different room (1 and 2).
On room 1 there is VBR (on VM) + 3 ESX to save.
On room 2 there is 2 ESX + 2 other ESX.

There is 2 vcenter : 1 with 3 (on cluster1) + 2 remote (cluster B) and an other vcenter for 2 other.

The 5 host in the same vcenter use the same storage array
The 2 others ESX are on dedicated storage (datacore).

Room 1 is backuped on Hardened linux repository with 12 task
Room 2 is backup on second hardened linux repository with 12 task.

I use NBD as transport mode because network is 10Gb.
I installed 1 proxy / ESX with 4 task for each one

On each backup, my bottleneck is the source (99%) and then my target with 60%.

Backup is arround 400MB/s processing and sometimes it’s only 70MB/s probably because there is some process on the VM external to the backup.

I have backup copy job with mirroring but it doesn’t start until my backup job finish.

I would like to know :
How could I «improve» performance with task number on proxy and repository ? I choose them randomly here.
Do I have to use different schedule for all my backup job ? Actually, they all start at the same time.
When veeam is backuping my vcenter, it can’t backup any other VM because it can’t connect to vcenter to process them (I suppose) I have error about licence ([Error Unhandled exception was thrown during licensing process]) and this error was only for few VM and then all other are working perfectly.

I use per vm backup job.

I can give all the details you need, I just would like to understand how can I maximize performance for this customer.
Thanks for your help.

Re: Sizing and task number

Post by soncscy » Oct 17, 2021 7:11 pm 1 person likes this post

> On each backup, my bottleneck is the source (99%) and then my target with 60%.

So the host can’t handle the load, which is a little surprising for me (4 tasks is not that much in my opinion!) and I think you maybe want to triple check performance. Remember, NBD has some limits and if these NICs are shared for other VMware management operations, even with the improvements in 7.0u1 for vSphere, you’ll be gated a bit. 400 MB/s is pretty good for NBD, but it can stream faster with dedicated backup networks, which I’m guessing you have.

I’d start with checking for usual things like open snapshots in the environment, but maybe check the number of tasks running in Veeam when the performance dips.

Also, you might get even better performance by switching to hotadd and putting the hotadd proxy on the same 10Gbit network if you can.

Re: Sizing and task number

Post by matteu » Oct 17, 2021 8:44 pm this post

Thanks for your answer.
I was on 460MB/s. The nic is normally dedicated for management and vmware is 6.5 or 6.7 (I don’t remember but I will check version and usage tomorow).

There is total of 5 proxy with 4 task for each on the same storage = 20 tasks.
There is 2 proxy with 4 task on the second storage (datacore) = 8 tasks.

400MB would be excellent but I need to see why it go down to 80MB/s then. I will try to check on vsphere component usage (cpu / disk) but without veeamone it will not be easy

I don’t this there are snapshot on the VM but I ll check it !
There is the same task number when performance is good or not :/

Hot add has lot of issue on lot of customer I had with virtual disk stay connected on proxy and it’s lot of wasted phone time to help customer and solve it.
I didn’t choose how this should be implemented, it was pre sales company but I understand perfectly his choice.

Finally, how do you «manage» scheduling when you have backup + backup copy X 2 ? I’m not sure I did the good choice here.

Re: Sizing and task number

Post by jamcool » Oct 18, 2021 12:41 am this post

Nice diagram. . I have a similar setup with about 30 ESXi 7U2 host in each datacenter with about 700 TB of datastores in each datacenter too.

I also use NBD, with 10 Gbps networking all around and get maybe 600 MBps range if nothing else is going on. When it gets down to just a few VMs backing up (maybe less than 10), it can drop performance to under 50 MBps. Just limitations with what ESXi will allow using NBD (as mentioned). There are some TIDs about multi-threading NBD you may want to Google. I had tried it on a test server but did not see any real improvement.

As far as proxy go, we do 4 vCPU and set the threads to 8. I have about 1 proxy (virtual/windows) per 3 ESXi server. With NBD, it does not use that much CPU. If you were doing virtual (hot-add), then it would consume a lot of more CPU. We moved to NBD because Hot-add would not always release the disk and we found that with NBD and doing just incremental it is much faster.

You may already be doing this as part of best practices but make sure your repository is REFS (windows) or XFS (Linux) and you are doing Synthetic Full on a weekly basis. This way only having to do incremental from VMWARE and less impact on your VMWARE servers.

For VMWARE, make sure DRS is setup to move VMs around to load balance the cluster.

One tip, for those first backups that will be full on large system (TBs), do virtual transport on proxy if you can as it is great performance for large amount of storage and then when done, switch proxy servers back to NBD.

Источник

Veeam R&D Forums

Technical discussions about Veeam products and related data center technologies

Since VC 7 Update — VC «hang» caused by Veeam Backups

Since VC 7 Update — VC «hang» caused by Veeam Backups

Post by mkretzer » Feb 17, 2021 6:26 am this post

This might be a pure VMware issue — but i wonder if anyone else in the forum has this:

We upgraded from VC 6.7 to VC 7u1c end of december and now 7u1d. Since the upgrade about every week (but sometimes every 2 days) our VCenter process «hangs» during the backup. VCenter is still reachable via SSH and there is no problem with the VC VM itself or any other VMs in the environment. In the service status the «profile driven storage» service has issues. The problem solves itself, most of the time after 30 minutes, sometimes after 60 minutes. We also can cleanly reboot VC via SSH without issues if we need to have it fixed faster. Veeam gets «Unhandled exception was thrown during licensing process » in the backups, all retry after the problem have no issue.

This *only* happens in the evening when a backup of about

2800 of our VMs is running (this takes about 6 1/2 hours). All these backups use NBD. It never happened in the night when we backup the rest of our machines via hot-add (

The only thing we changed was the VC version. VMware has ruled out most external problems (host problems of the hosts VC was running on and so on). We increased RAM allocation for different VC processes. We reduced backup concurrency. No luck so far.

Источник

The OptionKey Blog

The OptionKey is a Technology blog. How to tips and useful documentation on Apple, Microsoft, Open Source Project.

Friday, August 21, 2020

Veeam licence error when trying to replicate or backup

My organization uses Veeam as our main backup utility. It really is a great piece of software but it does have some quirky things about it as everything does. I do think though this is a bit of an outlier to be fair. I have 2 production servers that need to be release and support was no longer available for it. We were hoping that they would last until early next year but the main production server just had one of it’s drives in a RAID 10 die so we decided to fail everything over to the backup server and make that the production server.

After permanently failing over the virtual servers, I was trying to setup a new replication job to make the production server the backup. I reverse my source and destination for my backup jobs, and tried to run my backups and replication and was getting an error.

VM $myVM task has finished with ‘Failed’ state.

Task details: Failed to process replication task Error: Unable to process workload: license has been exceeded

Even though I have 2 servers the Veeam licence is clearly a one way street, and backing up from the first server to the second server is fine, backing up from the second server to the first one is not. If I had 4 core licencing instead of 2 this probably would have worked since these are both 2 socket servers. Regardless of this minor issue, I still want to setup a failover to the server with the bad drive, because stuff happens.

So to reset the license to enable backup and replication,

Go Menu -> License

Select the «Sockets» tab

Select the server -> Then Revoke.

Once you do this and you update your backup and replication the license will be applied to the other server and your backups and replications will begin working again.

Источник

Veeam R&D Forums

Technical discussions about Veeam products and related data center technologies

[ID# 03429116] After upgrade 9.5.0.1922 to 9.5u4 — error starting veeam.backup.shell

[ID# 03429116] After upgrade 9.5.0.1922 to 9.5u4 — error starting veeam.backup.shell

Post by stevenfoo » Feb 23, 2019 2:31 am this post

We tested an upgrade on a test server before performing on the production server.

However in our test, after upgrading from 9.5.0.1922 to 9.5.4.2615.Update4 we tried to start the veeam backup shell (GUI).

We end up with the following error and we cannot start the GUI now.

Re: [ID# 03429116] After upgrade 9.5.0.1922 to 9.5u4 — error starting veeam.backup.shell

Post by nielsengelen » Feb 24, 2019 3:35 pm this post

[MERGED] event : CLR20r3 veeam.backup.shell.exe Stopped Working

Post by artgineer » Apr 04, 2019 1:18 pm this post

I have successfully updraded 9.5.0.1922 with the update 4a iso as recommended.

After rebooting, I cannot start the Backup and Replication Console.

This is the error I get :

It’s impossible for me to export the logs, as a I can’t start the console.

I have opened a support case : ID# 03497428

Re: [ID# 03429116] After upgrade 9.5.0.1922 to 9.5u4 — error starting veeam.backup.shell

Post by foggy » Apr 04, 2019 4:05 pm this post

Re: [ID# 03429116] After upgrade 9.5.0.1922 to 9.5u4 — error starting veeam.backup.shell

Post by fedor » Apr 26, 2019 2:36 pm this post

Resolved a very similar error by reinstalling .NET 4.6 on the backup server

VBR version: 9.5.4.2615
OS: Windows 2008 SP 2

Problem signature :
Problem Event Name: CLR20r3
Problem Signature 01: veeam.backup.shell.exe
Problem Signature 02: 9.5.4.2615
Problem Signature 03: 5c45f544
Problem Signature 04: PresentationCore
Problem Signature 05: 4.6.81.0

Who is online

Users browsing this forum: No registered users and 11 guests

• Main
• All times are UTC
• Delete cookies
• Members
• The team
• Contact us

DISCLAIMER: All feature and release plans are subject to change without notice.

Powered by phpBB® Forum Software © phpBB Limited

Источник

Veeam R&D Forums

Technical discussions about Veeam products and related data center technologies

Veeam Backup & Replication Console Fails to Open

Veeam Backup & Replication Console Fails to Open

Post by johnb23 » Nov 08, 2020 3:03 am this post

Hi,
I have been running Veeam Backup & Replication v9 Community for around 6 months and recently the Console Fails to Open.

I tried reinstalling the console, then reinstalling the product, then upgrading to version 10a. I have also checked that the services are running (SQL Server service + Veeam Services), ran Windows updates, ran Veeam’s «Configure Database Connection Settings», and just going next > next > finish to apply the settings.

But none of this has solved the issue. The system is Windows Server 2012R2.

Re: Veeam Backup & Replication Console Fails to Open

Post by Egor Yakovlev » Nov 08, 2020 8:04 am this post

Please provide a support case ID for this issue, as requested when you click New Topic, otherwise we will have to play a guessing game here.
Does it give any error message? Check console logs under C:ProgramDataVeeamBackup

Re: Veeam Backup & Replication Console Fails to Open

Post by johnb23 » Nov 08, 2020 9:27 am this post

Hi Egor,
Thanks for your reply. Since posting this I have spent several hours investigating and discovered Windows Event Log entries as per below:

Faulting application name: veeam.backup.shell.exe, version: 10.0.1.4854, time stamp: 0x5f171f15
Faulting module name: KERNELBASE.dll, version: 6.3.9600.19724, time stamp: 0x5ec5262a
Exception code: 0xe0434352
Fault offset: 0x0000000000007afc
Faulting process id: 0x2604
Faulting application start time: 0x01d6b5802e48b267
Faulting application path: C:Program FilesVeeamBackup and ReplicationConsoleveeam.backup.shell.exe
Faulting module path: C:Windowssystem32KERNELBASE.dll
Report Id: 6c2bee16-2173-11eb-8144-20474784743e
Faulting package full name:
Faulting package-relative application ID:

I have also lodged a support ticket (case number is 04483683).

In terms of logs. I have checked these four directories which are under C:Program FilesVeeam and couldn’t find any logs.
Backup and Replication
Backup File System VSS Integration
Endpoint Backup
Veeam Distribution Service

Re: Veeam Backup & Replication Console Fails to Open

Post by soncscy » Nov 08, 2020 9:47 am this post

Program data , not Files

I just checked my lab installation and there’s a Backup and Replication folder there, which I suppose you need. Maybe the setup one too, but who knows, better more than less right?

Also, anything lead up to that message in your event log?

Re: Veeam Backup & Replication Console Fails to Open

Post by johnb23 » Nov 09, 2020 8:49 am this post

Hi,
I have checked and could not find ProgramDataVeeamBackup under C drive, but did find the directory under F Drive. I navigated all the way down to F:ProgramDataVeeamBackupIRCacheNfsDatastore but no files existed. (I checked to ensure that the directories are showing hidden files).

In regards to the event log — these are the three events posted under the Application event log when I attempt to open Veeam Backup & Replication Console

Re: Veeam Backup & Replication Console Fails to Open

Post by johnb23 » Nov 09, 2020 8:54 am this post

Источник