-
TitaniumCoder477
- Service Provider
- Posts: 252
- Liked: 35 times
- Joined: Apr 07, 2015 1:53 pm
- Full Name: James Wilmoth
- Contact:
Socket has been forcibly closed
Veeam Backup & Replication 9.5.0.1038
Background:
We were using a copy job to send backups across a 100/100 VPN connection to a NAS.
The VMs were copying OK, but the diff merge kept taking wayyyyyy too long. I think the job was attempting to pull the data back across the tunnel to perform the merge locally.
So two days ago:
We deployed an Ubuntu Server 16.04.3 LTS server (a VM) in our COLO.
I mounted the NAS share via NFS.
I added the Linux server in Veeam and created a Linux repo mapping to that NFS path.
I removed the copy jobs from DB, rescanned the new repo, and then mapped the job to them.
Issue:
The job runs flawlessly for 15 minutes.
Then the copying VMs fail with «Error: Socket has been forcibly closed. Failed to recover connection.»
The pending VMs eventually fail as well (without copying any data).
Things I have tried:
Dropping from 4 simultaneous tasks to 1; no change in symptom.
Throttling; no change in symptom.
Increasing RAM in Linux VM from 4 to 8GB; no change in symptom.
Monitored a continuous ping test across VPN to Linux VM and to NAS repo; also monitored NFS mount; no issues even as symptom occurred religiously.
I opened a ticket with Veeam and submitted logs. Still waiting for a solid indication of the root cause and resolution.
Any ideas?
This is my NFS mount command, just in case someone asks:
192.168.5.20:/volume1/Veeam /media/backups nfs hard,intr,nfsvers=3,tcp,bg,_netdev,auto,nolock 0 0
-
foggy
- Veeam Software
- Posts: 20911
- Liked: 2063 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Socket has been forcibly closed
Post
by foggy » Nov 07, 2017 4:11 pm
Hi James, please provide the support case ID for this issue. Thanks.
-
TitaniumCoder477
- Service Provider
- Posts: 252
- Liked: 35 times
- Joined: Apr 07, 2015 1:53 pm
- Full Name: James Wilmoth
- Contact:
Re: Socket has been forcibly closed
Post
by TitaniumCoder477 » Nov 08, 2017 12:02 am
This is the case number 02368996.
The source-side firewall is a Dell SonicWall. The destination is a Cisco router. Today I did comb through the SonicWall GUI’s VPN settings and tweak a few things based on information I found online. Unfortunately, that didn’t affect the symptom so I set the variables back to what they were. I also reviewed the logs and noticed something there that I brought to our network engineer’s attention, something possibly related to the two routers negotiating.
The odd thing is that when I run a continuous ping test from B&R server to Linux VM and also to repo in destination, the ping tests are virtually flawless. Low latency. Maybe one packet dropped every couple hundred etc.
Historically, the symptom has occurred right at about 15 minutes. But once today it continued to about 30 minutes and then did the same thing.
-
TitaniumCoder477
- Service Provider
- Posts: 252
- Liked: 35 times
- Joined: Apr 07, 2015 1:53 pm
- Full Name: James Wilmoth
- Contact:
Re: Socket has been forcibly closed
Post
by TitaniumCoder477 » Nov 08, 2017 1:31 pm
I ran the job again this morning while monitoring the Linux agent logs. The job was running fine for about 15 minutes and then the connection died and this showed up:
Code: Select all
Every 1.0s: tail -n 5 Agent.Copy_job.Target_cli.log Wed Nov 8 08:31:09 2017
[08.11.2017 08:23:46] <140394612381440> cli| - 14%, workload rps: 99/1/0, client: {381a4508-bd7f-5a43-9228-a15d876ffc84}
[08.11.2017 08:23:57] <140394612381440> cli| - 14%, workload rps: 99/1/0, client: {381a4508-bd7f-5a43-9228-a15d876ffc84}
[08.11.2017 08:24:08] <140394612381440> cli| - 14%, workload rps: 99/1/0, client: {381a4508-bd7f-5a43-9228-a15d876ffc84}
[08.11.2017 08:24:17] <49617024> cli| WARN|Input stream was closed.
[08.11.2017 08:24:17] <49617024> cli| ERR |Signal [1] received. Raising signal for default handling.
What I don’t understand is why in Veeam B&R it says, «11/8/2017 8:30:13 AM :: Processing [vm name xyz] Error: Socket has been forcibly closed. Failed to recover connection,» when if I just right-click > Sync Now and start a new cycle, it runs fine for another 15 minutes or so. Why does the job just hang? What is it waiting for?
-
nitramd
- Expert
- Posts: 296
- Liked: 83 times
- Joined: Feb 16, 2017 8:05 pm
-
Contact:
Re: Socket has been forcibly closed
Post
by nitramd » Nov 08, 2017 2:32 pm
Wonder if it would be a port issue on one of the machines… See this link for Veeam port usage https://helpcenter.veeam.com/docs/backu … tml?ver=95
Have you checked the event logs on the Windows VBR server? Firewall on same server? I’ve had somewhat similar issues with some machines in my environment. One thing I’ve learned is to check the firewall on a machine.
You can use an interesting tool called cryping, it allows you to ping a port — runs on Windows. I’ve found it to be useful.
Hope this helps.
-
TitaniumCoder477
- Service Provider
- Posts: 252
- Liked: 35 times
- Joined: Apr 07, 2015 1:53 pm
- Full Name: James Wilmoth
- Contact:
Re: Socket has been forcibly closed
Post
by TitaniumCoder477 » Nov 08, 2017 9:33 pm
CryPing was actually VERY helpful and a new tool in my toolbox—thanks!!
So here is what I did:
- Configured Notepad++ to refresh automatically
- Started CryPing.exe -p 22 -t 192.168.5.21 > results22.txt
- Started another CryPing.exe -p 2500 -t 192.168.5.21 > results2500.txt (since I could see in the log below that 2500 was the negotiated port)
- Opened these two files and C:ProgramDataVeeamBackupCopy_jobJob.Copy_job.BackupSync.log in Notepad++ (with silent refresh on etc)
- SSH’d to Linux backup repo server and enabled firewall and added rules to allow port 22 and 2500-5000/tcp
- On same, used Screen to split and monitor /var/log/ufw.log and Top
- Kicked off job
Results:
- Ping to SSH was always successful and remained successful throughout.
- Ping to 2500 was only successful when job started. I’m assuming this because the Linux agent only listens after it is activated by the SSH command from B&R.
- The job ran flawlessly at about 15MB/s for approximately 15 minutes. Then the connection died. See log excerpt below.
- Ping to 2500 started timing out at the same time.
- UFW on Linux backup repo did not report any denies or drops.
Conclusion: Because port 2500 only responded to pings when I started the B&R job, which leads me to believe the Linux agent then started listening, could it be that when the connection and the pings mutually died, the cause is the Linux agent? Could it be the agent died or dropped the connection for some reason? What logs should I pull to look at this symptom from that angle?
Log excerpt from C:ProgramDataVeeamBackupCopy_jobJob.Copy_job.BackupSync.log. This excerpt starts right after one VM successfully copied and the next one started. Identifying characters have been replaced with XYZ etc.:
Code: Select all
[08.11.2017 15:33:19] <01> Info [RemoteBackupTaskBuilder] Skipping VM XYZ-DC1 processing: completed
[08.11.2017 15:33:20] <01> Info [NewRemoteOibAction] About to process source Oib (id='dd3539d0-564f-4b92-8f22-59b8dde294c2', CreationTime='11/7/2017 10:33:29 PM', Point='\XYZ-SYN-001VeeamBackup of Host1Backup of Host1D2017-11-07T220047.vbk') into the target storage: '50879984-6da7-441e-9bf5-f1436b3e090b'
[08.11.2017 15:33:20] <01> Info [ViTaskBuilder] Found source Oib (Storage: '\XYZ-SYN-001VeeamBackup of Host1Backup of Host1D2017-11-07T220047.vbk', CreationTime: '11/7/2017 10:33:29 PM') for VM 'XYZ-Print'
[08.11.2017 15:33:20] <01> Info Task total size: 60.0 GB
[08.11.2017 15:33:20] <01> Info Task progress: '0%', '0' of '64,424,509,440' bytes, object '0' of '0'
[08.11.2017 15:33:20] <01> Info Task total size: 60.0 GB
[08.11.2017 15:33:20] <01> Info Task progress: '0%', '0' of '64,424,509,440' bytes, object '0' of '0'
[08.11.2017 15:33:20] <01> Info [RemoteBackupTaskBuilder] Preparing new task for execution. XYZ-Print
[08.11.2017 15:33:20] <01> Info [IdleStateVmController] Stopping idle
[08.11.2017 15:33:20] <01> Info [TaskExecutor] Preparing task XYZ-Print
[08.11.2017 15:33:20] <01> Info [TaskRepositoryResourceHelper] Adding target repository resource request to task XYZ-Print
[08.11.2017 15:33:20] <01> Info [Id: 1db0ba92-3466-4569-990a-d034a75679c9, Type: vbk, State: Existing, IsSlotNeeded: True, Access: Read, ObjectId: <no object id>]
[08.11.2017 15:33:20] <01> Info [TaskRepositoryResourceHelper] Adding source repository resource request to task XYZ-Print
[08.11.2017 15:33:20] <01> Info [Id: 1db0ba92-3466-4569-990a-d034a75679c9, Type: vbk, State: Existing, IsSlotNeeded: True, Access: Read, ObjectId: <no object id>]
[08.11.2017 15:33:20] <01> Info [TaskRepositoryResourceHelper] Adding file sessions resource request for 43 files to task XYZ-Print
[08.11.2017 15:33:20] <01> Info Job size: 16.8 GB (16.8 GB provisioned).
[08.11.2017 15:33:20] <01> Info [TaskExecutor] Task XYZ-Print prepared
[08.11.2017 15:33:20] <01> Info [ParallelTaskExecutor] Starting task XYZ-Print execution
[08.11.2017 15:33:20] <01> Info Waiting for backup infrastructure resources availability
[08.11.2017 15:33:28] <07> Info [BackupMetaUpdater] Generating backup meta because it was requested by the following threads:
[08.11.2017 15:33:28] <07> Info [BackupMetaUpdater] Thread <24> CPL-DSM
[08.11.2017 15:33:28] <07> Error [JobSession] LastUserCryptoSpec is null. Cannot find the user key by the last storage.
[08.11.2017 15:33:28] <07> Info Created mutex [GlobalBackupMeta[0e5a6d41-f687-43d8-ad41-ea05f214fbbe]]
[08.11.2017 15:33:28] <07> Info Acquired mutex [GlobalBackupMeta[0e5a6d41-f687-43d8-ad41-ea05f214fbbe]]
[08.11.2017 15:33:28] <07> Info [IncrementalMetaGenerator] Generating meta for backup [88a5f8f3-7411-4a96-820d-fbde0cb7680a], usn [2342835]
[08.11.2017 15:33:28] <07> Info [BackupMeta] Saving backup meta to '|media|backups|seed|Seed job for COLO.vbm'
[08.11.2017 15:33:28] <07> Info [Ssh] Check existing '/media/backups/seed'
[08.11.2017 15:33:28] <07> Info [Ssh] Saving binary to '/media/backups/seed/Seed job for COLO.vbm_4_tmp'.
[08.11.2017 15:33:29] <07> Info [Ssh] Check existing '/media/backups/seed/Seed job for COLO.vbm'
[08.11.2017 15:33:29] <07> Info [Ssh] Check existing '/media/backups/seed/Seed job for COLO.vbm'
[08.11.2017 15:33:29] <07> Info [Ssh] Deleting "/media/backups/seed/Seed job for COLO.vbm"
[08.11.2017 15:33:29] <07> Info [Ssh] Check existing '/media/backups/seed/Seed job for COLO.vbm_4_tmp'
[08.11.2017 15:33:29] <07> Info [Backup] SetMetaUpdateTimeAndVersion, old value: [11/8/2017 3:33:13 PM,3021] new value: [11/8/2017 3:33:29 PM,3022]
[08.11.2017 15:33:29] <07> Info Releasing mutex [GlobalBackupMeta[0e5a6d41-f687-43d8-ad41-ea05f214fbbe]]
[08.11.2017 15:33:29] <07> Info [BackupMetaUpdater] Backup meta generated
[08.11.2017 15:33:43] <40> Info [AP] (3202) command: 'Invoke: Network.UpdateThrottlingn{n (EInt32) KbpsWriteBandwidth = 5760n}n'
[08.11.2017 15:33:43] <29> Info [AP] (3202) output: <VCPCommandResult result="true" exception="" />
[08.11.2017 15:33:43] <31> Info [AP] (3202) output: <VCPCommandArgs />
[08.11.2017 15:33:43] <31> Info [AP] (3202) output: >
[08.11.2017 15:39:46] <15> Error (SSH Transport) error: ChannelError: An existing connection was forcibly closed by the remote host
[08.11.2017 15:39:46] <15> Error [AP] (Client) error: An existing connection was forcibly closed by the remote host
[08.11.2017 15:39:46] <15> Error (Client) error: ChannelError: An existing connection was forcibly closed by the remote hostAn existing connection was forcibly closed by the remote hostChannelError: An existing connection was forcibly closed by the remote host
[08.11.2017 15:39:47] <48> Error [ReconnectableSocket][PacketAsyncReceiver] Exception on [b93e3bab-647e-4d36-8a0c-f3b6aae7eb29].
[08.11.2017 15:39:47] <48> Error Unexpected socket shutdown. (System.Exception)
[08.11.2017 15:39:47] <48> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.PushData(CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:47] <48> Info [ReconnectableSocket] Nonretryable error on [b93e3bab-647e-4d36-8a0c-f3b6aae7eb29].
[08.11.2017 15:39:47] <48> Info [ReconnectableSocket] Reconnecting [b93e3bab-647e-4d36-8a0c-f3b6aae7eb29] (shutdown notification)...
[08.11.2017 15:39:47] <48> Info [NetSocket] Connect
[08.11.2017 15:39:47] <44> Error [ReconnectableSocket][PacketAsyncReceiver] Exception on [aac8507e-72bf-4412-9d64-3e490eaa53f7].
[08.11.2017 15:39:47] <44> Error Unexpected socket shutdown. (System.Exception)
[08.11.2017 15:39:47] <44> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.PushData(CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:47] <44> Info [ReconnectableSocket] Nonretryable error on [aac8507e-72bf-4412-9d64-3e490eaa53f7].
[08.11.2017 15:39:47] <44> Info [ReconnectableSocket] Reconnecting [aac8507e-72bf-4412-9d64-3e490eaa53f7] (shutdown notification)...
[08.11.2017 15:39:47] <44> Info [NetSocket] Connect
[08.11.2017 15:39:48] <48> Error Failed to connect to agent's endpoint '192.168.5.21:2500'. Host: '192.168.5.21'.
[08.11.2017 15:39:48] <48> Error No connection could be made because the target machine actively refused it 192.168.5.21:2500 (System.Net.Sockets.SocketException)
[08.11.2017 15:39:48] <48> Error at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[08.11.2017 15:39:48] <48> Error at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
[08.11.2017 15:39:48] <48> Error at Veeam.Backup.Common.CNetSocket.Connect(IPEndPoint remoteEp)
[08.11.2017 15:39:48] <48> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)
[08.11.2017 15:39:48] <44> Error Failed to connect to agent's endpoint '192.168.5.21:2500'. Host: '192.168.5.21'.
[08.11.2017 15:39:48] <44> Error No connection could be made because the target machine actively refused it 192.168.5.21:2500 (System.Net.Sockets.SocketException)
[08.11.2017 15:39:48] <44> Error at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[08.11.2017 15:39:48] <44> Error at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
[08.11.2017 15:39:48] <44> Error at Veeam.Backup.Common.CNetSocket.Connect(IPEndPoint remoteEp)
[08.11.2017 15:39:48] <44> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)
[08.11.2017 15:39:49] <44> Warning [ReconnectableSocket] Failed to recover [aac8507e-72bf-4412-9d64-3e490eaa53f7].
[08.11.2017 15:39:49] <44> Error Reconnect operation was aborted. (Veeam.Backup.Common.CRegeneratedTraceException)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.CExceptionUtil.RegenTraceExc(Exception originalExc, String formatString, Object[] args)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CNetworkReconnectContext.WaitOnRetry(TimeSpan interval, Exception firstChanceEx)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.RunConnectCycle()
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.DoReconnect()
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.PerformReconnectSafe()
[08.11.2017 15:39:49] <44> Error No connection could be made because the target machine actively refused it 192.168.5.21:2500 (System.Net.Sockets.SocketException)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.Connect(IPEndPoint& connectedEndpoint)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.AgentProvider.CReconnectableSessionStarter.Reconnect(CAgentEndpointConnecter connecter, Boolean redirectOutput)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.RunConnectCycle()
[08.11.2017 15:39:49] <48> Warning [ReconnectableSocket] Failed to recover [b93e3bab-647e-4d36-8a0c-f3b6aae7eb29].
[08.11.2017 15:39:49] <44> Error Reconnect operation was aborted. (Veeam.Backup.Common.CRegeneratedTraceException)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.CExceptionUtil.RegenTraceExc(Exception originalExc, String formatString, Object[] args)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CErrorPort.TryRegenExc(Exception ex)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CInterlockedReconnectEngine.UnlockAfterAccess()
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.PushData(CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.OnRawDataReceived(Object sender, CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:49] <44> Error Unexpected socket shutdown. (System.Exception)
[08.11.2017 15:39:49] <44> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.PushData(CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:49] <48> Error Reconnect operation was aborted. (Veeam.Backup.Common.CRegeneratedTraceException)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.CExceptionUtil.RegenTraceExc(Exception originalExc, String formatString, Object[] args)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CNetworkReconnectContext.WaitOnRetry(TimeSpan interval, Exception firstChanceEx)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.RunConnectCycle()
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.DoReconnect()
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.PerformReconnectSafe()
[08.11.2017 15:39:49] <48> Error No connection could be made because the target machine actively refused it 192.168.5.21:2500 (System.Net.Sockets.SocketException)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.Connect(IPEndPoint& connectedEndpoint)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.AgentProvider.CReconnectableSessionStarter.Reconnect(CAgentEndpointConnecter connecter, Boolean redirectOutput)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CReconnectEngine.RunConnectCycle()
[08.11.2017 15:39:49] <48> Error Reconnect operation was aborted. (Veeam.Backup.Common.CRegeneratedTraceException)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.CExceptionUtil.RegenTraceExc(Exception originalExc, String formatString, Object[] args)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CErrorPort.TryRegenExc(Exception ex)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CInterlockedReconnectEngine.UnlockAfterAccess()
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.PushData(CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.OnRawDataReceived(Object sender, CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:49] <48> Error Unexpected socket shutdown. (System.Exception)
[08.11.2017 15:39:49] <48> Error at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.PushData(CSocketAsyncEventArgsEx e)
[08.11.2017 15:39:49] <15> Info [AP] (1309) state: closed-
nitramd
- Expert
- Posts: 296
- Liked: 83 times
- Joined: Feb 16, 2017 8:05 pm
- Contact:
Re: Socket has been forcibly closed
Post
by nitramd » Nov 15, 2017 2:35 pm
It looks like the socket closure is happening ~6 minutes into the job based on the log file. You might find within one of the Linux log files «connection reset» — you can grep for it. If you find this string, look around at other close by entries to see if there is any indication as to what is causing the issue.
Any word from Support?
-
TitaniumCoder477
- Service Provider
- Posts: 252
- Liked: 35 times
- Joined: Apr 07, 2015 1:53 pm
- Full Name: James Wilmoth
- Contact:
Re: Socket has been forcibly closed
Post
by TitaniumCoder477 » Nov 17, 2017 6:28 pm
1 person likes this post
RESOLVED!
We increased the SonicWall TCP timeout from 20 minutes to 60 minutes and ran the job. It ran for a full on hour before timing out! That coincided with SW renewal of support, so today our network engineer jumped on a call with SW support who clarified that TCP packets contain timeout. Apparently even though data is streaming across port 2500, the SW is not seeing a keep alive packet so basically thinks the connection is dead. So at the timeout, it drops the connection. We increased the TCP timeout value to 1440 minutes, and the job is still running at 3 1/2 hours.
I really appreciate everyone who has contributed to figuring this out. Veeam support was right in the end regarding the source of the problem, but I am thankful that everyone was willing examine and consider other possibilities.
thanks for your replies
the bpclntcmd is working fine from bothe direction, I checked for network configuration everything seems ok
now for the logs from client bpcd :
10:34:39.796 [3152.3520] <2> bpcd main: offset to GMT -10800
10:34:39.812 [3152.3520] <2> bpcd main: Got socket for input 356
10:34:39.812 [3152.3520] <2> logconnections: BPCD ACCEPT FROM 1**.17.0.2.585 TO 1**.17.15.16.13782
10:34:39.828 [3152.3520] <2> bpcd main: setup_sockopts complete
10:34:39.859 [3152.3520] <2> vauth_acceptor: ..libvlibsvauth_comm.c.356: no methods for address: no authentication required
10:34:39.859 [3152.3520] <2> bpcd main: no authentication required
10:34:39.859 [3152.3520] <2> bpcd peer_hostname: Connection from host srv02-backup (1**.17.0.2) port 585
10:34:39.859 [3152.3520] <2> bpcd valid_server: comparing Srv02-Backup and srv02-backup
10:34:39.859 [3152.3520] <4> bpcd valid_server: hostname comparison succeeded
10:34:40.328 [3152.3520] <2> bpcd main: output socket port number = 665
10:34:40.328 [3152.3520] <2> bpcd peer_hostname: Connection from host srv02-backup (1**.17.0.2) port 585
10:34:40.328 [3152.3520] <2> bpcd main: Peer hostname is srv02-backup
10:34:40.343 [3152.3520] <2> nb_bind_on_port_addr: bound to port 995
10:34:40.343 [3152.3520] <2> bpcd main: Got socket for output 1932, lport = 995
10:34:40.359 [3152.3520] <2> bpcd main: Connected on output socket
10:34:40.359 [3152.3520] <2> bpcd main: Duplicated socket on stderr
10:34:40.359 [3152.3520] <2> bpcd main: <—- NetBackup 5.1 0 ————initiated
10:34:40.359 [3152.3520] <2> bpcd main: Not using VxSS authentication with srv02-backup
Interface: 1**.17.15.16 — 0x2
Internet Address Physical Address Type
1**.17.0.2 00-30-05-81-27-41 dynamic
10:34:41.218 [3152.3520] <2> bpcd main:
10:34:41.218 [3152.3520] <2> bpcd main: BPCD_GET_VERSION_RQST
10:34:41.218 [3152.3520] <2> bpcd main: BPCD_READ_HOST_CONFIG_RQST
10:34:41.531 [3152.3520] <2> bpcd main: BPCD_GET_PLATFORM_RQST
10:34:41.750 [3152.3520] <2> bpcd main: BPCD_GET_VERSION_RQST
10:34:41.953 [3152.3520] <2> bpcd main: BPCD_GET_NB_VERSION_RQST
10:34:41.953 [3152.3520] <4> bpcd main: Version string is NetBackup 5.1 510000 «C:Program FilesVERITASNetBackupbin»
10:34:42.140 [3152.3520] <2> bpcd main: BPCD_GET_UNAME_RQST
10:34:42.156 [3152.3520] <2> getUnameInfo: UNAME_SYSNAME = WindowsXP
10:34:42.156 [3152.3520] <2> getUnameInfo: UNAME_NODENAME = D15-16
10:34:42.156 [3152.3520] <2> getUnameInfo: UNAME_RELEASE = 5
10:34:42.156 [3152.3520] <2> getUnameInfo: UNAME_VERSION = 1
10:34:42.156 [3152.3520] <2> getUnameInfo: UNAME_MACHINE = Intel x86 — Unknown — Revision x304
10:34:42.156 [3152.3520] <2> getUnameInfo: END
10:34:42.343 [3152.3520] <2> bpcd main: BPCD_GET_VERSION_RQST
10:34:42.546 [3152.3520] <2> bpcd main: BPCD_READ_TEXT_FILE_RQST
10:34:42.546 [3152.3520] <2> read_text_file: .bpcd_util.c.3046: file_id = 3
10:34:42.546 [3152.3520] <2> read_text_file: .bpcd_util.c.3097: file_name = <C:Program FilesVERITASNetBackupshareversion_crypt.txt>
10:34:42.843 [3152.3520] <2> bpcd main: BPCD_DISCONNECT_RQST
10:34:42.843 [3152.3520] <2> bpcd exit_bpcd: exit status 0 ————>exiting
10:34:43.093 [2556.640] <2> bpcd main: offset to GMT -10800
10:34:43.109 [2556.640] <2> bpcd main: Got socket for input 352
10:34:43.109 [2556.640] <2> logconnections: BPCD ACCEPT FROM 1**.17.0.2.1018 TO 1**.17.15.16.13782
10:34:43.109 [2556.640] <2> bpcd main: setup_sockopts complete
10:34:43.140 [2556.640] <2> vauth_acceptor: ..libvlibsvauth_comm.c.356: no methods for address: no authentication required
10:34:43.140 [2556.640] <2> bpcd main: no authentication required
10:34:43.140 [2556.640] <2> bpcd peer_hostname: Connection from host srv02-backup (1**.17.0.2) port 1018
10:34:43.156 [2556.640] <2> bpcd valid_server: comparing Srv02-Backup and srv02-backup
10:34:43.156 [2556.640] <4> bpcd valid_server: hostname comparison succeeded
10:34:43.156 [2556.640] <2> bpcd main: output socket port number = 856
10:34:43.156 [2556.640] <2> bpcd peer_hostname: Connection from host srv02-backup (1**.17.0.2) port 1018
10:34:43.156 [2556.640] <2> bpcd main: Peer hostname is srv02-backup
10:34:43.171 [2556.640] <2> nb_bind_on_port_addr: bound to port 972
10:34:43.171 [2556.640] <2> bpcd main: Got socket for output 1932, lport = 972
10:34:43.171 [2556.640] <2> bpcd main: Connected on output socket
10:34:43.171 [2556.640] <2> bpcd main: Duplicated socket on stderr
10:34:43.171 [2556.640] <2> bpcd main: <—- NetBackup 5.1 0 ————initiated
10:34:43.171 [2556.640] <2> bpcd main: Not using VxSS authentication with srv02-backup
Interface: 1**.17.15.16 — 0x2
Internet Address Physical Address Type
1**.17.0.2 00-30-05-81-27-41 dynamic
10:34:43.453 [2556.640] <2> bpcd main:
10:34:43.468 [2556.640] <2> bpcd main: BPCD_FORK_CMD_RQST
10:34:43.468 [2556.640] <2> bpcd main: fork cmd = /usr/openv/netbackup/bin/bpfis bpfis create -nbu -owner NBU -id d15-16_1217403269 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 13 -rg other -class TESt -sched Cumulative-Inc -fso -S srv02-backup -WOFB -fim VSP
10:34:43.468 [2556.640] <2> bpcd main: filter type = -1
10:34:43.468 [2556.640] <2> bpcd main: Convert args to CreateProcess format
10:34:43.468 [2556.640] <2> bpcd main: Done converting args to CreateProcess format
10:34:43.468 [2556.640] <2> bpcd main: new fork cmd = C:Program FilesVERITASNetBackupbinbpfis.exe create -nbu -owner NBU -id d15-16_1217403269 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 13 -rg other -class TESt -sched Cumulative-Inc -fso -S srv02-backup -WOFB -fim VSP
10:34:43.468 [2556.640] <2> bpcd main: Before CreateProcess
10:34:43.468 [2556.640] <2> bpcd main: StdOutput assigned the value STDOUTSOCK
10:34:43.500 [2556.640] <2> bpcd main: After CreateProcess, pid = 2700
10:34:43.500 [2556.640] <2> bpcd exit_bpcd: exit status 0 ————>exiting
10:34:43.734 [1428.3332] <2> bpcd main: offset to GMT -10800
10:34:43.765 [1428.3332] <2> bpcd main: Got socket for input 336
10:34:43.781 [1428.3332] <2> logconnections: BPCD ACCEPT FROM 1**.17.0.2.673 TO 1**.17.15.16.13782
10:34:43.781 [1428.3332] <2> bpcd main: setup_sockopts complete
10:34:43.812 [1428.3332] <2> vauth_acceptor: ..libvlibsvauth_comm.c.356: no methods for address: no authentication required
10:34:43.812 [1428.3332] <2> bpcd main: no authentication required
10:34:43.828 [1428.3332] <2> bpcd peer_hostname: Connection from host srv02-backup (1**.17.0.2) port 673
10:34:43.828 [1428.3332] <2> bpcd valid_server: comparing Srv02-Backup and srv02-backup
10:34:43.828 [1428.3332] <4> bpcd valid_server: hostname comparison succeeded
10:34:43.828 [1428.3332] <2> bpcd main: output socket port number = 953
10:34:43.828 [1428.3332] <2> bpcd peer_hostname: Connection from host srv02-backup (1**.17.0.2) port 673
10:34:43.828 [1428.3332] <2> bpcd main: Peer hostname is srv02-backup
10:34:43.843 [1428.3332] <2> nb_bind_on_port_addr: bound to port 881
10:34:43.843 [1428.3332] <2> bpcd main: Got socket for output 1932, lport = 881
10:34:43.843 [1428.3332] <2> bpcd main: Connected on output socket
10:34:43.843 [1428.3332] <2> bpcd main: Duplicated socket on stderr
10:34:43.843 [1428.3332] <2> bpcd main: <—- NetBackup 5.1 0 ————initiated
10:34:43.843 [1428.3332] <2> bpcd main: Not using VxSS authentication with srv02-backup
-
MiniTool
-
MiniTool News Center
- An Existing Connection Was Forcibly Closed by the Remote Host
By Sonya | Follow |
Last Updated November 28, 2022
What will you do if you get the “An existing connection was forcibly closed by the remote host” error? If you have no idea to deal with it, then this post from MiniTool is what you need. You can find several useful methods in this post.
It is very convenient to manage remote computers using a sort of computer network. But sometimes, you will get an error message stating that “An existing connection was forcibly closed by the remote host” when you try to connect the remote host. Then why does this error occur and how to fix it?
Causes of an Existing Connection Was Forcibly Closed by the Remote Host in Windows
There are several causes of an existing connection was forcibly closed by the remote host, which have been listed below:
- TLS 1.1/1.0 Usage: If applications are running on TLS 1.1 or TLS 1.0, this error may be triggered because they have been depreciated. TLS 1.2 is the method when selecting the protocol used by the application.
- Cryptography Disabled: If your computer has cryptography disabled, it will prevent the use of TLS 1.2 and will fall back to TLS 1.0, which may cause the error.
- Socket Implementation: In some cases, a specific type of socket implementation can cause the error.
- Missing Code: If you are using Entity Framework, some lines of code missing will trigger this error.
- Outdate “.NET” Framework: Sometimes, this error may be triggered if the “.NET” framework has been disabled. Some tasks require the “.NET” framework to be updated to the latest version for them to work normally.
How to Fix an Existing Connection Was Forcibly Closed by the Remote Host?
After knowing some causes of the “an existing connection was forcibly closed” error, then this part will offer you some methods to fix the “connection forcibly closed by remote host” error.
Method 1: Enable Cryptography
You can enable Cryptography in Registry Editor to solve the issue. Here is a quick guide:
Tip: You’d better back up your Registry keys before you editing Registry Editor.
Step 1: Press the Win + R keys at the same time to open the Run box.
Step 2: Type regedit in the box and then click OK to open Registry Editor.
Step 3: Go to the following path:
HKEY_LOCAL_MACHINESOFTWAREWow6432NodeMicrosoft.NETFrameworkv4.0.30319.
Step 4: With the v4.0.30319 key selected in the left panel, double-click SchUseStrongCryto in the right panel and change the Value data to 1. Click OK to save changes.
Step 5: Restart your computer and then check if the problem that an existing connection was forcibly closed by the remote host is solved.
Method 2: Change Socket Implementation
An error or failure in a socket implementation may prevent some elements of the application from functioning properly, which will cause the “An existing connection was forcibly closed by the remote host” error. Thus, you can change socket implementation to solve the problem.
Step 1: Make sure you have a StateObject class with public byte[] buffer = new byte[1024], public Socket socket.
Step 2: Call the Receive(Socket s) function and call the following code in void ReceiveCallback(IAsyncResult ar):
SocketError errorCode;
int nBytesRec = socket.EndReceive(ar, out errorCode);
if (errorCode != SocketError.Success)
{
nBytesRec = 0;
}
Step 3: After implementing this code, restart your computer to check if the error is fixed.
Method 3: Add Command Lines (Only for Entity Framework)
You may lose some certain lines of code if you are using the Entity Framework, so you can try to add command lines to solve the problem that “An existing connection was forcibly closed by the remote host”.
Step 1: Open your .edmx file and open the .context.tt file below it.
Step 2: Open the .context.cs file and add the following command lines to your constructor:
public DBEntities()
: base(«name=DBEntities»)
{
this.Configuration.ProxyCreationEnabled = false; // ADD THIS LINE !
}
Step 3: After you have added the command lines, reboot your computer to check if
Method 4: Update .NET Framework
If you don’t get the latest version of the .NET Framework, then you may receive the error message stating that an existing connection was forcibly closed by the remote host. Therefore, you need to update the .NET Framework to deal with the problem:
Step 1: Go to this link to download the setup.
Step 2: Run the .exe file to start the installation process, and then follow the instructions showing on the screen to finish installing the application on your computer
Step 3: Restart your computer to see whether the error is fixed.
About The Author
Position: Columnist
Author Sonya has been engaged in editing for a long time and likes to share useful methods to get rid of the common problems of Windows computers, such as Windows Update error. And she aims to help more people to protect their data. What’s more, she offers some useful ways to convert audio and video file formats. By the way, she likes to travel, watch movies and listen to music.
I modified the TestClass attributes from:
[TestFixture, Parallelizable(ParallelScope.All)]
to
[TestFixture]
Is this what you are referring to by «assembly-level parallel attribute» ?
I removed the Parallel attribute and ran the test sequentially. Test ran for about 1 hr before the Socket Exception was thrown. However, this time it appears to have captured some different information: (NOTE: This is without the —inprocess switch)
Unhandled Exception: NUnit.Engine.NUnitEngineException: Remote test agent was terminated due to a stack overflow.
at NUnit.Engine.Services.TestAgency.OnAgentExit(Process process, Guid agentId)
at NUnit.Engine.Services.TestAgency.<>c__DisplayClass7_0.b__0(Object sender, EventArgs e)
at System.Diagnostics.Process.OnExited()
at System.Diagnostics.Process.RaiseOnExited()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)
Errors, Failures and Warnings
- Error :
System.Net.Sockets.SocketException : An existing connection was forcibly closed by the remote host
—SocketException
An existing connection was forcibly closed by the remote host
Server stack trace:
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
Additional Logging file generated by the —trace=Verbose option:
NUnit Logs — not Parallel.zip