Corba transient error

Symptoms

Pleks Automation tasks fail with CORBA errors:

• CORBA/TIMEOUT

• CORBA/TRANSIENT

• CORBA/COMM_FAILURE

Examples of errors:

1. Destination host 'server1.example.com' (#32), IP '192.0.2.2' :
   The remote server is temporarily down. Please, make sure that destination host is accessible from POA management node and POA agent is running there.
   
   Details: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
   OMG minor code (2), described as 'No usable profile in IOR.', completed = NO
   

2. Destination host 'server1.example.com' (#32), IP '192.0.2.2' :
   Communication failure with remote server. Please, make sure that destination host is accessible from POA management node and POA agent is running there.
   
   Details: system exception, ID 'IDL:omg.org/CORBA/COMM_FAILURE:1.0'
   TAO exception, minor code = 1 (failed to recv request response; ETIMEOUT), completed = MAYBE
   

3. Request has been timed out, details:
   
   system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0'
   TAO exception, minor code = 79 (timeout during recv; low 7 bits of errno: 121 No error), completed = MAYBE
   

4. Operation execution is aborted by timeout. Please increase timeout for this operation and try again.
   Details: system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0' TAO exception, minor code = 3e (timeout during recv; low 7 bits of errno:  62 Timer expired), completed = MAYBE
   

Cause

The tasks fail as the result of various communication problems between the PA Management Node and managed hosts.

PA uses CORBA (Common Object Request Broker Architecture) technology to communicate with managed hosts through the PA Agent.

NOTICE: /etc/init.d/pem — This service starts a lots of instances of the process pleskd. This process is responsible for all PA functionality. Also several instances of pleskd process are responsible for PA agent which performs communication with PA agents on Service Nodes.

The PA Management Node sends requests to managed hosts to perform required provisioning operations using CORBA requests to the PA Agent, which must be installed on any host registered in and managed by PA.

In the event of problems sending requests to or receiving responses from the PA Agent, whatever the cause, background tasks may fail in PA with CORBA errors.

The most likely causes of communication problems between the PA Management Node and the PA Agent working on managed hosts are.

• The managed host is down

• The PA Agent has stopped on the managed host or is stuck

• The PA Management Node cannot establish a connection to the PA Agent on the managed host due to problems with routing or a firewall

Resolution

Make sure the PA Management Node can establish a connection to the PA Agent working on the managed host, send a request to the PA Agent, and receive a response from it.

Follow the troubleshooting steps:

1. Restart the failed task. If the problem with connectivity between the PA Management Node and the PA Agent was temporary, the task will be processed. If the task fails with a CORBA error after restarting, follow the next steps.

2. Find the hostname and/or IP address of the problem host. This is mentioned in the failed task in the PA Task Manager as the destination host.

3. Make sure the problem host is online:

   • Make sure the host does not experience any performance issues (if the PA task is one that takes a long time to run, its execution time may go beyond the timeout limit on a slow server)

   • Check if the host is connected to network

   • Try to log in to the host’s local console

   • Make sure there is enough diskspace on the affected host

If the managed server is down, powered off, or stuck, reboot it. Then, once the managed server is online, restart the failed task in PA.

1. Make sure the pem service is up and running on the problem host:

   Linux:

   # /etc/init.d/pem status
   pleskd is running
   

   Windows:

   C:UsersAdministrator>net start pem
   The requested service has already been started.
   

2. Restart the pem service on the managed host:

   Linux:

   # /etc/init.d/pem start
   Starting pleskd:                                           [  OK  ]
   

   Windows:

   C:UsersAdministrator>net start pem
   The PEM service is starting.
   The PEM service was started successfully.
   

Restart the failed task in PA.

Important: The managed host will be unavailable for provisioning tasks during a PA Agent restart. This means the host will be unavailable for new service provisioning and for tasks to modify/reconfigure/remove existing customers’ services. Customer services such as websites or mailboxes will not be affected by a PA Agent restart.

Step 6. Make sure that communication ports are open for connections between the PA Management Node and all managed hosts in the PA-managed environment.

Use the telnet utility on the PA Management Node to check that the necessary ports are open (i.e. to check if the PA Management Node can connect to the PA Agent):

telnet 192.0.2.2 8352

Restart the failed task in PA.

Use the tcpdump utility on Linux and the Network Monitor or WireShark utilities on the Windows PA Management Node and/or managed host to find out which port(s) the PA Management Node is trying to connect to when a PA task is running, then open the necessary ports.

Internal content