Callback request failed due to an internal server error

Searching on a solution for the error “Webhook 500 internal server error”? We can help you.

Usually, this error indicates bad scripting, permission problems, etc.

At Bobcares, we often get requests to fix Webhook errors, as a part of our Server Management Services.

Today, let’s see how our Support Engineers fix the Webhook 500 error in the server.

Why does Webhook show internal server error?

Before getting deeper into the error, it’s important to know about Webhook. So let’s get into it.

Webhook is one of the easiest ways to set up custom callbacks. These user-defined HTTP callbacks can serve many purposes like getting real-time data, passing it on to an application, etc.

Similarly, it also processes the data and gives something in return. Like, sending welcome emails when a domain gets a new user. This reduces the task of monitoring every activity on a website.

But, errors in Webhook is tricky. One such error is 500 internal server error.

This is a very general status code. Usually, it indicates errors in the webserver. And the webserver cannot complete the user request. That is, the server is unable to identify the reason for it.

Since the webhook is customizable, the error normally occurs due to bad scripting, insecure URL requests, etc.

How we fix the Webhook 500 error?

Internal server errors are difficult to troubleshoot. However, our Support Engineers with expertise over a decade fix these errors for our customers. Let’s see a few instances of this.

Use of insecure connection

Recently, one of our customers approached us with a Webhook 500 error. He had set up a Dialogflow, and the Webhook keeps on failing. Dialogflow is a chatbot building framework.

The error message on its diagnostic info appeared as:

So, our Support Team checked his script. We found that his code used HTTP requests instead of https. Hence the URLs were not publicly accessible and this leads to the error. We corrected the code and updated the same.

At the same time, the website should also have an SSL certificate. Setting up a self-signed certificate is not enough as Dialogflow doesn’t support it.

Bad scripting

Webhook users can customize its use. But many times it ends up in errors. And the reason for this is bad scripting.

Our customers approach us for 500 errors in Webhook. Firstly, we check their scripts. If our Experts find errors we correct it and update the same. In most situations, this can fix the error.

Permission errors

Usually, file and folder permissions can result in Webhook 500 errors. So, whenever our Support Team gets requests to fix this error, we check permissions.

Server files and folders need correct permissions and ownerships set in order to work properly. If Websites requests files for which the server has no access then it will lead to an error.

[Need assistance in fixing Webhook errors? – We’ll help you.]

Conclusion

In short, webhook 500 internal server error occurs due to bad scripting, permission errors, insecure URL requests and so on. Today, we saw how our Support Engineers fix this error for our customers.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;

You’re viewing Apigee Edge documentation.
View
Apigee X documentation.

Videos

Watch the following videos to learn more about solving 500 Internal Server Errors.

Symptom

The client application gets an HTTP status code of 500 with the message
«Internal Server Error» as a response for API calls. The 500 Internal Server
error could be caused by an error during the execution of any policy within Edge or by an error
on the target/backend server.

The HTTP status code 500 is a generic error response. It means that the server encountered an
unexpected condition that prevented it from fulfilling the request. This error is usually
returned by the server when no other error code is suitable.

Error Messages

You may get the following error message:

HTTP/1.1 500 Internal Server Error

In some cases, you may observe another error message which has more details. Here is a sample
error message:

{
   "fault":{
      "detail":{
         "errorcode":"steps.servicecallout.ExecutionFailed"
      },
      "faultstring":"Execution of ServiceCallout callWCSAuthServiceCallout failed. Reason: ResponseCode 400 is treated as error"
   }
}

Possible Causes

The 500 Internal Server Error could be thrown due to a number of different causes. In Edge,
the causes can be classified into two main categories based on where the error occurred:

Execution Error in an Edge Policy

A Policy within
the API proxy may fail for some reason. This section explains how to troubleshoot the issue if
the 500 Internal Server Error occurs during the execution of a policy.

Diagnosis

Diagnostic Steps for Private and Public Cloud Users

If you have the trace UI session for the error, then:

1. Verify that the error was caused by the execution of a policy. See Determining the source of the problem for details.
2. If the error occurred during policy execution, continue.. If the error was caused by the
   backend server, go to Error in the Backend Server.
3. Select the API request that is failing with 500 Internal Server Error in the trace.
4. Examine the request and select the specific policy that has failed or the flow named
   «Error» that is immediately following the failed policy in the trace.
5. Get more details about the error either by checking the «error» field under the Properties
   section or the Error content.
6. Using the details you’ve collected about the error, try to determine its cause.

Diagnostic Steps for Private Cloud Users Only

If you don’t have the trace UI session, then:

1. Verify that the error occurred during the execution of a policy. See Determining the source of the problem for details.
2. If the error was caused by policy execution, continue. If the error occurred during policy
   execution, continue. If the error was caused by the backend server, go to Error in the Backend Server.
3. Use the NGINX access logs as explained in Determining
   the source of the problem to determine the failing policy in the API proxy and also the
   unique request message id
4. Check the Message Processor logs
   (/opt/apigee/var/log/edge-message-processor/logs/system.log) and search for the
   unique request message id in it.
5. If you do find the unique request message ID, see if you can get more information about the
   cause for the failure.

Resolution

If you have determined the cause of the issue with the policy, try to correct the problem by
fixing the policy and redeploying the proxy.

The following examples illustrate how to determine the cause and resolution for different
types of issues.

If you need further assistance in troubleshooting 500 Internal Server Error or you suspect
that it’s an issue within Edge, contact Apigee
Support.

Example 1: Failure in Service Callout policy due to an error in the backend
server

If the call to the backend server fails within the Service Callout policy with any error such
as 4XX or 5XX, then it will be treated as 500 Internal Server Error.

1. Here’s an example where the backend service fails with a 404 error within the Service
   Callout policy. The following error message is sent to the end user:

   {
   "fault":
        { "detail":
              { "errorcode":"steps.servicecallout.ExecutionFailed"
              },"faultstring":"Execution of ServiceCallout service_callout_v3_store_by_lat_lon
    failed. Reason: ResponseCode 404 is treated as error"
             }
        }
   }
   

2. The following trace UI session shows 500 status code caused due to an error in Service
   Callout policy:

   

3. In this example, the «error» property lists the reason for the Service Callout policy
   failure as «ResponseCode 404 is treated as error». This error might occur if
   the resource being accessed via the backend server URL in the Service Callout policy is not
   available.
4. Check the availability of the resource on the backend server. It might not be available
   temporarily/permanently or it might have been moved to a different location.

Example 1 Resolution

1. Check the availability of the resource on the backend server. It might not be available
   temporarily/permanently or it might have been moved to a different location.
2. Fix the backend server URL in the Service Callout policy to point to a valid and existing
   resource.
3. If the resource is only temporarily unavailable, then try making the API request once the
   resource is available.

Let’s now look at another example, where 500 Internal Server Error is caused due to an error
in the Extract Variables policy and see how to troubleshoot and resolve the issue.

1. The following trace in UI session shows 500 status code due to an error in Extract
   Variables policy:

   

2. Select the failing Extract Variables policy, scroll down and look at the «Error
   Content» section for more details:

   

3. The Error Content indicates that
   the«serviceCallout.oamCookieValidationResponse» variable is not available in
   the Extract Variables policy. As the name of the variable indicates, it should be holding the
   response of the preceding Service Callout policy.
4. Select the Service Callout policy in the trace and you might find that the
   «serviceCallout.oamCookieValidationResponse» variable was not set. This
   indicates that the call to the backend service failed, resulting in an empty response
   variable.
5. Though the Service Callout policy has failed, the execution of the policies after Service
   Callout policy continue because the «continueOnError» flag in the Service Callout policy is set
   to true, as shown below:

   <ServiceCallout async="false" continueOnError="true" enabled="true" name="Callout.OamCookieValidation">
     <DisplayName>Callout.OamCookieValidation</DisplayName>
     <Properties />
     <Request clearPayload="true" variable="serviceCallout.oamCookieValidationRequest">
       <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
     </Request>
     <Response>serviceCallout.oamCookieValidationResponse</Response>
     <HTTPTargetConnection>
       <Properties />
       <URL>http://{Url}</URL>
     </HTTPTargetConnection>
   </ServiceCallout>
   

6. Note the unique message id «X-Apigee.Message-ID» for this specific API
   request from the trace, as follows:
   1. Select the «Analytics Data Recorded» phase from the request.
   2. Scroll down and note the value of X-Apigee.Message-ID.

   

7. View the Message Processor log
   (/opt/apigee/var/log/edge-message-processor/system.log) and search for the unique
   message id noted down in step #6. The following error message was observed for the specific API
   request:

   2017-05-05 07:48:18,653 org:myorg env:prod api:myapi rev:834 messageid:rrt-04984fed9e5ad3551-c-wo-32168-77563  NIOThread@5 ERROR HTTP.CLIENT - HTTPClient$Context.onTimeout() : ClientChannel[C:]@149081 useCount=1 bytesRead=0 bytesWritten=0 age=3002ms lastIO=3002ms .onConnectTimeout connectAddress=mybackend.domain.com/XX.XX.XX.XX:443 resolvedAddress=mybackend.domain.com/XX.XX.XX.XX
   

   The above error indicates that the Service Callout policy failed due to a connection
   timeout error while connecting to the backend server.

8. To determine the cause for the connection timeout error, executed the
   telnet command to the backend server from the Message Processor(s). The telnet
   command gave «Connection timed out» error as shown below:

   telnet mybackend.domain.com 443
   Trying XX.XX.XX.XX...
   telnet: connect to address XX.XX.XX.XX: Connection timed out
   

   Typically, this error is observed under the following circumstances:

   • When the backend server is not configured to allow traffic from the Edge Message
     Processors.
   • If the backend server is not listening on the specific port.

   In the above illustrated example, though the Extract Variables policy failed, the actual
   cause was that Edge was unable to connect to the backend server in the Service Callout
   policy. And the cause for this failure was that the backend end server was not configured to
   allow traffic from the Edge Message Processors.

   Your own Extract Variables policy will behave differently and may fail for a different
   reason. You can troubleshoot the issue appropriately depending on the cause for failure of
   your Extract Variables policy by checking the message in the error
   property.

Example 2 Resolution

1. Fix the cause for error or failure in Extract Variables policy appropriately.
2. In the illustrated example above, the solution was to rectify the network configuration to
   allow the traffic from Edge Message Processors to your backend server. This was done by
   allowlisting the Message Processors’ IP addresses on the specific backend server. For example,
   on Linux, you could use iptables to allow the traffic from
   Message Processor’s IP addresses on the backend server.

Example 3: Failure in JavaCallout policy

Let’s now look at one more example, where 500 Internal Server Error is caused due to an error
in Java Callout policy and see how to troubleshoot and resolve the issue.

1. The following UI trace shows 500 status code due to an error in Java Callout Policy:

   
2. Select the Flow named «Error» followed by the failed Java Callout Policy
   to get the error details as shown in the figure below:

   

3. In this example, the «error» property under the Properties section reveals
   that the failure is due to expired password being used while connecting to the Oracle Database
   from within the JavaCallout policy. Your own Java callout will behave differently and will
   populate a different message in the error property.
4. Check the JavaCallout policy code and confirm the correct configuration that needs to be
   used.

Example 3 Resolution

Fix the Java callout code or configuration appropriately to avoid the runtime exception. In
the illustrated Java callout failure example above, one would need to use the correct password
for connecting to the Oracle database to resolve the issue.

Error in the Backend Server

A 500 Internal Server Error could also originate from the backend server. This section
explains how to troubleshoot the issue if the error comes from the backend server.

Diagnosis

Diagnostic Steps for All Users

The cause of other backend errors can vary widely. You will need to diagnose each situation
independently.

1. Verify that the error was caused by the backend server. See Determining the source of the problem for details.
2. If the error was caused by the backend server, continue. If the error occurred during
   policy execution, go to Execution Error in Edge
   Policy.
3. Follow the steps below depending on whether or not you have access to a Trace session for
   the failed API, or if the backend is a Node.js server:

If you do not have a Trace session for the failed API call:

1. If the UI trace is not available for the failing request, then check the backend server
   logs to get details about the error.
2. If possible, enable the debug mode on the backend server to get more details about the
   error and the cause.

If you do have a Trace session for the failed API call:

If you have a Trace session, then the following steps will help you diagnose the problem.

1. In the Trace tool, select the API request that has failed with 500 Internal Server
   Error.
2. Select the «Response received from target server» phase from the failing
   API request as shown in the figure below:

   
3. Check the «Response Content» section to get details about the error.

   

4. In this example, the Response Content which is a SOAP Envelope, shows the fault string as
   «Not Authorized» message. The most likely cause for this
   issue is that the proper credentials (username/password, access token, etc.) are not passed to
   the backend server by the user. This issue can be fixed by passing the correct credentials to
   the backend server.

If the backend is a Node.js server:

1. If the backend is a Node.js Backend Server, then check the Node.js logs
   for the specific API Proxy in the Edge UI (both Public and Private Cloud users can
   check the Node.js logs). If you are an Edge Private Cloud user, you
   can also check your Message Processor logs
   (/opt/apigee/var/log/edge-message-processor/logs/system.log) for more details
   about the error.

   NodeJS Logs option in the Edge UI — Overview Tab of API Proxy

   

Resolution

1. Once you’ve identified the cause of the error, fix the issue in your backend server.
2. If it’s a Node.js backend server:
   1. Check if the error is thrown from your custom code and fix the issue, if possible.
   2. If the error is not thrown from your custom code or if you need assistance, contact
      Apigee Support.

If you need further assistance in troubleshooting 500 Internal Server Error or you suspect
that it’s an issue within Edge, contact Apigee
Support.

Determining the source of the problem

Use one of the following procedures to determine if the 500 Internal Server Error was thrown
during the execution of a policy within the API proxy or by the backend server.

Using Trace in UI

Note: The steps in this section can be performed by both Public and
Private Cloud users.

1. If the issue is still active, enable the trace in UI for the affected API.
2. Once you have captured the trace, select the API request that shows the response code as
   500.
3. Navigate through all the phases of the failing API request and check which phase returns
   the 500 Internal Server Error:
   1. If the error is thrown during the execution of a policy, then proceed to Execution Error in an Edge Policy.
   2. If the backend server has responded with 500 Internal Server, then proceed to Error in the Backend Server.

Using API Monitoring

Note: The steps in this section can be performed by Public Cloud users only.

API Monitoring enables you to isolate problem areas quickly to diagnose error, performance, and latency issues and their source,
such as developer apps, API proxies, backend targets, or the API platform.

Step through a sample scenario that demonstrates how to troubleshoot 5xx issues with your APIs using API Monitoring.
For example, you may want to set up an alert to be notified when the number of 500 status codes or steps.servicecallout.ExecutionFailed faults exceeds a particular threshold.

Using NGINX Access
Logs

Note: The steps in this section are for Edge Private Cloud users
only.

You can also refer to NGINX Access logs to determine whether the 500 status code was thrown
during the execution of a policy within the API proxy or by the backend server. This is
particularly useful if the issue has occurred in the past or if the issue is intermittent and you
are unable to capture the trace in UI. Use the following steps to determine this information from
NGINX access logs:

1. Check the NGINX access logs (/opt/apigee/var/log/edge-router/nginx/ <org>~
<env>.<port#>_access_log ).
2. Search if there are any 500 Errors for the specific API proxy at the specific
   duration.
3. If there are any 500 Errors, then check if the error is a policy or a target server error,
   as shown below:

   Sample Entry showing a Policy Error

   

   Sample Entry showing a Target Server Error

   

4. Once you’ve identified whether it is a policy or target server error:
   1. Proceed to Execution Error in an Edge Policy if
      it is a policy error.
   2. Proceed to Error in Backend Server if it is a target
      server error.