Response error parse error

I am managing an app built by third parts in python.

I have this url dispatcher

urls += [(r'/path/objectAlpha/(.*)', objectAlphaHandler)]  # this was made by third parts, it is expected to work

and this class

class objectAlphaHandler(BaseHandler):

    def __init__(self, application, request, **kwargs):
        super(objectAlphaHandler, self).__init__(application, request, **kwargs)  # do the init of the basehandler

    @gen.coroutine
    def post(self, action=''):
        response = {}
        
            ...     
            
            response = yield self.my_method(json_data)
        
            ... 

        self.write(json.dumps(response))

    def my_method(self, json_data)
        ...

I want to check that the app correctly receives the request and returns some response.

So I try to access that url with Postman

request type:

POST

URL:

http://<machine_ip>:<machine_port>/path/objectAlpha/

I get this error from Postman response box

Parse Error: The server returned a malformed response

and when I click on «view in console» I see

POST http://machine_ip>:<machine_port>/path/objectAlpha/
Error: Parse Error: Expected HTTP/
Request Headers
Content-Type: application/json
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: d644d7dd-699b-4d77-b32f-46a575ae31fc
Host: xx.xxx.x.xx:22
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Request Body

What does Error: Parse Error: Expected HTTP/ mean?

I checked my app logs but it seems it is not handling any request, even if Postman indicates that the server is returning a (malformed) response.

I also tryed to chenge the target url to:

https...

but it returns

 Error: write EPROTO 28427890592840:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:../../third_party/boringssl/src/ssl/tls_record.cc:242:

which I found out it indicates I should stick with HTTP

Then I tried also:

http://<machine_ip>/path/objectAlpha/

and

<machine_ip>/path/objectAlpha/

which generically return:

Error: connect ECONNREFUSED <machine_ip>:80

I also tryed to substitute line

urls += [(r'/path/objectAlpha/(.*)', objectAlphaHandler)]  

with

urls += [(r'/path/objectAlpha/', objectAlphaHandler)]  

and

urls += [(r'/path/objectAlpha', objectAlphaHandler)]  

but none of these worked.

What is wrong? How can I fix it?

UPDATE

Apparently, according to this thread on Postman Github, the problem happens only on Postman Desktop and not on Postman on browser.

So I tryed to send the request form Postman on my browser but I get

 Cloud Agent Error: Can not send requests to reserved address. Make sure address is publicly accessible or select a different agent.

because, according to this other thread,

Postman Website cannot send a request to your computer’s localhost. It first needs to connect to your PC with the Postman desktop client

and even if I follow the indications in that answer

Run it [ndr. Postman desktop], then go to the Postman workspace in your browser -> send the request and it will work.

I still get the same

Error: Parse Error: Expected HTTP/

on both Postman Desktop and Postman on browser.

UPDATE

Going on debugging, I tryed to cast a curl on that URL from my terminal:

myuser@mymachine-VirtualBox:~$ curl --verbose "http://<target_machine_ip>:<target_machine_port>/path/objectAlpha"

and I got:

*   Trying <target_machine_ip>:<target_machine_port>...
* Connected to <target_machine_ip> (<target_machine_ip>) port <target_machine_port> (#0)
> GET /orkpos5/receipt HTTP/1.1
> Host: <target_machine_ip>:<target_machine_port>    
> User-Agent: curl/7.74.0    
> Accept: */*    
>     
* Received HTTP/0.9 when not allowed  
    
* Closing connection 0    
curl: (1) Received HTTP/0.9 when not allowed

Usually, when using Retrofit 2, we have two callback listeners: onResponse and onFailure If onResponse is called, it doesn’t always mean that we get the success condition. Usually a response is considered success if the status scode is 2xx and Retrofit has already provides isSuccessful() method. The problem is how to parse the response body which most likely has different format. In this tutorial, I’m going to show you how to parse custom JSON response body in Retrofit 2.

Preparation

First, we create RetrofitClientInstance class for creating new instance of Retrofit client and the MyService interface which defines the endpoint we’re going to call.

RetrofitClientInstance.java

  public class RetrofitClientInstance {
      private static final String BASE_URL = "http://159.89.185.115:3500";

      public static Retrofit getRetrofitInstance() {
            return new retrofit2.Retrofit.Builder()
                .baseUrl(BASE_URL)
                .addConverterFactory(GsonConverterFactory.create())
                .build();
      }
  }

MyService.java

  public interface MyServie {
      @GET("/api/items")
      Call<List> getItems();
  }

Let’s say we have an endpoint /api/items which in normal case, it returns the list of Item objects.

Item.java

  public class Item {
      private String id;
      private String name;

      public Item(String id, String name) {
          this.id = id;
          this.name = name;
      }

     /* Getter and Setter here */
  }

Below is the standard way to call the endpoint.

Example.java

  MyService myService = RetrofitClientInstance
          .getRetrofitInstance()
          .create(MyService.class);
  Call<List<Item>> call = retailService.getItems();

  call.enqueue(new Callback<List<Item>>() {

      @Override
      public void onResponse(final Call<List<Item>> call, final Response<List<Item>> response) {

          List<Item> items = response.body();
          Toast.makeText(getContext(), "Success").show();
      }

      @Override
      public void onFailure(final Call call, final Throwable t) {
          Toast.makeText(getContext(), "Failed").show();
      }
  });

Parsing Error Response

The problem is how to parse the response if it’s not success. The idea is very simple. We need to create custom util for parsing the error. For example, we know that the server will return a JSON object with the following structure when an error occurs.

  {
    "success": false,
    "errors": [
      "Error message 1",
      "Error message 2"
    ]
  }

We need to create a util for parsing the error. It returns an Object containing parsed error body. First, we define the class which represents the parsed error.

APIError.java

  public class APIError {
      private boolean success;
      private ArrayList messages;

      public static class Builder {
          public Builder() {}

          public Builder success(final boolean success) {
              this.success = success;
              return this;
          }

          public Builder messages(final ArrayList messages) {
              this.messages = messages;
              return this;
          }

          public Builder defaultError() {
              this.messages.add("Something error");
              return this;
          }

          public APIError build() { return new APIError(this); }
      }

      private APIError(final Builder builder) {
          success = builder.successs;
          messages = builder.messages;
      }
  }

And here’s the util for parsing the response body. If you have different response body format, you can adjust it to suit your case.

ErrorUtils.java

  public class ErrorUtils {
      public static APIError parseError(final Response<?> response) {
          JSONObject bodyObj = null;
          boolean success;
          ArrayList messages = new ArrayList<>();

          try {
              String errorBody = response.errorBody().string();

              if (errorBody != null) {
                  bodyObj = new JSONObject(errorBody);

                  success = bodyObj.getBoolean("success");
                  JSONArray errors = bodyObj.getJSONArray("errors");

                  for (int i = 0; i < errors.length(); i++) {
                      messages.add(errors.get(i));
                  }
              } else {
                  success = false;
                  messages.add("Unable to parse error");
              }
          } catch (Exception e) {
              e.printStackTrace();

              success = false;
              messages.add("Unable to parse error");
          }

          return new APIError.Builder()
                  .success(false)
                  .messages(messages)
                  .build();
      }
  }

Finally, change the onResponse and onFailure methods. If response.isSuccessful() is false, we use the error parser. In addition, if the code go through onFailure which most likely we’re even unable to get the error response body, just return the default error.

Example.java

  MyService myService = RetrofitClientInstance
          .getRetrofitInstance()
          .create(MyService.class);
  Call<List<Item>> call = retailService.getItems();

  call.enqueue(new Callback<List<Item>>() {
      @Override
      public void onResponse(final Call<List<Item>> call, final Response<List<Item>> response) {
          if (response.isSuccessful()) {
              List<Item> items = response.body();
              Toast.makeText(getContext(), "Success").show();
          } else {
              apiError = ErrorUtils.parseError(response);
              Toast.makeText(getContext(), R.string.cashier_create_failed,
                      Toast.LENGTH_LONG).show();
          }
      }

      @Override
      public void onFailure(final Call<Cashier> call, final Throwable t) {
          apiError = new APIError.Builder().defaultError().build();
          Toast.makeText(getContext(), "failed").show();
      }
  });

That’s how to parse error body in Retrofit 2. If you also need to define custom GSON converter factory, read this tutorial.

Two weeks ago, you’ve seen how to log requests and responses for debugging purposes. Requests might not finish successfully and you have to take care of failure situations. Most of the time, you need to manually apply the correct action like showing an error message as user feedback. If you get more than just the response status code, you can use the additional data to set the user in the right context and provide more information about the current error situation. That’s what this post is about: how to apply simple error handling using Retrofit 2.

Retrofit Series Overview

Error Handling Preparations

Even though you want your app to always work like expected and there shouldn’t be any issues while executing requests. However, you’re not in control of when servers will fail or users will put wrong data which results in errors returned from the requested API. In those cases, you want to provide as much feedback to the user required to set him/her into the right context so that he/she understands what the issue is.

Before diving into the actual request which results in an error, we’re going to prepare classes to parse the response body which contains more information.

Error Object

At first, we create the error object representing the response you’re receiving from your requested API. Let’s assume your API sends a JSON error body like this:

{
    statusCode: 409,
    message: "Email address already registered"
}

If we would just show the user a generic error message like There went something wrong, he/she would immediately be upset about this stupid app which isn’t able to show what went wrong.

To avoid these bad user experiences, we’re mapping the response body to a Java object, represented by the following class.

public class APIError {

    private int statusCode;
    private String message;

    public APIError() {
    }

    public int status() {
        return statusCode;
    }

    public String message() {
        return message;
    }
}

We don’t actually need the status code inside the response body, it’s just for illustration purposes and this way you don’t need to extra fetch it from the response.

Simple Error Handler

We’ll make use of the following class only having one static method which returns an APIError object. The parseError method expects the response as parameter. Further, you need to make your Retrofit instance available to apply the appropriate response converter for the received JSON error response.

public class ErrorUtils {

    public static APIError parseError(Response<?> response) {
        Converter<ResponseBody, APIError> converter = 
                ServiceGenerator.retrofit()
                        .responseBodyConverter(APIError.class, new Annotation[0]);

        APIError error;

        try {
            error = converter.convert(response.errorBody());
        } catch (IOException e) {
            return new APIError();
        }

        return error;
    }
}

We’re exposing our Retrofit instance from ServiceGenerator via static method (if you’re not familiar with the ServiceGenerator, please read the introductory post of this series). Please bear with us that we’re using a kind of hacky style by exposing the Retrofit object via static method. The thing that is required to parse the JSON error is the response converter. And the response converter is available via our Retrofit object.

At first, we’re getting the error converter from the ServiceGenerator.retrofit() instance by additionally passing our APIError class as the parameter to the responseBodyConverter method. The responseConverter method will return the appropriate converter to parse the response body type. In our case, we’re expecting a JSON converter, because we’ve received JSON data.

Further, we call converter.convert to parse the received response body data into an APIError object. Afterwards, we’ll return the created object.

Error Handler in Action

Retrofit 2 has a different concept of handling «successful» requests than Retrofit 1. In Retrofit 2, all requests that can be executed (sent to the API) and for which you’re receiving a response are seen as «successful». That means, for these requests the onResponse callback is fired and you need to manually check whether the request is actual successful (status 200-299) or erroneous (status 400-599).

If the request finished successfully, we can use the response object and do whatever we wanted. In case the error actually failed (remember, status 400-599), we want to show the user appropriate information about the issue.

Call<User> call = service.me();  
call.enqueue(new Callback<User>() {  
    @Override
    public void onResponse(Call<User> call, Response<User> response) {
        if (response.isSuccessful()) {
            // use response data and do some fancy stuff :)
        } else {
            // parse the response body …
            APIError error = ErrorUtils.parseError(response);
            // … and use it to show error information

            // … or just log the issue like we’re doing :)
            Log.d("error message", error.message());
        }
    }

    @Override
    public void onFailure(Call<User> call, Throwable t) {
        // there is more than just a failing request (like: no internet connection)
    }
});

As you can see, we use the ErrorUtils class to parse the error body and get an APIError object. Use this object and the contained information to show a meaningful message instead of a generic error message.

Outlook

This article shows you a simple way to manage errors and extract information from the response body. Most APIs will send you specific information on what went wrong and you should make use of it.

This is just the tip of the iceberg when it comes to error handling. Within Retrofit 1, you had the opportunity to add a custom error handler. This option was removed from Retrofit 2 and we think it’s good the way it is. We’ll tell you about more advanced techniques on error handling with Retrofit 2 within a future blog post.

If you run into any issue or have a question, please let us know in the comments below or tweet us @futurestud_io.

Handling errors is really important in Go. Errors are first class citizens and there are many different approaches for handling them. Initially I started off basing my error handling almost entirely on a blog post from Rob Pike and created a carve-out from his code to meet my needs. It served me well for a long time, but found over time I wanted a way to easily get a stacktrace of the error, which led me to Dave Cheney’s https://github.com/pkg/errors package. I now use a combination of the two. The implementation below is sourced from my go-api-basic repo, indeed, this post will be folded into its README as well.

Requirements

My requirements for REST API error handling are the following:

• Requests for users who are not properly authenticated should return a 401 Unauthorized error with a WWW-Authenticate response header and an empty response body.
• Requests for users who are authenticated, but do not have permission to access the resource, should return a 403 Forbidden error with an empty response body.
• All requests which are due to a client error (invalid data, malformed JSON, etc.) should return a 400 Bad Request and a response body which looks similar to the following:

{
    "error": {
        "kind": "input_validation_error",
        "param": "director",
        "message": "director is required"
    }
}

• All requests which incur errors as a result of an internal server or database error should return a 500 Internal Server Error and not leak any information about the database or internal systems to the client. These errors should return a response body which looks like the following:

{
    "error": {
        "kind": "internal_error",
        "message": "internal server error - please contact support"
    }
}

All errors should return a Request-Id response header with a unique request id that can be used for debugging to find the corresponding error in logs.

Implementation

All errors should be raised using custom errors from the domain/errs package. The three custom errors correspond directly to the requirements above.

Typical Errors

Typically, errors raised throughout go-api-basic are the custom errs.Error, which looks like:

type Error struct {
   // User is the username of the user attempting the operation.
   User UserName
   // Kind is the class of error, such as permission failure,
   // or "Other" if its class is unknown or irrelevant.
   Kind Kind
   // Param represents the parameter related to the error.
   Param Parameter
   // Code is a human-readable, short representation of the error
   Code Code
   // The underlying error that triggered this one, if any.
   Err error
}

These errors are raised using the E function from the domain/errs package. errs.E is taken from Rob Pike’s upspin errors package (but has been changed based on my requirements). The errs.E function call is variadic and can take several different types to form the custom errs.Error struct.

Here is a simple example of creating an error using errs.E:

err := errs.E("seems we have an error here")

When a string is sent, an error will be created using the errors.New function from github.com/pkg/errors and added to the Err element of the struct, which allows retrieval of the error stacktrace later on. In the above example, User, Kind, Param and Code would all remain unset.

You can set any of these custom errs.Error fields that you like, for example:

func (m *Movie) SetReleased(r string) (*Movie, error) {
    t, err := time.Parse(time.RFC3339, r)
    if err != nil {
        return nil, errs.E(errs.Validation,
            errs.Code("invalid_date_format"),
            errs.Parameter("release_date"),
            err)
    }
    m.Released = t
    return m, nil
}

Above, we used errs.Validation to set the errs.Kind as Validation. Valid error Kind are:

const (
    Other           Kind = iota // Unclassified error. This value is not printed in the error message.
    Invalid                     // Invalid operation for this type of item.
    IO                          // External I/O error such as network failure.
    Exist                       // Item already exists.
    NotExist                    // Item does not exist.
    Private                     // Information withheld.
    Internal                    // Internal error or inconsistency.
    BrokenLink                  // Link target does not exist.
    Database                    // Error from database.
    Validation                  // Input validation error.
    Unanticipated               // Unanticipated error.
    InvalidRequest              // Invalid Request
)

errs.Code represents a short code to respond to the client with for error handling based on codes (if you choose to do this) and is any string you want to pass.

errs.Parameter represents the parameter that is being validated or has problems, etc.

Note in the above example, instead of passing a string and creating a new error inside the errs.E function, I am directly passing the error returned by the time.Parse function to errs.E. The error is then added to the Err field using errors.WithStack from the github.com/pkg/errors package. This will enable stacktrace retrieval later as well.

There are a few helpers in the errs package as well, namely the errs.MissingField function which can be used when validating missing input on a field. This idea comes from this Mat Ryer post and is pretty handy.

Here is an example in practice:

// IsValid performs validation of the struct
func (m *Movie) IsValid() error {
    switch {
    case m.Title == "":
        return errs.E(errs.Validation, errs.Parameter("title"), errs.MissingField("title"))

The error message for the above would read title is required

There is also errs.InputUnwanted which is meant to be used when a field is populated with a value when it is not supposed to be.

Typical Error Flow

As errors created with errs.E move up the call stack, they can just be returned, like the following:

func inner() error {
    return errs.E("seems we have an error here")
}

func middle() error {
    err := inner()
    if err != nil {
        return err
    }
    return nil
}

func outer() error {
    err := middle()
    if err != nil {
        return err
    }
    return nil
}

In the above example, the error is created in the inner function — middle and outer return the error as is typical in Go.

You can add additional context fields (errs.Code, errs.Parameter, errs.Kind) as the error moves up the stack, however, I try to add as much context as possible at the point of error origin and only do this in rare cases.

Handler Flow

At the top of the program flow for each service is the app service handler (for example, Server.handleMovieCreate). In this handler, any error returned from any function or method is sent through the errs.HTTPErrorResponse function along with the http.ResponseWriter and a zerolog.Logger.

For example:

response, err := s.CreateMovieService.Create(r.Context(), rb, u)
if err != nil {
    errs.HTTPErrorResponse(w, logger, err)
    return
}

errs.HTTPErrorResponse takes the custom error (errs.Error, errs.Unauthenticated or errs.UnauthorizedError), writes the response to the given http.ResponseWriter and logs the error using the given zerolog.Logger.

return must be called immediately after errs.HTTPErrorResponse to return the error to the client.

Typical Error Response

For the errs.Error type, errs.HTTPErrorResponse writes the HTTP response body as JSON using the errs.ErrResponse struct.

// ErrResponse is used as the Response Body
type ErrResponse struct {
    Error ServiceError `json:"error"`
}

// ServiceError has fields for Service errors. All fields with no data will
// be omitted
type ServiceError struct {
    Kind    string `json:"kind,omitempty"`
    Code    string `json:"code,omitempty"`
    Param   string `json:"param,omitempty"`
    Message string `json:"message,omitempty"`
}

When the error is returned to the client, the response body JSON looks like the following:

{
    "error": {
        "kind": "input_validation_error",
        "code": "invalid_date_format",
        "param": "release_date",
        "message": "parsing time "1984a-03-02T00:00:00Z" as "2006-01-02T15:04:05Z07:00": cannot parse "a-03-02T00:00:00Z" as "-""
    }
}

In addition, the error is logged. If zerolog.ErrorStackMarshaler is set to log error stacks (more about this in a later post), the logger will log the full error stack, which can be super helpful when trying to identify issues.

The error log will look like the following (I cut off parts of the stack for brevity):

{
    "level": "error",
    "ip": "127.0.0.1",
    "user_agent": "PostmanRuntime/7.26.8",
    "request_id": "bvol0mtnf4q269hl3ra0",
    "stack": [{
        "func": "E",
        "line": "172",
        "source": "errs.go"
    }, {
        "func": "(*Movie).SetReleased",
        "line": "76",
        "source": "movie.go"
    }, {
        "func": "(*MovieController).CreateMovie",
        "line": "139",
        "source": "create.go"
    }, {
    ...
    }],
    "error": "parsing time "1984a-03-02T00:00:00Z" as "2006-01-02T15:04:05Z07:00": cannot parse "a-03-02T00:00:00Z" as "-"",
    "HTTPStatusCode": 400,
    "Kind": "input_validation_error",
    "Parameter": "release_date",
    "Code": "invalid_date_format",
    "time": 1609650267,
    "severity": "ERROR",
    "message": "Response Error Sent"
}

Note: E will usually be at the top of the stack as it is where the errors.New or errors.WithStack functions are being called.

Internal or Database Error Response

There is logic within errs.HTTPErrorResponse to return a different response body if the errs.Kind is Internal or Database. As per the requirements, we should not leak the error message or any internal stack, etc. when an internal or database error occurs. If an error comes through and is an errs.Error with either of these error Kind or is unknown error type in any way, the response will look like the following:

{
    "error": {
        "kind": "internal_error",
        "message": "internal server error - please contact support"
    }
}

Unauthenticated Errors

type UnauthenticatedError struct {
    // WWWAuthenticateRealm is a description of the protected area.
    // If no realm is specified, "DefaultRealm" will be used as realm
    WWWAuthenticateRealm string

    // The underlying error that triggered this one, if any.
    Err error
}

The spec for 401 Unauthorized calls for a WWW-Authenticate response header along with a realm. The realm should be set when creating an Unauthenticated error. The errs.NewUnauthenticatedError function initializes an UnauthenticatedError.

I generally like to follow the Go idiom for brevity in all things as much as possible, but for Unauthenticated vs. Unauthorized errors, it’s confusing enough as it is already, I don’t take any shortcuts.

func NewUnauthenticatedError(realm string, err error) *UnauthenticatedError {
    return &UnauthenticatedError{WWWAuthenticateRealm: realm, Err: err}
}

Unauthenticated Error Flow

The errs.Unauthenticated error should only be raised at points of authentication as part of a middleware handler. I will get into application flow in detail later, but authentication for go-api-basic happens in middleware handlers prior to calling the app handler for the given route.

• The WWW-Authenticate realm is set to the request context using the defaultRealmHandler middleware in the app package prior to attempting authentication.
• Next, the Oauth2 access token is retrieved from the Authorization http header using the accessTokenHandler middleware. There are several access token validations in this middleware, if any are not successful, the errs.Unauthenticated error is returned using the realm set to the request context.
• Finally, if the access token is successfully retrieved, it is then converted to a User via the GoogleAccessTokenConverter.Convert method in the gateway/authgateway package. This method sends an outbound request to Google using their API; if any errors are returned, an errs.Unauthenticated error is returned.

In general, I do not like to use context.Context, however, it is used in go-api-basic to pass values between middlewares. The WWW-Authenticate realm, the Oauth2 access token and the calling user after authentication, all of which are request-scoped values, are all set to the request context.Context.

Unauthenticated Error Response

Per requirements, go-api-basic does not return a response body when returning an Unauthenticated error. The error response from cURL looks like the following:

HTTP/1.1 401 Unauthorized
Request-Id: c30hkvua0brkj8qhk3e0
Www-Authenticate: Bearer realm="go-api-basic"
Date: Wed, 09 Jun 2021 19:46:07 GMT
Content-Length: 0

Unauthorized Errors

type UnauthorizedError struct {
    // The underlying error that triggered this one, if any.
    Err error
}

The errs.NewUnauthorizedError function initializes an UnauthorizedError.

Unauthorized Error Flow

The errs.Unauthorized error is raised when there is a permission issue for a user when attempting to access a resource. Currently, go-api-basic’s placeholder authorization implementation Authorizer.Authorize in the domain/auth package performs rudimentary checks that a user has access to a resource. If the user does not have access, the errs.Unauthorized error is returned.

Per requirements, go-api-basic does not return a response body when returning an Unauthorized error. The error response from cURL looks like the following:

HTTP/1.1 403 Forbidden
Request-Id: c30hp2ma0brkj8qhk3f0
Date: Wed, 09 Jun 2021 19:54:50 GMT
Content-Length: 0

404 Error Illustration by Pixeltrue from Ouch!