Grpc 701 error

Error handling

How gRPC deals with errors, and gRPC error codes.

Standard error model

As you’ll have seen in our concepts document and examples, when a gRPC call
completes successfully the server returns an OK status to the client
(depending on the language the OK status may or may not be directly used in
your code). But what happens if the call isn’t successful?

If an error occurs, gRPC returns one of its error status codes instead, with an
optional string error message that provides further details about what happened.
Error information is available to gRPC clients in all supported languages.

Richer error model

The error model described above is the official gRPC error model,
is supported by all gRPC client/server libraries, and is independent of
the gRPC data format (whether protocol buffers or something else). You
may have noticed that it’s quite limited and doesn’t include the
ability to communicate error details.

If you’re using protocol buffers as your data format, however, you may
wish to consider using the richer error model developed and used
by Google as described
here. This
model enables servers to return and clients to consume additional
error details expressed as one or more protobuf messages. It further
specifies a standard set of error message
types
to cover the most common needs (such as invalid parameters, quota
violations, and stack traces). The protobuf binary encoding of this
extra error information is provided as trailing metadata in the
response.

This richer error model is already supported in the C++, Go, Java,
Python, and Ruby libraries, and at least the grpc-web and Node.js
libraries have open issues requesting it. Other language libraries may
add support in the future if there’s demand, so check their github
repos if interested. Note however that the grpc-core library written
in C will not likely ever support it since it is purposely data format
agnostic.

You could use a similar approach (put error details in trailing
response metadata) if you’re not using protocol buffers, but you’d
likely need to find or develop library support for accessing this data
in order to make practical use of it in your APIs.

There are important considerations to be aware of when deciding whether to
use such an extended error model, however, including:

• Library implementations of the extended error model may not be consistent
  across languages in terms of requirements for and expectations of the error
  details payload
• Existing proxies, loggers, and other standard HTTP request
  processors don’t have visibility into the error details and thus
  wouldn’t be able to leverage them for monitoring or other purposes
• Additional error detail in the trailers interferes with head-of-line
  blocking, and will decrease HTTP/2 header compression efficiency due to
  more frequent cache misses
• Larger error detail payloads may run into protocol limits (like
  max headers size), effectively losing the original error

Error status codes

Errors are raised by gRPC under various circumstances, from network failures to
unauthenticated connections, each of which is associated with a particular
status code. The following error status codes are supported in all gRPC
languages.

General errors

Network failures

Protocol errors

Sample code

For sample code illustrating how to handle various gRPC errors, see the
grpc-errors repo.

gRPC uses a set of well defined status codes as part of the RPC API. These statuses are defined as such:

All RPCs started at a client return a status object composed of an integer code and a string message. The server-side can choose the status it returns for a given RPC.

The gRPC client and server-side implementations may also generate and return status on their own when errors happen. Only a subset of the pre-defined status codes are generated by the gRPC libraries. This allows applications to be sure that any other code it sees was actually returned by the application (although it is also possible for the server-side to return one of the codes generated by the gRPC libraries).

The following table lists the codes that may be returned by the gRPC libraries (on either the client-side or server-side) and summarizes the situations in which they are generated.

The following status codes are never generated by the library:

• INVALID_ARGUMENT
• NOT_FOUND
• ALREADY_EXISTS
• FAILED_PRECONDITION
• ABORTED
• OUT_OF_RANGE
• DATA_LOSS

Applications that may wish to retry failed RPCs must decide which status codes on which to retry. As shown in the table above, the gRPC library can generate the same status code for different cases. Server applications can also return those same status codes. Therefore, there is no fixed list of status codes on which it is appropriate to retry in all applications. As a result, individual applications must make their own determination as to which status codes should cause an RPC to be retried.

This chapter provides an overview of the error model for Google APIs. It also
provides general guidance to developers on how to properly generate and
handle errors.

Google APIs use a simple protocol-agnostic error model, which allows us to
offer a consistent experience across different APIs, different API protocols
(such as gRPC or HTTP), and different error contexts (such as asynchronous,
batch, or workflow errors).

Error Model

The error model for Google APIs is logically defined by
google.rpc.Status,
an instance of which is returned to the client when an API error occurs. The
following code snippet shows the overall design of the error model:

package google.rpc;

// The `Status` type defines a logical error model that is suitable for
// different programming environments, including REST APIs and RPC APIs.
message Status {
  // A simple error code that can be easily handled by the client. The
  // actual error code is defined by `google.rpc.Code`.
  int32 code = 1;

  // A developer-facing human-readable error message in English. It should
  // both explain the error and offer an actionable resolution to it.
  string message = 2;

  // Additional error information that the client code can use to handle
  // the error, such as retry info or a help link.
  repeated google.protobuf.Any details = 3;
}

Because most Google APIs use resource-oriented API design, the error handling
follows the same design principle by using a small set of standard errors with a
large number of resources. For example, instead of defining different kinds of
«not found» errors, the server uses one standard google.rpc.Code.NOT_FOUND
error code and tells the client which specific resource was not found. The
smaller error space reduces the complexity of documentation, affords better
idiomatic mappings in client libraries, and reduces client logic complexity
while not restricting the inclusion of actionable information.

Error Codes

Google APIs must use the canonical error codes defined by
google.rpc.Code.
Individual APIs must avoid defining additional error codes, since
developers are very unlikely to write logic to handle a large number of error
codes. For reference, handling an average of three error codes per API call
would mean most application logic would just be for error handling, which would
not be a good developer experience.

Error Messages

The error message should help users understand and resolve the API error
easily and quickly. In general, consider the following guidelines when writing
error messages:

• Do not assume the user is an expert user of your API. Users could be client
  developers, operations people, IT staff, or end-users of apps.
• Do not assume the user knows anything about your service implementation or
  is familiar with the context of the errors (such as log analysis).
• When possible, error messages should be constructed such that a technical
  user (but not necessarily a developer of your API) can respond to the error
  and correct it.
• Keep the error message brief. If needed, provide a link where a confused
  reader can ask questions, give feedback, or get more information that
  doesn’t cleanly fit in an error message. Otherwise, use the details field to
  expand.

Error Details

Google APIs define a set of standard error payloads for error details, which you
can find in
google/rpc/error_details.proto.
These cover the most common needs for API errors, such as quota failure and
invalid parameters. Like error codes, developers should use these standard
payloads whenever possible.

Additional error detail types should only be introduced if they can assist
application code to handle the errors. If the error information can only be
handled by humans, rely on the error message content and let developers handle
it manually rather than introducing additional error detail types.

Here are some example error_details payloads:

• ErrorInfo: Provides structured error information that is both stable
  and extensible.
• RetryInfo: Describes when clients can retry a failed request, may be
  returned on Code.UNAVAILABLE or Code.ABORTED
• QuotaFailure: Describes how a quota check failed, may be returned on
  Code.RESOURCE_EXHAUSTED
• BadRequest: Describes violations in a client request, may be returned on
  Code.INVALID_ARGUMENT

Error Info

ErrorInfo is a special kind of error payload. It provides stable and
extensible error information that both humans and applications can depend on.
Each ErrorInfo has three pieces of information: an error domain, an error
reason, and a set of error metadata, such as this
example.
For more information, see the
ErrorInfo
definition.

For Google APIs, the primary error domain is googleapis.com, and the
corresponding error reasons are defined by google.api.ErrorReason enum.
For more information, see the
google.api.ErrorReason
definition.

Error Localization

The message field in
google.rpc.Status
is developer-facing and must be in English.

If a user-facing error message is needed, use
google.rpc.LocalizedMessage
as your details field. While the message field in
google.rpc.LocalizedMessage
can be localized, ensure that the message field in
google.rpc.Status
is in English.

By default, the API service should use the authenticated user’s locale or HTTP
Accept-Language header or the language_code parameter in the request to
determine the language for the localization.

Error Mapping

Google APIs are accessible in different programming environments. Each
environment typically has its own way of error handling. The following
sections explain how the error model is mapped in commonly used environments.

HTTP Mapping

While proto3 messages have native JSON encoding, Google’s API Platform uses a
different error schema for Google’s JSON HTTP APIs for backward compatibility
reasons.

Schema:

// This message defines the error schema for Google's JSON HTTP APIs.
message Error {
  // Deprecated. This message is only used by error format v1.
  message ErrorProto {}
  // This message has the same semantics as `google.rpc.Status`. It uses HTTP
  // status code instead of gRPC status code. It has extra fields `status` and
  // `errors` for backward compatibility with [Google API Client
  // Libraries](https://developers.google.com/api-client-library).
  message Status {
    // The HTTP status code that corresponds to `google.rpc.Status.code`.
    int32 code = 1;
    // This corresponds to `google.rpc.Status.message`.
    string message = 2;
    // Deprecated. This field is only used by error format v1.
    repeated ErrorProto errors = 3;
    // This is the enum version for `google.rpc.Status.code`.
    google.rpc.Code status = 4;
    // This corresponds to `google.rpc.Status.details`.
    repeated google.protobuf.Any details = 5;
  }
  // The actual error payload. The nested message structure is for backward
  // compatibility with [Google API Client
  // Libraries](https://developers.google.com/api-client-library). It also
  // makes the error more readable to developers.
  Status error = 1;
}

Example (link):

{
  "error": {
    "code": 400,
    "message": "API key not valid. Please pass a valid API key.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "API_KEY_INVALID",
        "domain": "googleapis.com",
        "metadata": {
          "service": "translate.googleapis.com"
        }
      }
    ]
  }
}

gRPC Mapping

Different RPC protocols map the error model differently. For
gRPC, the error model is natively supported by the generated
code and the runtime library in each supported language. You can find out more
in gRPC’s API documentation. For example, see gRPC Java’s
io.grpc.Status.

Client Library Mapping

Google client libraries may choose to surface errors differently per language to
be consistent with established idioms. For example, the
google-cloud-go
library will return an error that implements the same interface as
google.rpc.Status,
while
google-cloud-java
will raise an Exception.

Handling Errors

Below is a table containing all of the gRPC error codes defined in
google.rpc.Code
and a short description of their cause. To handle an error, you can check the
description for the returned status code and modify your call accordingly.

Retrying Errors

Clients may retry on 503 UNAVAILABLE errors with exponential backoff.
The minimum delay should be 1s unless it is documented otherwise. The default
retry repetition should be once unless it is documented otherwise.

For 429 RESOURCE_EXHAUSTED errors, the client may retry at the higher level
with minimum 30s delay. Such retries are only useful for long running
background jobs.

For all other errors, retry may not be applicable. First ensure your request
is idempotent, and see
google.rpc.RetryInfo
for guidance.

Propagating Errors

If your API service depends on other services, you should not blindly propagate
errors from those services to your clients. When translating errors, we suggest
the following:

• Hide implementation details and confidential information.
• Adjust the party responsible for the error. For example, a server that
  receives an INVALID_ARGUMENT error from another service should propagate
  an INTERNAL to its own caller.

Reproducing Errors

If you cannot resolve errors through analysis of logs and monitoring, you should
try to reproduce the errors with a simple and repeatable test. You can use the
test to collect more information for troubleshooting, which you can provide
when contacting technical support.

We recommend you use curl -v and
System Parameters to reproduce errors with
Google APIs. Together they can reproduce almost all Google API requests,
and provide you verbose debug information. For more information, see the
respective documentation pages for the API you are calling.

Generating Errors

If you are a server developer, you should generate errors with enough
information to help client developers understand and resolve the problem. At the
same time, you must be aware of the security and privacy of the user data, and
avoid disclosing sensitive information in the error message and error details,
since errors are often logged and may be accessible by others. For example, an
error message like «Client IP address is not on allowlist 128.0.0.0/8» exposes
information about the server-side policy, which may not be accessible to the
user who has access to the logs.

To generate proper errors, you first need to be familiar with google.rpc.Code to
choose the most suitable error code for each error condition. A server
application may check multiple error conditions in parallel, and return the
first one.

The following table lists each error code and an example of a good error
message.

Error Payloads

The google.rpc package defines a set of standard error payloads, which are
preferred to custom error payloads. The following table lists each error code
and its matching standard error payload, if applicable. We recommend advanced
applications look for these error payloads in google.rpc.Status when they
handle errors.