Обработка ошибок python requests - Исправление ошибок и поиск оптимальных решений проблем

How to handle exceptions with python library requests?
For example how to check is PC connected to internet?

When I try

try:
    requests.get('http://www.google.com')
except ConnectionError:
    # handle the exception

it gives me error name ConnectionError is not defined

Kevin Burke

58.9k73 gold badges184 silver badges298 bronze badges

asked Jan 29, 2012 at 16:46

Assuming you did import requests, you want requests.ConnectionError. ConnectionError is an exception defined by requests. See the API documentation here.

Thus the code should be:

try:
   requests.get('http://www.google.com')
except requests.ConnectionError:
   # handle the exception

^{The original link to the Python v2 API documentation from the original answer no longer works.}

tripleee

170k31 gold badges261 silver badges306 bronze badges

answered Jan 29, 2012 at 16:52

kindallkindall

175k35 gold badges271 silver badges302 bronze badges

As per the documentation, I have added the below points:

In the event of a network problem (refused connection e.g internet issue), Requests will raise a ConnectionError exception.
```
try:
   requests.get('http://www.google.com')
except requests.ConnectionError:
   # handle ConnectionError the exception
```
In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.
Response.raise_for_status() will raise an HTTPError if the HTTP request returned an unsuccessful status code.
```
try:
   r = requests.get('http://www.google.com/nowhere')
   r.raise_for_status()
except requests.exceptions.HTTPError as err:
   #handle the HTTPError request here
```
In the event of times out of request, a Timeout exception is raised.

You can tell Requests to stop waiting for a response after a given number of seconds, with a timeout arg.

    requests.get('https://github.com/', timeout=0.001)
    # timeout is not a time limit on the entire response download; rather, 
    # an exception is raised if the server has not issued a response for
    # timeout seconds

All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException. So a base handler can look like,
```
try:
   r = requests.get(url)
except requests.exceptions.RequestException as e:
   # handle all the errors here
```

^{The original link to the Python v2 documentation no longer works, and now points to the new documentation.}

tripleee

170k31 gold badges261 silver badges306 bronze badges

answered Jul 28, 2019 at 9:47

Actually, there are much more exceptions that requests.get() can generate than just ConnectionError. Here are some I’ve seen in production:

from requests import ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError

try:
    r = requests.get(url, timeout=6.0)
except (ConnectTimeout, HTTPError, ReadTimeout, Timeout, ConnectionError):
    continue

Falko

16.7k13 gold badges55 silver badges103 bronze badges

answered Sep 6, 2017 at 9:57

kravietzkravietz

10.3k2 gold badges33 silver badges27 bronze badges

Include the requests module using import requests .

It is always good to implement exception handling. It does not only help to avoid unexpected exit of script but can also help to log errors and info notification. When using Python requests I prefer to catch exceptions like this:

try:
    res = requests.get(adress,timeout=30)
except requests.ConnectionError as e:
    print("OOPS!! Connection Error. Make sure you are connected to Internet. Technical Details given below.n")
    print(str(e))            
    continue
except requests.Timeout as e:
    print("OOPS!! Timeout Error")
    print(str(e))
    continue
except requests.RequestException as e:
    print("OOPS!! General Error")
    print(str(e))
    continue
except KeyboardInterrupt:
    print("Someone closed the program")

answered May 23, 2018 at 20:19

Tanmoy DattaTanmoy Datta

1,5341 gold badge16 silver badges15 bronze badges

for clarity, that is

except requests.ConnectionError:

NOT

import requests.ConnectionError

You can also catch a general exception (although this isn’t recommended) with

except Exception:

answered Feb 22, 2015 at 10:50

StackGStackG

2,6805 gold badges26 silver badges45 bronze badges

Источник

Python request module is a simple and elegant Python HTTP library. It provides methods for accessing Web resources via HTTP. In the following article, we will use the HTTP GET method in the Request module. This method requests data from the server and the Exception handling comes in handy when the response is not successful. Here, we will go through such situations. We will use Python’s try and except functionality to explore the exceptions that arise from the Requests module.

url: Returns the URL of the response
raise_for_status(): If an error occur, this method returns a HTTPError object
request: Returns the request object that requested this response
status_code: Returns a number that indicates the status (200 is OK, 404 is Not Found)

Successful Connection Request

The first thing to know is that the response code is 200 if the request is successful.

Python3

Output:

Exception Handling for HTTP Errors

Here, we tried the following URL sequence and then passed this variable to the Python requests module using raised_for_status(). If the try part is successful, we will get the response code 200, if the page that we requested doesn’t exist. This is an HTTP error, which was handled by the Request module’s exception HTTPError and you probably got the error 404.

Python3

import requests

try:

r = requests.get(url, timeout=1)

r.raise_for_status()

except requests.exceptions.HTTPError as errh:

print("HTTP Error")

print(errh.args[0])

print(r)

Output:

HTTP Error
404 Client Error: Not Found for url: https://www.amazon.com/nothing_here
<Response [404]>

General Exception Handling

You could also use a general exception from the Request module. That is requests.exceptions.RequestException.

Python3

try:

r = requests.get(url, timeout=1)

r.raise_for_status()

except requests.exceptions.RequestException as errex:

print("Exception request")

Output:

Exception request

Now, you may have noticed that there is an argument ‘timeout’ passed into the Request module. We could prescribe a time limit for the requested connection to respond. If this has not happened, we could catch that using the exception requests.exceptions.ReadTimeout. To demonstrate this let us find a website that responds successfully.

Python3

import requests

try:

r = requests.get(url, timeout=1)

r.raise_for_status()

except requests.exceptions.ReadTimeout as errrt:

print("Time out")

print(r)

Output:

<Response [200]>

If we change timeout = 0.01, the same code would return, because the request could not possibly be that fast.

Time out
<Response [200]>

Exception Handling for Missing Schema

Another common error is that we might not specify HTTPS or HTTP in the URL. For example, We cause use requests.exceptions.MissingSchema to catch this exception.

Python3

url = "www.google.com"

try:

r = requests.get(url, timeout=1)

r.raise_for_status()

except requests.exceptions.MissingSchema as errmiss:

print("Missing schema: include http or https")

except requests.exceptions.ReadTimeout as errrt:

print("Time out")

Output:

Missing scheme: include http or https

Exception Handling for Connection Error

Let us say that there is a site that doesn’t exist. Here, the error will occur even when you can’t make a connection because of the lack of an internet connection

Python3

try:

r = requests.get(url, timeout = 1, verify = True)

r.raise_for_status()

except requests.exceptions.HTTPError as errh:

print("HTTP Error")

print(errh.args[0])

except requests.exceptions.ReadTimeout as errrt:

print("Time out")

except requests.exceptions.ConnectionError as conerr:

print("Connection error")

Output:

Connection error

Putting Everything Together

Here, We put together everything we tried so far the idea is that the exceptions are handled according to the specificity.

For example, url = “https://www.gle.com”, When this code is run for this URL will produce an Exception request. Whereas, In the absence of connection requests.exceptions.ConnectionError will print the Connection Error, and when the connection is not made the general exception is handled by requests.exceptions.RequestException.

Python3

try:

r = requests.get(url, timeout=1, verify=True)

r.raise_for_status()

except requests.exceptions.HTTPError as errh:

print("HTTP Error")

print(errh.args[0])

except requests.exceptions.ReadTimeout as errrt:

print("Time out")

except requests.exceptions.ConnectionError as conerr:

print("Connection error")

except requests.exceptions.RequestException as errex:

print("Exception request")

Output:

Note: The output may change according to requests.

 Time out

Источник

.. module:: requests.models

Eager to get started? This page gives a good introduction in how to get started
with Requests.

First, make sure that:

Requests is :ref:`installed <install>`
Requests is :ref:`up-to-date <updates>`

Let’s get started with some simple examples.

Make a Request

Making a request with Requests is very simple.

Begin by importing the Requests module:

>>> import requests

Now, let’s try to get a webpage. For this example, let’s get GitHub’s public
timeline:

>>> r = requests.get('https://api.github.com/events')

Now, we have a :class:`Response <requests.Response>` object called r. We can
get all the information we need from this object.

Requests’ simple API means that all forms of HTTP request are as obvious. For
example, this is how you make an HTTP POST request:

>>> r = requests.post('https://httpbin.org/post', data={'key': 'value'})

Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD and
OPTIONS? These are all just as simple:

>>> r = requests.put('https://httpbin.org/put', data={'key': 'value'})
>>> r = requests.delete('https://httpbin.org/delete')
>>> r = requests.head('https://httpbin.org/get')
>>> r = requests.options('https://httpbin.org/get')

That’s all well and good, but it’s also only the start of what Requests can
do.

Passing Parameters In URLs

You often want to send some sort of data in the URL’s query string. If
you were constructing the URL by hand, this data would be given as key/value
pairs in the URL after a question mark, e.g. httpbin.org/get?key=val.
Requests allows you to provide these arguments as a dictionary of strings,
using the params keyword argument. As an example, if you wanted to pass
key1=value1 and key2=value2 to httpbin.org/get, you would use the
following code:

>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.get('https://httpbin.org/get', params=payload)

You can see that the URL has been correctly encoded by printing the URL:

>>> print(r.url)
https://httpbin.org/get?key2=value2&key1=value1

Note that any dictionary key whose value is None will not be added to the
URL’s query string.

You can also pass a list of items as a value:

>>> payload = {'key1': 'value1', 'key2': ['value2', 'value3']}

>>> r = requests.get('https://httpbin.org/get', params=payload)
>>> print(r.url)
https://httpbin.org/get?key1=value1&key2=value2&key2=value3

Response Content

We can read the content of the server’s response. Consider the GitHub timeline
again:

>>> import requests

>>> r = requests.get('https://api.github.com/events')
>>> r.text
'[{"repository":{"open_issues":0,"url":"https://github.com/...

Requests will automatically decode content from the server. Most unicode
charsets are seamlessly decoded.

When you make a request, Requests makes educated guesses about the encoding of
the response based on the HTTP headers. The text encoding guessed by Requests
is used when you access r.text. You can find out what encoding Requests is
using, and change it, using the r.encoding property:

>>> r.encoding
'utf-8'
>>> r.encoding = 'ISO-8859-1'

If you change the encoding, Requests will use the new value of r.encoding
whenever you call r.text. You might want to do this in any situation where
you can apply special logic to work out what the encoding of the content will
be. For example, HTML and XML have the ability to specify their encoding in
their body. In situations like this, you should use r.content to find the
encoding, and then set r.encoding. This will let you use r.text with
the correct encoding.

Requests will also use custom encodings in the event that you need them. If
you have created your own encoding and registered it with the codecs
module, you can simply use the codec name as the value of r.encoding and
Requests will handle the decoding for you.

Binary Response Content

You can also access the response body as bytes, for non-text requests:

>>> r.content
b'[{"repository":{"open_issues":0,"url":"https://github.com/...

The gzip and deflate transfer-encodings are automatically decoded for you.

The br transfer-encoding is automatically decoded for you if a Brotli library
like brotli or brotlicffi is installed.

For example, to create an image from binary data returned by a request, you can
use the following code:

>>> from PIL import Image
>>> from io import BytesIO

>>> i = Image.open(BytesIO(r.content))

JSON Response Content

There’s also a builtin JSON decoder, in case you’re dealing with JSON data:

>>> import requests

>>> r = requests.get('https://api.github.com/events')
>>> r.json()
[{'repository': {'open_issues': 0, 'url': 'https://github.com/...

In case the JSON decoding fails, r.json() raises an exception. For example, if
the response gets a 204 (No Content), or if the response contains invalid JSON,
attempting r.json() raises requests.exceptions.JSONDecodeError. This wrapper exception
provides interoperability for multiple exceptions that may be thrown by different
python versions and json serialization libraries.

It should be noted that the success of the call to r.json() does not
indicate the success of the response. Some servers may return a JSON object in a
failed response (e.g. error details with HTTP 500). Such JSON will be decoded
and returned. To check that a request is successful, use
r.raise_for_status() or check r.status_code is what you expect.

Raw Response Content

In the rare case that you’d like to get the raw socket response from the
server, you can access r.raw. If you want to do this, make sure you set
stream=True in your initial request. Once you do, you can do this:

>>> r = requests.get('https://api.github.com/events', stream=True)

>>> r.raw
<urllib3.response.HTTPResponse object at 0x101194810>

>>> r.raw.read(10)
'x1fx8bx08x00x00x00x00x00x00x03'

In general, however, you should use a pattern like this to save what is being
streamed to a file:

with open(filename, 'wb') as fd:
    for chunk in r.iter_content(chunk_size=128):
        fd.write(chunk)

Using Response.iter_content will handle a lot of what you would otherwise
have to handle when using Response.raw directly. When streaming a
download, the above is the preferred and recommended way to retrieve the
content. Note that chunk_size can be freely adjusted to a number that
may better fit your use cases.

Note

An important note about using Response.iter_content versus Response.raw.
Response.iter_content will automatically decode the gzip and deflate
transfer-encodings. Response.raw is a raw stream of bytes — it does not
transform the response content. If you really need access to the bytes as they
were returned, use Response.raw.

Custom Headers

If you’d like to add HTTP headers to a request, simply pass in a dict to the
headers parameter.

For example, we didn’t specify our user-agent in the previous example:

>>> url = 'https://api.github.com/some/endpoint'
>>> headers = {'user-agent': 'my-app/0.0.1'}

>>> r = requests.get(url, headers=headers)

Note: Custom headers are given less precedence than more specific sources of information. For instance:

Authorization headers set with headers= will be overridden if credentials
are specified in .netrc, which in turn will be overridden by the auth=
parameter. Requests will search for the netrc file at ~/.netrc, ~/_netrc,
or at the path specified by the NETRC environment variable.
Authorization headers will be removed if you get redirected off-host.
Proxy-Authorization headers will be overridden by proxy credentials provided in the URL.
Content-Length headers will be overridden when we can determine the length of the content.

Furthermore, Requests does not change its behavior at all based on which custom headers are specified. The headers are simply passed on into the final request.

Note: All header values must be a string, bytestring, or unicode. While permitted, it’s advised to avoid passing unicode header values.

POST a Multipart-Encoded File

Requests makes it simple to upload Multipart-encoded files:

>>> url = 'https://httpbin.org/post'
>>> files = {'file': open('report.xls', 'rb')}

>>> r = requests.post(url, files=files)
>>> r.text
{
  ...
  "files": {
    "file": "<censored...binary...data>"
  },
  ...
}

You can set the filename, content_type and headers explicitly:

>>> url = 'https://httpbin.org/post'
>>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}

>>> r = requests.post(url, files=files)
>>> r.text
{
  ...
  "files": {
    "file": "<censored...binary...data>"
  },
  ...
}

If you want, you can send strings to be received as files:

>>> url = 'https://httpbin.org/post'
>>> files = {'file': ('report.csv', 'some,data,to,sendnanother,row,to,sendn')}

>>> r = requests.post(url, files=files)
>>> r.text
{
  ...
  "files": {
    "file": "some,data,to,send\nanother,row,to,send\n"
  },
  ...
}

In the event you are posting a very large file as a multipart/form-data
request, you may want to stream the request. By default, requests does not
support this, but there is a separate package which does —
requests-toolbelt. You should read the toolbelt’s documentation for more details about how to use it.

For sending multiple files in one request refer to the :ref:`advanced <advanced>`
section.

Warning

It is strongly recommended that you open files in :ref:`binary
mode <tut-files>`. This is because Requests may attempt to provide
the Content-Length header for you, and if it does this value
will be set to the number of bytes in the file. Errors may occur
if you open the file in text mode.

Response Status Codes

We can check the response status code:

>>> r = requests.get('https://httpbin.org/get')
>>> r.status_code
200

Requests also comes with a built-in status code lookup object for easy
reference:

>>> r.status_code == requests.codes.ok
True

If we made a bad request (a 4XX client error or 5XX server error response), we
can raise it with
:meth:`Response.raise_for_status() <requests.Response.raise_for_status>`:

>>> bad_r = requests.get('https://httpbin.org/status/404')
>>> bad_r.status_code
404

>>> bad_r.raise_for_status()
Traceback (most recent call last):
  File "requests/models.py", line 832, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error

But, since our status_code for r was 200, when we call
raise_for_status() we get:

>>> r.raise_for_status()
None

All is well.

Response Headers

We can view the server’s response headers using a Python dictionary:

>>> r.headers
{
    'content-encoding': 'gzip',
    'transfer-encoding': 'chunked',
    'connection': 'close',
    'server': 'nginx/1.0.4',
    'x-runtime': '148ms',
    'etag': '"e1ca502697e5c9317743dc078f67693f"',
    'content-type': 'application/json'
}

The dictionary is special, though: it’s made just for HTTP headers. According to
RFC 7230, HTTP Header names
are case-insensitive.

So, we can access the headers using any capitalization we want:

>>> r.headers['Content-Type']
'application/json'

>>> r.headers.get('content-type')
'application/json'

It is also special in that the server could have sent the same header multiple
times with different values, but requests combines them so they can be
represented in the dictionary within a single mapping, as per
RFC 7230:

A recipient MAY combine multiple header fields with the same field name
into one «field-name: field-value» pair, without changing the semantics
of the message, by appending each subsequent field value to the combined
field value in order, separated by a comma.

Cookies

If a response contains some Cookies, you can quickly access them:

>>> url = 'http://example.com/some/cookie/setting/url'
>>> r = requests.get(url)

>>> r.cookies['example_cookie_name']
'example_cookie_value'

To send your own cookies to the server, you can use the cookies
parameter:

>>> url = 'https://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')

>>> r = requests.get(url, cookies=cookies)
>>> r.text
'{"cookies": {"cookies_are": "working"}}'

Cookies are returned in a :class:`~requests.cookies.RequestsCookieJar`,
which acts like a dict but also offers a more complete interface,
suitable for use over multiple domains or paths. Cookie jars can
also be passed in to requests:

>>> jar = requests.cookies.RequestsCookieJar()
>>> jar.set('tasty_cookie', 'yum', domain='httpbin.org', path='/cookies')
>>> jar.set('gross_cookie', 'blech', domain='httpbin.org', path='/elsewhere')
>>> url = 'https://httpbin.org/cookies'
>>> r = requests.get(url, cookies=jar)
>>> r.text
'{"cookies": {"tasty_cookie": "yum"}}'

Redirection and History

By default Requests will perform location redirection for all verbs except
HEAD.

We can use the history property of the Response object to track redirection.

The :attr:`Response.history <requests.Response.history>` list contains the
:class:`Response <requests.Response>` objects that were created in order to
complete the request. The list is sorted from the oldest to the most recent
response.

For example, GitHub redirects all HTTP requests to HTTPS:

>>> r = requests.get('http://github.com/')

>>> r.url
'https://github.com/'

>>> r.status_code
200

>>> r.history
[<Response [301]>]

If you’re using GET, OPTIONS, POST, PUT, PATCH or DELETE, you can disable
redirection handling with the allow_redirects parameter:

>>> r = requests.get('http://github.com/', allow_redirects=False)

>>> r.status_code
301

>>> r.history
[]

If you’re using HEAD, you can enable redirection as well:

>>> r = requests.head('http://github.com/', allow_redirects=True)

>>> r.url
'https://github.com/'

>>> r.history
[<Response [301]>]

Timeouts

You can tell Requests to stop waiting for a response after a given number of
seconds with the timeout parameter. Nearly all production code should use
this parameter in nearly all requests. Failure to do so can cause your program
to hang indefinitely:

>>> requests.get('https://github.com/', timeout=0.001)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
requests.exceptions.Timeout: HTTPConnectionPool(host='github.com', port=80): Request timed out. (timeout=0.001)

Note

timeout is not a time limit on the entire response download;
rather, an exception is raised if the server has not issued a
response for timeout seconds (more precisely, if no bytes have been
received on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests do
not time out.

Errors and Exceptions

In the event of a network problem (e.g. DNS failure, refused connection, etc),
Requests will raise a :exc:`~requests.exceptions.ConnectionError` exception.

:meth:`Response.raise_for_status() <requests.Response.raise_for_status>` will
raise an :exc:`~requests.exceptions.HTTPError` if the HTTP request
returned an unsuccessful status code.

If a request times out, a :exc:`~requests.exceptions.Timeout` exception is
raised.

If a request exceeds the configured number of maximum redirections, a
:exc:`~requests.exceptions.TooManyRedirects` exception is raised.

All exceptions that Requests explicitly raises inherit from
:exc:`requests.exceptions.RequestException`.

Ready for more? Check out the :ref:`advanced <advanced>` section.

If you’re on the job market, consider taking this programming quiz. A substantial donation will be made to this project, if you find a job through this platform.

Источник

This document covers some of Requests more advanced features.

Session Objects¶

The Session object allows you to persist certain parameters across
requests. It also persists cookies across all requests made from the
Session instance, and will use urllib3’s connection pooling. So if
you’re making several requests to the same host, the underlying TCP
connection will be reused, which can result in a significant performance
increase (see HTTP persistent connection).

A Session object has all the methods of the main Requests API.

Let’s persist some cookies across requests:

s = requests.Session()

s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get('https://httpbin.org/cookies')

print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'

Sessions can also be used to provide default data to the request methods. This
is done by providing data to the properties on a Session object:

s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})

# both 'x-test' and 'x-test2' are sent
s.get('https://httpbin.org/headers', headers={'x-test2': 'true'})

Any dictionaries that you pass to a request method will be merged with the
session-level values that are set. The method-level parameters override session
parameters.

Note, however, that method-level parameters will not be persisted across
requests, even if using a session. This example will only send the cookies
with the first request, but not the second:

s = requests.Session()

r = s.get('https://httpbin.org/cookies', cookies={'from-my': 'browser'})
print(r.text)
# '{"cookies": {"from-my": "browser"}}'

r = s.get('https://httpbin.org/cookies')
print(r.text)
# '{"cookies": {}}'

If you want to manually add cookies to your session, use the
Cookie utility functions to manipulate
Session.cookies.

Sessions can also be used as context managers:

with requests.Session() as s:
    s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')

This will make sure the session is closed as soon as the with block is
exited, even if unhandled exceptions occurred.

Remove a Value From a Dict Parameter

Sometimes you’ll want to omit session-level keys from a dict parameter. To
do this, you simply set that key’s value to None in the method-level
parameter. It will automatically be omitted.

All values that are contained within a session are directly available to you.
See the Session API Docs to learn more.

Request and Response Objects¶

Whenever a call is made to requests.get() and friends, you are doing two
major things. First, you are constructing a Request object which will be
sent off to a server to request or query some resource. Second, a Response
object is generated once Requests gets a response back from the server.
The Response object contains all of the information returned by the server and
also contains the Request object you created originally. Here is a simple
request to get some very important information from Wikipedia’s servers:

>>> r = requests.get('https://en.wikipedia.org/wiki/Monty_Python')

If we want to access the headers the server sent back to us, we do this:

>>> r.headers
{'content-length': '56170', 'x-content-type-options': 'nosniff', 'x-cache':
'HIT from cp1006.eqiad.wmnet, MISS from cp1010.eqiad.wmnet', 'content-encoding':
'gzip', 'age': '3080', 'content-language': 'en', 'vary': 'Accept-Encoding,Cookie',
'server': 'Apache', 'last-modified': 'Wed, 13 Jun 2012 01:33:50 GMT',
'connection': 'close', 'cache-control': 'private, s-maxage=0, max-age=0,
must-revalidate', 'date': 'Thu, 14 Jun 2012 12:59:39 GMT', 'content-type':
'text/html; charset=UTF-8', 'x-cache-lookup': 'HIT from cp1006.eqiad.wmnet:3128,
MISS from cp1010.eqiad.wmnet:80'}

However, if we want to get the headers we sent the server, we simply access the
request, and then the request’s headers:

>>> r.request.headers
{'Accept-Encoding': 'identity, deflate, compress, gzip',
'Accept': '*/*', 'User-Agent': 'python-requests/1.2.0'}

Prepared Requests¶

Whenever you receive a Response object
from an API call or a Session call, the request attribute is actually the
PreparedRequest that was used. In some cases you may wish to do some extra
work to the body or headers (or anything else really) before sending a
request. The simple recipe for this is the following:

from requests import Request, Session

s = Session()

req = Request('POST', url, data=data, headers=headers)
prepped = req.prepare()

# do something with prepped.body
prepped.body = 'No, I want exactly this as the body.'

# do something with prepped.headers
del prepped.headers['Content-Type']

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

Since you are not doing anything special with the Request object, you
prepare it immediately and modify the PreparedRequest object. You then
send that with the other parameters you would have sent to requests.* or
Session.*.

However, the above code will lose some of the advantages of having a Requests
Session object. In particular,
Session-level state such as cookies will
not get applied to your request. To get a
PreparedRequest with that state
applied, replace the call to Request.prepare() with a call to
Session.prepare_request(), like this:

from requests import Request, Session

s = Session()
req = Request('GET',  url, data=data, headers=headers)

prepped = s.prepare_request(req)

# do something with prepped.body
prepped.body = 'Seriously, send exactly these bytes.'

# do something with prepped.headers
prepped.headers['Keep-Dead'] = 'parrot'

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

When you are using the prepared request flow, keep in mind that it does not take into account the environment.
This can cause problems if you are using environment variables to change the behaviour of requests.
For example: Self-signed SSL certificates specified in REQUESTS_CA_BUNDLE will not be taken into account.
As a result an SSL: CERTIFICATE_VERIFY_FAILED is thrown.
You can get around this behaviour by explicity merging the environment settings into your session:

from requests import Request, Session

s = Session()
req = Request('GET', url)

prepped = s.prepare_request(req)

# Merge environment settings into session
settings = s.merge_environment_settings(prepped.url, {}, None, None, None)
resp = s.send(prepped, **settings)

print(resp.status_code)

SSL Cert Verification¶

Requests verifies SSL certificates for HTTPS requests, just like a web browser.
By default, SSL verification is enabled, and Requests will throw a SSLError if
it’s unable to verify the certificate:

>>> requests.get('https://requestb.in')
requests.exceptions.SSLError: hostname 'requestb.in' doesn't match either of '*.herokuapp.com', 'herokuapp.com'

I don’t have SSL setup on this domain, so it throws an exception. Excellent. GitHub does though:

>>> requests.get('https://github.com')
<Response [200]>

You can pass verify the path to a CA_BUNDLE file or directory with certificates of trusted CAs:

>>> requests.get('https://github.com', verify='/path/to/certfile')

or persistent:

s = requests.Session()
s.verify = '/path/to/certfile'

Note

If verify is set to a path to a directory, the directory must have been processed using
the c_rehash utility supplied with OpenSSL.

This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.

Requests can also ignore verifying the SSL certificate if you set verify to False:

>>> requests.get('https://kennethreitz.org', verify=False)
<Response [200]>

By default, verify is set to True. Option verify only applies to host certs.

Client Side Certificates¶

You can also specify a local cert to use as client side certificate, as a single
file (containing the private key and the certificate) or as a tuple of both
files’ paths:

>>> requests.get('https://kennethreitz.org', cert=('/path/client.cert', '/path/client.key'))
<Response [200]>

or persistent:

s = requests.Session()
s.cert = '/path/client.cert'

If you specify a wrong path or an invalid cert, you’ll get a SSLError:

>>> requests.get('https://kennethreitz.org', cert='/wrong_path/client.pem')
SSLError: [Errno 336265225] _ssl.c:347: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib

Warning

The private key to your local certificate must be unencrypted.
Currently, Requests does not support using encrypted keys.

CA Certificates¶

Requests uses certificates from the package certifi. This allows for users
to update their trusted certificates without changing the version of Requests.

Before version 2.16, Requests bundled a set of root CAs that it trusted,
sourced from the Mozilla trust store. The certificates were only updated
once for each Requests version. When certifi was not installed, this led to
extremely out-of-date certificate bundles when using significantly older
versions of Requests.

For the sake of security we recommend upgrading certifi frequently!

Body Content Workflow¶

By default, when you make a request, the body of the response is downloaded
immediately. You can override this behaviour and defer downloading the response
body until you access the Response.content
attribute with the stream parameter:

tarball_url = 'https://github.com/requests/requests/tarball/master'
r = requests.get(tarball_url, stream=True)

At this point only the response headers have been downloaded and the connection
remains open, hence allowing us to make content retrieval conditional:

if int(r.headers['content-length']) < TOO_LONG:
  content = r.content
  ...

You can further control the workflow by use of the Response.iter_content()
and Response.iter_lines() methods.
Alternatively, you can read the undecoded body from the underlying
urllib3 urllib3.HTTPResponse at
Response.raw.

If you set stream to True when making a request, Requests cannot
release the connection back to the pool unless you consume all the data or call
Response.close. This can lead to
inefficiency with connections. If you find yourself partially reading request
bodies (or not reading them at all) while using stream=True, you should
make the request within a with statement to ensure it’s always closed:

with requests.get('https://httpbin.org/get', stream=True) as r:
    # Do things with the response here.

Keep-Alive¶

Excellent news — thanks to urllib3, keep-alive is 100% automatic within a session!
Any requests that you make within a session will automatically reuse the appropriate
connection!

Note that connections are only released back to the pool for reuse once all body
data has been read; be sure to either set stream to False or read the
content property of the Response object.

Streaming Uploads¶

Requests supports streaming uploads, which allow you to send large streams or
files without reading them into memory. To stream and upload, simply provide a
file-like object for your body:

with open('massive-body', 'rb') as f:
    requests.post('http://some.url/streamed', data=f)

Warning

It is strongly recommended that you open files in binary
mode. This is because Requests may attempt to provide
the Content-Length header for you, and if it does this value
will be set to the number of bytes in the file. Errors may occur
if you open the file in text mode.

Chunk-Encoded Requests¶

Requests also supports Chunked transfer encoding for outgoing and incoming requests.
To send a chunk-encoded request, simply provide a generator (or any iterator without
a length) for your body:

def gen():
    yield 'hi'
    yield 'there'

requests.post('http://some.url/chunked', data=gen())

For chunked encoded responses, it’s best to iterate over the data using
Response.iter_content(). In
an ideal situation you’ll have set stream=True on the request, in which
case you can iterate chunk-by-chunk by calling iter_content with a chunk_size
parameter of None. If you want to set a maximum size of the chunk,
you can set a chunk_size parameter to any integer.

POST Multiple Multipart-Encoded Files¶

You can send multiple files in one request. For example, suppose you want to
upload image files to an HTML form with a multiple file field ‘images’:

<input type="file" name="images" multiple="true" required="true"/>

To do that, just set files to a list of tuples of (form_field_name, file_info):

>>> url = 'https://httpbin.org/post'
>>> multiple_files = [
        ('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),
        ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))]
>>> r = requests.post(url, files=multiple_files)
>>> r.text
{
  ...
  'files': {'images': 'data:image/png;base64,iVBORw ....'}
  'Content-Type': 'multipart/form-data; boundary=3131623adb2043caaeb5538cc7aa0b3a',
  ...
}

Warning

Event Hooks¶

Requests has a hook system that you can use to manipulate portions of
the request process, or signal event handling.

Available hooks:

response:: The response generated from a Request.

You can assign a hook function on a per-request basis by passing a
{hook_name: callback_function} dictionary to the hooks request
parameter:

hooks={'response': print_url}

That callback_function will receive a chunk of data as its first
argument.

def print_url(r, *args, **kwargs):
    print(r.url)

If an error occurs while executing your callback, a warning is given.

If the callback function returns a value, it is assumed that it is to
replace the data that was passed in. If the function doesn’t return
anything, nothing else is affected.

def record_hook(r, *args, **kwargs):
    r.hook_called = True
    return r

Let’s print some request method arguments at runtime:

>>> requests.get('https://httpbin.org/', hooks={'response': print_url})
https://httpbin.org/
<Response [200]>

You can add multiple hooks to a single request. Let’s call two hooks at once:

>>> r = requests.get('https://httpbin.org/', hooks={'response': [print_url, record_hook]})
>>> r.hook_called
True

You can also add hooks to a Session instance. Any hooks you add will then
be called on every request made to the session. For example:

>>> s = requests.Session()
>>> s.hooks['response'].append(print_url)
>>> s.get('https://httpbin.org/')
 https://httpbin.org/
 <Response [200]>

A Session can have multiple hooks, which will be called in the order
they are added.

Custom Authentication¶

Requests allows you to use specify your own authentication mechanism.

Any callable which is passed as the auth argument to a request method will
have the opportunity to modify the request before it is dispatched.

Authentication implementations are subclasses of AuthBase,
and are easy to define. Requests provides two common authentication scheme
implementations in requests.auth: HTTPBasicAuth and
HTTPDigestAuth.

Let’s pretend that we have a web service that will only respond if the
X-Pizza header is set to a password value. Unlikely, but just go with it.

from requests.auth import AuthBase

class PizzaAuth(AuthBase):
    """Attaches HTTP Pizza Authentication to the given Request object."""
    def __init__(self, username):
        # setup any auth-related data here
        self.username = username

    def __call__(self, r):
        # modify and return the request
        r.headers['X-Pizza'] = self.username
        return r

Then, we can make a request using our Pizza Auth:

>>> requests.get('http://pizzabin.org/admin', auth=PizzaAuth('kenneth'))
<Response [200]>

Streaming Requests¶

With Response.iter_lines() you can easily
iterate over streaming APIs such as the Twitter Streaming
API. Simply
set stream to True and iterate over the response with
iter_lines:

import json
import requests

r = requests.get('https://httpbin.org/stream/20', stream=True)

for line in r.iter_lines():

    # filter out keep-alive new lines
    if line:
        decoded_line = line.decode('utf-8')
        print(json.loads(decoded_line))

When using decode_unicode=True with
Response.iter_lines() or
Response.iter_content(), you’ll want
to provide a fallback encoding in the event the server doesn’t provide one:

r = requests.get('https://httpbin.org/stream/20', stream=True)

if r.encoding is None:
    r.encoding = 'utf-8'

for line in r.iter_lines(decode_unicode=True):
    if line:
        print(json.loads(line))

Warning

iter_lines is not reentrant safe.
Calling this method multiple times causes some of the received data
being lost. In case you need to call it from multiple places, use
the resulting iterator object instead:

lines = r.iter_lines()
# Save the first line for later or just skip it

first_line = next(lines)

for line in lines:
    print(line)

Proxies¶

If you need to use a proxy, you can configure individual requests with the
proxies argument to any request method:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

requests.get('http://example.org', proxies=proxies)

You can also configure proxies by setting the environment variables
HTTP_PROXY and HTTPS_PROXY.

$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"

$ python
>>> import requests
>>> requests.get('http://example.org')

To use HTTP Basic Auth with your proxy, use the http://user:password@host/ syntax:

proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}

To give a proxy for a specific scheme and host, use the
scheme://hostname form for the key. This will match for
any request to the given scheme and exact hostname.

proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}

Note that proxy URLs must include the scheme.

SOCKS¶

New in version 2.10.0.

In addition to basic HTTP proxies, Requests also supports proxies using the
SOCKS protocol. This is an optional feature that requires that additional
third-party libraries be installed before use.

You can get the dependencies for this feature from pip:

$ pip install requests[socks]

Once you’ve installed those dependencies, using a SOCKS proxy is just as easy
as using a HTTP one:

proxies = {
    'http': 'socks5://user:pass@host:port',
    'https': 'socks5://user:pass@host:port'
}

Using the scheme socks5 causes the DNS resolution to happen on the client, rather than on the proxy server. This is in line with curl, which uses the scheme to decide whether to do the DNS resolution on the client or proxy. If you want to resolve the domains on the proxy server, use socks5h as the scheme.

Compliance¶

Requests is intended to be compliant with all relevant specifications and
RFCs where that compliance will not cause difficulties for users. This
attention to the specification can lead to some behaviour that may seem
unusual to those not familiar with the relevant specification.

Encodings¶

When you receive a response, Requests makes a guess at the encoding to
use for decoding the response when you access the Response.text attribute. Requests will first check for an
encoding in the HTTP header, and if none is present, will use chardet to attempt to guess the encoding.

The only time Requests will not do this is if no explicit charset
is present in the HTTP headers and the Content-Type
header contains text. In this situation, RFC 2616 specifies
that the default charset must be ISO-8859-1. Requests follows the
specification in this case. If you require a different encoding, you can
manually set the Response.encoding
property, or use the raw Response.content.

HTTP Verbs¶

Requests provides access to almost the full range of HTTP verbs: GET, OPTIONS,
HEAD, POST, PUT, PATCH and DELETE. The following provides detailed examples of
using these various verbs in Requests, using the GitHub API.

We will begin with the verb most commonly used: GET. HTTP GET is an idempotent
method that returns a resource from a given URL. As a result, it is the verb
you ought to use when attempting to retrieve data from a web location. An
example usage would be attempting to get information about a specific commit
from GitHub. Suppose we wanted commit a050faf on Requests. We would get it
like so:

>>> import requests
>>> r = requests.get('https://api.github.com/repos/requests/requests/git/commits/a050faf084662f3a352dd1a941f2c7c9f886d4ad')

We should confirm that GitHub responded correctly. If it has, we want to work
out what type of content it is. Do this like so:

>>> if r.status_code == requests.codes.ok:
...     print(r.headers['content-type'])
...
application/json; charset=utf-8

So, GitHub returns JSON. That’s great, we can use the r.json method to parse it into Python objects.

>>> commit_data = r.json()

>>> print(commit_data.keys())
[u'committer', u'author', u'url', u'tree', u'sha', u'parents', u'message']

>>> print(commit_data[u'committer'])
{u'date': u'2012-05-10T11:10:50-07:00', u'email': u'me@kennethreitz.com', u'name': u'Kenneth Reitz'}

>>> print(commit_data[u'message'])
makin' history

So far, so simple. Well, let’s investigate the GitHub API a little bit. Now,
we could look at the documentation, but we might have a little more fun if we
use Requests instead. We can take advantage of the Requests OPTIONS verb to
see what kinds of HTTP methods are supported on the url we just used.

>>> verbs = requests.options(r.url)
>>> verbs.status_code
500

Uh, what? That’s unhelpful! Turns out GitHub, like many API providers, don’t
actually implement the OPTIONS method. This is an annoying oversight, but it’s
OK, we can just use the boring documentation. If GitHub had correctly
implemented OPTIONS, however, they should return the allowed methods in the
headers, e.g.

>>> verbs = requests.options('http://a-good-website.com/api/cats')
>>> print(verbs.headers['allow'])
GET,HEAD,POST,OPTIONS

Turning to the documentation, we see that the only other method allowed for
commits is POST, which creates a new commit. As we’re using the Requests repo,
we should probably avoid making ham-handed POSTS to it. Instead, let’s play
with the Issues feature of GitHub.

This documentation was added in response to
Issue #482. Given that
this issue already exists, we will use it as an example. Let’s start by getting it.

>>> r = requests.get('https://api.github.com/repos/requests/requests/issues/482')
>>> r.status_code
200

>>> issue = json.loads(r.text)

>>> print(issue[u'title'])
Feature any http verb in docs

>>> print(issue[u'comments'])
3

Cool, we have three comments. Let’s take a look at the last of them.

>>> r = requests.get(r.url + u'/comments')
>>> r.status_code
200

>>> comments = r.json()

>>> print(comments[0].keys())
[u'body', u'url', u'created_at', u'updated_at', u'user', u'id']

>>> print(comments[2][u'body'])
Probably in the "advanced" section

Well, that seems like a silly place. Let’s post a comment telling the poster
that he’s silly. Who is the poster, anyway?

>>> print(comments[2][u'user'][u'login'])
kennethreitz

OK, so let’s tell this Kenneth guy that we think this example should go in the
quickstart guide instead. According to the GitHub API doc, the way to do this
is to POST to the thread. Let’s do it.

>>> body = json.dumps({u"body": u"Sounds great! I'll get right on it!"})
>>> url = u"https://api.github.com/repos/requests/requests/issues/482/comments"

>>> r = requests.post(url=url, data=body)
>>> r.status_code
404

Huh, that’s weird. We probably need to authenticate. That’ll be a pain, right?
Wrong. Requests makes it easy to use many forms of authentication, including
the very common Basic Auth.

>>> from requests.auth import HTTPBasicAuth
>>> auth = HTTPBasicAuth('fake@example.com', 'not_a_real_password')

>>> r = requests.post(url=url, data=body, auth=auth)
>>> r.status_code
201

>>> content = r.json()
>>> print(content[u'body'])
Sounds great! I'll get right on it.

Brilliant. Oh, wait, no! I meant to add that it would take me a while, because
I had to go feed my cat. If only I could edit this comment! Happily, GitHub
allows us to use another HTTP verb, PATCH, to edit this comment. Let’s do
that.

>>> print(content[u"id"])
5804413

>>> body = json.dumps({u"body": u"Sounds great! I'll get right on it once I feed my cat."})
>>> url = u"https://api.github.com/repos/requests/requests/issues/comments/5804413"

>>> r = requests.patch(url=url, data=body, auth=auth)
>>> r.status_code
200

Excellent. Now, just to torture this Kenneth guy, I’ve decided to let him
sweat and not tell him that I’m working on this. That means I want to delete
this comment. GitHub lets us delete comments using the incredibly aptly named
DELETE method. Let’s get rid of it.

>>> r = requests.delete(url=url, auth=auth)
>>> r.status_code
204
>>> r.headers['status']
'204 No Content'

Excellent. All gone. The last thing I want to know is how much of my ratelimit
I’ve used. Let’s find out. GitHub sends that information in the headers, so
rather than download the whole page I’ll send a HEAD request to get the
headers.

>>> r = requests.head(url=url, auth=auth)
>>> print(r.headers)
...
'x-ratelimit-remaining': '4995'
'x-ratelimit-limit': '5000'
...

Excellent. Time to write a Python program that abuses the GitHub API in all
kinds of exciting ways, 4995 more times.

Custom Verbs¶

From time to time you may be working with a server that, for whatever reason,
allows use or even requires use of HTTP verbs not covered above. One example of
this would be the MKCOL method some WEBDAV servers use. Do not fret, these can
still be used with Requests. These make use of the built-in .request
method. For example:

>>> r = requests.request('MKCOL', url, data=data)
>>> r.status_code
200 # Assuming your call was correct

Utilising this, you can make use of any method verb that your server allows.

Transport Adapters¶

As of v1.0.0, Requests has moved to a modular internal design. Part of the
reason this was done was to implement Transport Adapters, originally
described here. Transport Adapters provide a mechanism to define interaction
methods for an HTTP service. In particular, they allow you to apply per-service
configuration.

Requests ships with a single Transport Adapter, the HTTPAdapter. This adapter provides the default Requests
interaction with HTTP and HTTPS using the powerful urllib3 library. Whenever
a Requests Session is initialized, one of these is
attached to the Session object for HTTP, and one
for HTTPS.

Requests enables users to create and use their own Transport Adapters that
provide specific functionality. Once created, a Transport Adapter can be
mounted to a Session object, along with an indication of which web services
it should apply to.

>>> s = requests.Session()
>>> s.mount('https://github.com/', MyAdapter())

The mount call registers a specific instance of a Transport Adapter to a
prefix. Once mounted, any HTTP request made using that session whose URL starts
with the given prefix will use the given Transport Adapter.

Many of the details of implementing a Transport Adapter are beyond the scope of
this documentation, but take a look at the next example for a simple SSL use-
case. For more than that, you might look at subclassing the
BaseAdapter.

Example: Specific SSL Version¶

The Requests team has made a specific choice to use whatever SSL version is
default in the underlying library (urllib3). Normally this is fine, but from
time to time, you might find yourself needing to connect to a service-endpoint
that uses a version that isn’t compatible with the default.

You can use Transport Adapters for this by taking most of the existing
implementation of HTTPAdapter, and adding a parameter ssl_version that gets
passed-through to urllib3. We’ll make a Transport Adapter that instructs the
library to use SSLv3:

import ssl
from urllib3.poolmanager import PoolManager

from requests.adapters import HTTPAdapter


class Ssl3HttpAdapter(HTTPAdapter):
    """"Transport adapter" that allows us to use SSLv3."""

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(
            num_pools=connections, maxsize=maxsize,
            block=block, ssl_version=ssl.PROTOCOL_SSLv3)

Blocking Or Non-Blocking?¶

With the default Transport Adapter in place, Requests does not provide any kind
of non-blocking IO. The Response.content
property will block until the entire response has been downloaded. If
you require more granularity, the streaming features of the library (see
Streaming Requests) allow you to retrieve smaller quantities of the
response at a time. However, these calls will still block.

If you are concerned about the use of blocking IO, there are lots of projects
out there that combine Requests with one of Python’s asynchronicity frameworks.
Some excellent examples are requests-threads, grequests, and requests-futures.

Timeouts¶

Most requests to external servers should have a timeout attached, in case the
server is not responding in a timely manner. By default, requests do not time
out unless a timeout value is set explicitly. Without a timeout, your code may
hang for minutes or more.

The connect timeout is the number of seconds Requests will wait for your
client to establish a connection to a remote machine (corresponding to the
connect()) call on the socket. It’s a good practice to set connect timeouts
to slightly larger than a multiple of 3, which is the default TCP packet
retransmission window.

Once your client has connected to the server and sent the HTTP request, the
read timeout is the number of seconds the client will wait for the server
to send a response. (Specifically, it’s the number of seconds that the client
will wait between bytes sent from the server. In 99.9% of cases, this is the
time before the server sends the first byte).

If you specify a single value for the timeout, like this:

r = requests.get('https://github.com', timeout=5)

The timeout value will be applied to both the connect and the read
timeouts. Specify a tuple if you would like to set the values separately:

r = requests.get('https://github.com', timeout=(3.05, 27))

If the remote server is very slow, you can tell Requests to wait forever for
a response, by passing None as a timeout value and then retrieving a cup of
coffee.

r = requests.get('https://github.com', timeout=None)

Источник

24 Дек. 2015, Python, 342396 просмотров,

Стандартная библиотека Python имеет ряд готовых модулей по работе с HTTP.

urllib
httplib

Если уж совсем хочется хардкора, то можно и сразу с socket поработать. Но у всех этих модулей есть один большой недостаток — неудобство работы.

Во-первых, большое обилие классов и функций. Во-вторых, код получается вовсе не pythonic. Многие программисты любят Python за его элегантность и простоту, поэтому и был создан модуль, призванный решать проблему существующих и имя ему requests или HTTP For Humans. На момент написания данной заметки, последняя версия библиотеки — 2.9.1. С момента выхода Python версии 3.5 я дал себе негласное обещание писать новый код только на Py >= 3.5. Пора бы уже полностью перебираться на 3-ю ветку змеюки, поэтому в моих примерах print отныне является функцией, а не оператором

Что же умеет requests?

Для начала хочется показать как выглядит код работы с http, используя модули из стандартной библиотеки Python и код при работе с requests. В качестве мишени для стрельбы http запросами будет использоваться очень удобный сервис httpbin.org


>>> import urllib.request
>>> response = urllib.request.urlopen('https://httpbin.org/get')
>>> print(response.read())
b'{n  "args": {}, n  "headers": {n    "Accept-Encoding": "identity", n    "Host": "httpbin.org", n    "User-Agent": "Python-urllib/3.5"n  }, n  "origin": "95.56.82.136", n  "url": "https://httpbin.org/get"n}n'
>>> print(response.getheader('Server'))
nginx
>>> print(response.getcode())
200
>>>

Кстати, urllib.request это надстройка над «низкоуровневой» библиотекой httplib о которой я писал выше.

>>> import requests
>>> response = requests.get('https://httpbin.org/get')
>>> print(response.content)
b'{n  "args": {}, n  "headers": {n    "Accept": "*/*", n    "Accept-Encoding": "gzip, deflate", n    "Host": "httpbin.org", n    "User-Agent": "python-requests/2.9.1"n  }, n  "origin": "95.56.82.136", n  "url": "https://httpbin.org/get"n}n'
>>> response.json()
{'headers': {'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'python-requests/2.9.1', 'Host': 'httpbin.org', 'Accept': '*/*'}, 'args': {}, 'origin': '95.56.82.136', 'url': 'https://httpbin.org/get'}
>>> response.headers
{'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Server': 'nginx', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Origin': '*', 'Content-Length': '237', 'Date': 'Wed, 23 Dec 2015 17:56:46 GMT'}
>>> response.headers.get('Server')
'nginx'

В простых методах запросов значительных отличий у них не имеется. Но давайте взглянем на работы с Basic Auth:


>>> import urllib.request
>>> password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
>>> top_level_url = 'https://httpbin.org/basic-auth/user/passwd'
>>> password_mgr.add_password(None, top_level_url, 'user', 'passwd')
>>> handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
>>> opener = urllib.request.build_opener(handler)
>>> response = opener.open(top_level_url)
>>> response.getcode()
200
>>> response.read()
b'{n  "authenticated": true, n  "user": "user"n}n'


>>> import requests
>>> response = requests.get('https://httpbin.org/basic-auth/user/passwd', auth=('user', 'passwd'))
>>> print(response.content)
b'{n  "authenticated": true, n  "user": "user"n}n'
>>> print(response.json())
{'user': 'user', 'authenticated': True}

А теперь чувствуется разница между pythonic и non-pythonic? Я думаю разница на лицо. И несмотря на тот факт, что requests ничто иное как обёртка над urllib3, а последняя является надстройкой над стандартными средствами Python, удобство написания кода в большинстве случаев является приоритетом номер один.

В requests имеется:

Множество методов http аутентификации
Сессии с куками
Полноценная поддержка SSL
Различные методы-плюшки вроде .json(), которые вернут данные в нужном формате
Проксирование
Грамотная и логичная работа с исключениями

О последнем пункте мне бы хотелось поговорить чуточку подробнее.

Обработка исключений в requests

При работе с внешними сервисами никогда не стоит полагаться на их отказоустойчивость. Всё упадёт рано или поздно, поэтому нам, программистам, необходимо быть всегда к этому готовыми, желательно заранее и в спокойной обстановке.

Итак, как у requests дела обстоят с различными факапами в момент сетевых соединений? Для начала определим ряд проблем, которые могут возникнуть:

Хост недоступен. Обычно такого рода ошибка происходит из-за проблем конфигурирования DNS. (DNS lookup failure)
«Вылет» соединения по таймауту
Ошибки HTTP. Подробнее о HTTP кодах можно посмотреть здесь.
Ошибки SSL соединений (обычно при наличии проблем с SSL сертификатом: просрочен, не является доверенным и т.д.)

Базовым классом-исключением в requests является RequestException. От него наследуются все остальные

HTTPError
ConnectionError
Timeout
SSLError
ProxyError

И так далее. Полный список всех исключений можно посмотреть в requests.exceptions.

Timeout

В requests имеется 2 вида таймаут-исключений:

ConnectTimeout — таймаут на соединения
ReadTimeout — таймаут на чтение

>>> import requests
>>> try:
...     response = requests.get('https://httpbin.org/user-agent', timeout=(0.00001, 10))
... except requests.exceptions.ConnectTimeout:
...     print('Oops. Connection timeout occured!')
...     
Oops. Connection timeout occured!
>>> try:
...     response = requests.get('https://httpbin.org/user-agent', timeout=(10, 0.0001))
... except requests.exceptions.ReadTimeout:
...     print('Oops. Read timeout occured')
... except requests.exceptions.ConnectTimeout:
...     print('Oops. Connection timeout occured!')
...     
Oops. Read timeout occured

ConnectionError


>>> import requests
>>> try:
...     response = requests.get('http://urldoesnotexistforsure.bom')
... except requests.exceptions.ConnectionError:
...     print('Seems like dns lookup failed..')
...     
Seems like dns lookup failed..

HTTPError


>>> import requests
>>> try:
...     response = requests.get('https://httpbin.org/status/500')
...     response.raise_for_status()
... except requests.exceptions.HTTPError as err:
...     print('Oops. HTTP Error occured')
...     print('Response is: {content}'.format(content=err.response.content))
...     
Oops. HTTP Error occured
Response is: b''

Я перечислил основные виды исключений, которые покрывают, пожалуй, 90% всех проблем, возникающих при работе с http. Главное помнить, что если мы действительно намерены отловить что-то и обработать, то это необходимо явно запрограммировать, если же нам неважен тип конкретного исключения, то можно отлавливать общий базовый класс RequestException и действовать уже от конкретного случая, например, залоггировать исключение и выкинуть его дальше наверх. Кстати, о логгировании я напишу отдельный подробный пост.

У блога появился свой Telegram канал, где я стараюсь делиться интересными находками из сети на тему разработки программного обеспечения. Велком, как говорится

Полезные «плюшки»

httpbin.org очень полезный сервис для тестирования http клиентов, в частности удобен для тестирования нестандартного поведения сервиса
httpie консольный http клиент (замена curl) написанный на Python
responses mock библиотека для работы с requests
HTTPretty mock библиотека для работы с http модулями

💌 Присоединяйтесь к рассылке

Понравился контент? Пожалуйста, подпишись на рассылку.

Источник

Содержание

Введение в тему
Создание get и post запроса
Передача параметров в url
Содержимое ответа response
Бинарное содержимое ответа
Содержимое ответа в json
Необработанное содержимое ответа
Пользовательские заголовки
Более сложные post запросы
Post отправка multipart encoded файла
Коды состояния ответа
Заголовки ответов
Cookies
Редиректы и история
Тайм ауты
Ошибки и исключения

Введение в тему

Модуль python requests – это общепринятый стандарт для работы с запросами по протоколу HTTP.

Этот модуль избавляет Вас от необходимости работать с низкоуровневыми деталями. Работа с запросами становится простой и элегантной.

В этом уроке будут рассмотрены самые полезные функций библиотеки requests и различные способы их использования.

Перед использованием модуля его необходимо установить:

Создание get и post запроса

Сперва необходимо добавить модуль Requests в Ваш код:

Создадим запрос и получим ответ, содержащий страницу и все необходимые данные о ней.


import requests

response = requests.get('https://www.google.ru/')

В переменную response попадает ответ на запрос. Благодаря этому объекту можно использовать любую информацию, относящуюся к этому ответу.

Сделать POST запрос так же очень просто:


import requests

 

response = requests.post('https://www.google.ru/', data = {'foo':3})

Другие виды HTTP запросов, к примеру: PUT, DELETE, и прочих, выполнить ничуть не сложнее:


import requests

 

response = requests.put('https://www.google.ru/', data = {'foo':3})

response = requests.delete('https://www.google.ru/')

response = requests.head('https://www.google.ru/')

response = requests.options('https://www.google.ru/')

Передача параметров в url

Иногда может быть необходимо отправить различные данные вместе с запросом URL. При ручной настройке URL, параметры выглядят как пары ключ=значение после знака «?». Например, https://www.google.ru/search?q=Python. Модуль Requests предоставляет возможность передать эти параметры как словарь, применяя аргумент params. Если вы хотите передать q = Python и foo=’bar’ ресурсу google.ru/search, вы должны использовать следующий код:


import requests

params_dict = {'q':'Python', 'foo':'bar'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.url)

#Вывод:

https://www.google.ru/search?q=Python&foo=bar

Здесь мы видим, что URL был сформирован именно так, как это было задумано.

Пара ключ=значение, где значение равняется None, не будет добавлена к параметрам запроса URL.

Так же есть возможность передавать в запрос список параметров:


import requests

params_dict = {'q':'Python', 'foo':['bar', 'eggs']}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.url)
#Вывод:

https://www.google.ru/search?q=Python&foo=bar&foo=eggs

Содержимое ответа response

Код из предыдущего листинга создаёт объект Response, содержащий ответ сервера на наш запрос. Обратившись к его атрибуту .url можно просмотреть адрес, куда был направлен запрос. Атрибут .text позволяет просмотреть содержимое ответа. Вот как это работает:


import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.text)
#Вывод:<!doctype html><html lang="ru"><head><meta charset="UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png"…

Библиотека автоматически пытается определить кодировку ответа основываясь на его заголовках. Узнать, какую кодировку выбрал модуль, можно следующим образом:


import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.encoding)
#Вывод:

windows-1251

Можно так же самостоятельно установить кодировку используя атрибут .encoding.


import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
response.encoding = 'utf-8' # указываем необходимую кодировку вручную
print(response.encoding)
#Вывод:

utf-8

Бинарное содержимое ответа

Существует возможность просмотра ответа в виде байтов:


import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.content)
#Вывод:

b'<!doctype html><html lang="ru"><head><meta charset="UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" …

При передаче со сжатием ответ автоматически декодируется для Вас.

Содержимое ответа в json

Так же в Requests есть встроенная обработка ответов в формате JSON:

import requests
import json

response = requests.get(‘http://api.open-notify.org/astros.json’)
print(json.dumps(response.json(), sort_keys=True, indent=4))
#Вывод:

{

«message»: «success»,

«number»: 10,

«people»: [

{

«craft»: «ISS»,

«name»: «Mark Vande Hei»

{

«craft»: «ISS»,

«name»: «Oleg Novitskiy»

…

[/dm_code_snippet]

Если ответ не является JSON, то .json выбросит исключение:


import requests
import json

response = requests.get('https://www.google.ru/search')
print(json.dumps(response.json(), sort_keys=True, indent=4))
#Вывод:

…

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Необработанное содержимое ответа

Если Вам нужно получить доступ к ответу сервера в чистом виде на уровне сокета, обратитесь к атрибуту .raw. Для этого необходимо указать параметр stream=True в запросе. Этот параметр заставляет модуль читать данные по мере их прибытия.


import requests

response = requests.get('https://www.google.ru/', stream=True)
print(response.raw)
print('Q'*10)
print(response.raw.read(15))
#Вывод:

<urllib3.response.HTTPResponse object at 0x000001E368771FA0>

QQQQQQQQQQ

b'x1fx8bx08x00x00x00x00x00x02xffxc5[[sxdb'

Так же можно использовать метод .iter_content. Этот метод итерирует данные потокового ответа и это позволяет избежать чтения содержимого сразу в память для больших ответов. Параметр chunk_size – это количество байтов, которые он должен прочитать в памяти. Параметр chunk_size можно произвольно менять.


import requests

response = requests.get('https://www.google.ru/', stream=True)
print(response.iter_content)
print('Q'*10)
print([i for i in response.iter_content(chunk_size=256)])
#Вывод:

<bound method Response.iter_content of <Response [200]>>

QQQQQQQQQQ

[b'<!doctype html><html itemscope="" itemtype="http://sche', b'ma.org/WebPage" lang="ru"><head><meta content=…

response.iter_content будет автоматически декодировать сжатый ответ. Response.raw — чистый набор байтов, неизменённое содержимое ответа.

Пользовательские заголовки

Если необходимо установить заголовки в HTTP запросе, передайте словарь с ними в параметр headers. Значения заголовка должны быть типа string, bytestring или unicode. Имена заголовков не чувствительны к регистру символов.
В следующем примере мы устанавливаем информацию об используемом браузере:


import requests

response = requests.get('https://www.google.ru/', headers={'user-agent': 'unknown_browser'})
print(response.request.headers)
# Вывод:

{'user-agent': 'unknown_browser', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

Более сложные post запросы

Существует способ отправить данные так, будто это результат заполнения формы на сайте:


import requests

response = requests.post('https://httpbin.org/post', data={'foo': 'bar'})
print(response.text)
# Вывод:

{

"args": {},

"data": "",

"files": {},

"form": {

"foo": "bar"

},

"headers": {

…

Параметр data может иметь произвольное количество значений для каждого ключа. Для этого необходимо указать data в формате кортежа, либо в виде dict со списками значений.


import requests

response = requests.post('https://httpbin.org/post', data={'foo':['bar', 'eggs']})
print(response.json()['form'])
print('|'*10)
response = requests.post('https://httpbin.org/post', data=[('foo', 'bar'), ('foo', 'eggs')])
print(response.json()['form'])
# Вывод:

{'foo': ['bar', 'eggs']}

||||||||||

{'foo': ['bar', 'eggs']}

Если нужно отправить данные, не закодированные как данные формы, то передайте в запрос строку вместо словаря. Тогда данные отправятся в изначальном виде.


import requests

response = requests.post('https://httpbin.org/post', data={'foo': 'bar'})
print('URL:', response.request.url)
print('Body:', response.request.body)
print('-' * 10)
response = requests.post('https://httpbin.org/post', data='foo=bar')
print('URL:', response.request.url)
print('Body:', response.request.body)
# Вывод:

URL: https://httpbin.org/post

URL: https://httpbin.org/post

Body: foo=bar

----------

URL: https://httpbin.org/post

Body: foo=bar

Post отправка multipart encoded файла

Запросы упрощают загрузку файлов с многостраничным кодированием (Multipart-Encoded):


import requests

url = 'https://httpbin.org/post'

files = {'file': open('report.xls', 'rb')}

response = requests.post(url, files=files)

print(response.text)

# Вывод:

{

...

"files": {

"file": "<censored...binary...data>"

},

...

}

Вы можете установить имя файла, content_type и заголовки в явном виде:


import requests

url = 'https://httpbin.org/post'

files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}

response = requests.post(url, files=files)

print(response.text)

# Вывод:

{

...

"files": {

"file": "<censored...binary...data>"

},

...

}

Можете отправить строки, которые будут приняты в виде файлов:


import requests

url = 'https://httpbin.org/post'

files = {'file': ('report.csv', 'some,data,to,sendnanother,row,to,sendn')}

response = requests.post(url, files=files)

print(response.text)

# Вывод:

{

...

"files": {

"file": "some,data,to,send\nanother,row,to,send\n"

},

...

}

Коды состояния ответа

Возможно, наиболее важные данные (первые – уж точно), которые вы можете получить, используя библиотеку requests, является код состояния ответа.

Так, 200 статус означает, что запрос выполнен успешно, тогда как 404 статус означает, что ресурс не найден.

Важнее всего то, с какой цифры начинается код состояния:

1XX — информация
2XX — успешно
3XX — перенаправление
4XX — ошибка клиента (ошибка на нашей стороне)
5XX — ошибка сервера (самые страшные коды для разработчика)

Используя атрибут .status_code можно получить статус, который вернул сервер:


import requests

response = requests.get('https://www.google.ru/')
print(response.status_code)

# Вывод:

200

.status_code вернул 200 — это означает, что запрос успешно выполнен и сервер вернул запрашиваемые данные.

При желании, такую информацию можно применить в Вашем Пайтон скрипте для принятия решений:


import requests

response = requests.get('https://www.google.ru/')
if response.status_code == 200:    print('Успех!')elif response.status_code == 404:    print('Страница куда-то пропала…')

# Вывод:

Успех!

Если код состояния response равен 200, то скрипт выведет «Успех!», но, если он равен 404, то скрипт вернёт «Страница куда-то пропала…».

Если применить модуль Response в условном выражении и проверить логическое значение его экземпляра (if response) то он продемонстрирует значение True, если код ответа находится в диапазоне между 200 и 400, и False во всех остальных случаях.

Упростим код из предыдущего примера:


import requests

response = requests.get('https://www.google.ru/fake/')
if response:
print('Успех!')
else:
print('Хьюстон, у нас проблемы!')
# Вывод:

Хьюстон, у нас проблемы!

Данный способ не проверяет, что код состояния равен именно 200.
Причиной этого является то, что response с кодом в диапазоне от 200 до 400, такие как 204 и 304, тоже являются успешными, ведь они возвращают обрабатываемый ответ. Следовательно, этот подход делит все запросы на успешные и неуспешные – не более того. Во многих случаях Вам потребуется более детальная обработка кодов состояния запроса.

Вы можете вызвать exception, если requests.get был неудачным. Такую конструкцию можно создать вызвав .raise_for_status() используя конструкцию try- except:


import requests

from requests.exceptions import HTTPError

for url in ['https://www.google.ru/', 'https://www.google.ru/invalid']:
try:
response = requests.get(url)

response.raise_for_status()
except HTTPError:
print(f'Возникла ошибка HTTP: {HTTPError}')
except Exception as err:
print(f'Возникла непредвиденная ошибка: {err}')
else:
print('Успех!')
# Вывод:

Успех!

Возникла ошибка HTTP: <class 'requests.exceptions.HTTPError'>

Заголовки ответов

Мы можем просматривать заголовки ответа сервера:


import requests

response = requests.get('https://www.google.ru/')
print(response.headers)
# Вывод:

{'Date': 'Sun, 27 Jun 2021 13:43:17 GMT', 'Expires': '-1', 'Cache-Control': 'private, max-age=0', 'Content-Type': 'text/html; charset=windows-1251', 'P3P': 'CP="This is not a P3P policy! See g.co/p3phelp for more info."', 'Content-Encoding': 'gzip', 'Server': 'gws', 'X-XSS-Protection': '0', 'X-Frame-Options': …

Cookies

Можно просмотреть файлы cookie, которые сервер отправляет вам обратно с помощью атрибута .cookies. Запросы также позволяют отправлять свои собственные cookie-файлы.

Чтобы добавить куки в запрос, Вы должны использовать dict, переданный в параметр cookie.


import requests

url = 'https://www.google.ru/'
headers = {'user-agent': 'your-own-user-agent/0.0.1'}
cookies = {'visit-month': 'February'}

response = requests.get(url, headers=headers, cookies=cookies)

print(response.request.headers)
# Вывод:

{'user-agent': 'your-own-user-agent/0.0.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'visit-month=February'}

Редиректы и история

По умолчанию модуль Requests выполняет редиректы для всех HTTP глаголов, кроме HEAD.

Существует возможность использовать параметр history объекта Response, чтобы отслеживать редиректы.

Например, GitHub перенаправляет все запросы HTTP на HTTPS:


import requests

response = requests.get('https://www.google.ru/')
print(response.url)
print(response.status_code)
print(response.history)
# Вывод:

https://www.google.ru/

200

[]

Тайм ауты

Так же легко можно управлять тем, сколько программа будет ждать возврат response. Время ожидания задаётся параметром timeout. Это очень важный параметр, так как, если его не использовать, написанный Вами скрипт может «зависнуть» в вечном ожидании ответа от сервера. Используем предыдущий код:


import requests
response = requests.get(‘https://www.google.ru/’, timeout=0.001)

print(response.url)

print(response.status_code)

print(response.history)

# Вывод:
…
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host=’www.google.ru’, port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000001E331681C70>, ‘Connection to www.google.ru timed out. (connect timeout=0.001)’))

Модуль не ждёт полной загрузки ответа. Исключение возникает, если сервер не отвечает (хотя бы один байт) за указанное время.

Ошибки и исключения

Если возникнет непредвиденная ситуация – ошибка соединения, модуль Requests выбросит эксепшн ConnectionError.

response.raise_for_status() возвращает объект HTTPError, если в процессе произошла ошибка. Его применяют для отладки модуля и, поэтому, он является неотъемлемой частью запросов Python.

Если выйдет время запроса, вызывается исключение Timeout. Если слишком много перенаправлений, то появится исключение TooManyRedirects.

Источник

Successful Connection Request

Python3

Exception Handling for HTTP Errors

Python3

General Exception Handling

Python3

Python3

Exception Handling for Missing Schema

Python3

Exception Handling for Connection Error

Python3

Putting Everything Together

Python3

Make a Request

Passing Parameters In URLs

Response Content

Binary Response Content

JSON Response Content

Raw Response Content

Custom Headers

More complicated POST requests

POST a Multipart-Encoded File

Response Status Codes

Response Headers

Cookies

Redirection and History

Timeouts

Errors and Exceptions

Session Objects¶

Request and Response Objects¶

Prepared Requests¶

SSL Cert Verification¶

Client Side Certificates¶

CA Certificates¶

Body Content Workflow¶

Keep-Alive¶

Streaming Uploads¶

Chunk-Encoded Requests¶

POST Multiple Multipart-Encoded Files¶

Event Hooks¶

Custom Authentication¶

Streaming Requests¶

Proxies¶

SOCKS¶

Compliance¶

Encodings¶

HTTP Verbs¶

Custom Verbs¶

Transport Adapters¶

Example: Specific SSL Version¶

Blocking Or Non-Blocking?¶

Timeouts¶

Что же умеет requests?

Обработка исключений в requests

Timeout

ConnectionError

HTTPError

Полезные «плюшки»

💌 Присоединяйтесь к рассылке

Введение в тему

Создание get и post запроса

Передача параметров в url

Содержимое ответа response

Бинарное содержимое ответа

Содержимое ответа в json

Необработанное содержимое ответа

Пользовательские заголовки

Более сложные post запросы

Post отправка multipart encoded файла

Коды состояния ответа

Заголовки ответов

Cookies

Редиректы и история

Тайм ауты

Ошибки и исключения