Http response error python

I'd like to raise a Python-standard exception when an HTTP response code from querying an API is not 200, but what specific exception should I use? For now I raise an OSError: if response.status_c...

I’d like to raise a Python-standard exception when an HTTP response code from querying an API is not 200, but what specific exception should I use? For now I raise an OSError:

if response.status_code != 200:
  raise OSError("Response " + str(response.status_code)
                  + ": " + response.content)

I’m aware of the documentation for built-in exceptions.

asked Jun 15, 2015 at 22:01

BoltzmannBrain's user avatar

BoltzmannBrainBoltzmannBrain

4,86210 gold badges44 silver badges75 bronze badges

1

You can simply call Response.raise_for_status() on your response:

>>> import requests
>>> url = 'http://stackoverflow.com/doesnt-exist'
>>> r = requests.get(url)
>>>
>>> print r.status_code
404
>>> r.raise_for_status()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/models.py", line 831, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found

This will raise a requests.HTTPError for any 4xx or 5xx response.

See the docs on Response Status Code for a more complete example.

Note that this does not exactly do what you asked (status != 200): It will not raise an exception for 201 Created or 204 No Content, or any of the 3xx redirects — but this is most likely the behavior you want: requests will just follow the redirects, and the other 2xx are usually just fine if you’re dealing with an API.

answered Jun 15, 2015 at 22:21

Lukas Graf's user avatar

The built-in Python exceptions are probably not a good fit for what you are doing. You will want to subclass the base class Exception, and throw your own custom exceptions based on each scenario you want to communicate.

A good example is how the Python Requests HTTP library defines its own exceptions:

In the event of a network problem (e.g. DNS failure, refused
connection, etc), Requests will raise a ConnectionError exception.

In the rare event of an invalid HTTP response, Requests will raise an
HTTPError exception.

If a request times out, a Timeout exception is raised.

If a request exceeds the configured number of maximum redirections, a
TooManyRedirects exception is raised.

All exceptions that Requests explicitly raises inherit from
requests.exceptions.RequestException.

answered Jun 15, 2015 at 22:10

Martin Konecny's user avatar

Martin KonecnyMartin Konecny

56.3k19 gold badges134 silver badges153 bronze badges

3

Источник

Содержание

  1. Введение в тему
  2. Создание get и post запроса
  3. Передача параметров в url
  4. Содержимое ответа response
  5. Бинарное содержимое ответа
  6. Содержимое ответа в json
  7. Необработанное содержимое ответа
  8. Пользовательские заголовки
  9. Более сложные post запросы
  10. Post отправка multipart encoded файла
  11. Коды состояния ответа
  12. Заголовки ответов
  13. Cookies
  14. Редиректы и история
  15. Тайм ауты
  16. Ошибки и исключения

Введение в тему

Модуль python requests – это общепринятый стандарт для работы с запросами по протоколу HTTP.

Этот модуль избавляет Вас от необходимости работать с низкоуровневыми деталями. Работа с запросами становится простой и элегантной.

В этом уроке будут рассмотрены самые полезные функций библиотеки requests и различные способы их использования.

Перед использованием модуля его необходимо установить:

Создание get и post запроса

Сперва необходимо добавить модуль Requests в Ваш код:

Создадим запрос и получим ответ, содержащий страницу и все необходимые данные о ней.



import requests

response = requests.get('https://www.google.ru/')

В переменную response попадает ответ на запрос. Благодаря этому объекту можно использовать любую информацию, относящуюся к этому ответу.

Сделать POST запрос так же очень просто:



import requests

 

response = requests.post('https://www.google.ru/', data = {'foo':3})

Другие виды HTTP запросов, к примеру: PUT, DELETE, и прочих, выполнить ничуть не сложнее:



import requests

 

response = requests.put('https://www.google.ru/', data = {'foo':3})

response = requests.delete('https://www.google.ru/')

response = requests.head('https://www.google.ru/')

response = requests.options('https://www.google.ru/')

Передача параметров в url

Иногда может быть необходимо отправить различные данные вместе с запросом URL. При ручной настройке URL, параметры выглядят как пары ключ=значение после знака «?». Например, https://www.google.ru/search?q=Python. Модуль Requests предоставляет возможность передать эти параметры как словарь, применяя аргумент params. Если вы хотите передать q = Python и foo=’bar’ ресурсу google.ru/search, вы должны использовать следующий код:



import requests

params_dict = {'q':'Python', 'foo':'bar'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.url)

#Вывод:

https://www.google.ru/search?q=Python&foo=bar

Здесь мы видим, что URL был сформирован именно так, как это было задумано.

Пара ключ=значение, где значение равняется None, не будет добавлена к параметрам запроса URL.

Так же есть возможность передавать в запрос список параметров:



import requests

params_dict = {'q':'Python', 'foo':['bar', 'eggs']}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.url)
#Вывод:

https://www.google.ru/search?q=Python&foo=bar&foo=eggs

Содержимое ответа response

Код из предыдущего листинга создаёт объект Response, содержащий ответ сервера на наш запрос. Обратившись к его атрибуту .url можно просмотреть адрес, куда был направлен запрос. Атрибут .text позволяет просмотреть содержимое ответа. Вот как это работает:



import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.text)
#Вывод:<!doctype html><html lang="ru"><head><meta charset="UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png"…

Библиотека автоматически пытается определить кодировку ответа основываясь на его заголовках. Узнать, какую кодировку выбрал модуль, можно следующим образом:



import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.encoding)
#Вывод:

windows-1251

Можно так же самостоятельно установить кодировку используя атрибут .encoding.



import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
response.encoding = 'utf-8' # указываем необходимую кодировку вручную
print(response.encoding)
#Вывод:

utf-8

Бинарное содержимое ответа

Существует возможность просмотра ответа в виде байтов:



import requests

params_dict = {'q':'Python'}
response = requests.get('https://www.google.ru/search', params=params_dict)
print(response.content)
#Вывод:

b'<!doctype html><html lang="ru"><head><meta charset="UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" …

При передаче со сжатием ответ автоматически декодируется для Вас.

Содержимое ответа в json

Так же в Requests есть встроенная обработка ответов в формате JSON:

import requests
import json

response = requests.get(‘http://api.open-notify.org/astros.json’)
print(json.dumps(response.json(), sort_keys=True, indent=4))
#Вывод:

{

«message»: «success»,

«number»: 10,

«people»: [

{

«craft»: «ISS»,

«name»: «Mark Vande Hei»

},

{

«craft»: «ISS»,

«name»: «Oleg Novitskiy»

},

…

[/dm_code_snippet]

Если ответ не является JSON, то .json выбросит исключение:



import requests
import json

response = requests.get('https://www.google.ru/search')
print(json.dumps(response.json(), sort_keys=True, indent=4))
#Вывод:

…

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Необработанное содержимое ответа

Если Вам нужно получить доступ к ответу сервера в чистом виде на уровне сокета, обратитесь к атрибуту .raw. Для этого необходимо указать параметр stream=True в запросе. Этот параметр заставляет модуль читать данные по мере их прибытия.



import requests

response = requests.get('https://www.google.ru/', stream=True)
print(response.raw)
print('Q'*10)
print(response.raw.read(15))
#Вывод:

<urllib3.response.HTTPResponse object at 0x000001E368771FA0>

QQQQQQQQQQ

b'x1fx8bx08x00x00x00x00x00x02xffxc5[[sxdb'

Так же можно использовать метод .iter_content. Этот метод итерирует данные потокового ответа и это позволяет избежать чтения содержимого сразу в память для больших ответов. Параметр chunk_size – это количество байтов, которые он должен прочитать в памяти.  Параметр chunk_size можно произвольно менять.



import requests

response = requests.get('https://www.google.ru/', stream=True)
print(response.iter_content)
print('Q'*10)
print([i for i in response.iter_content(chunk_size=256)])
#Вывод:

<bound method Response.iter_content of <Response [200]>>

QQQQQQQQQQ

[b'<!doctype html><html itemscope="" itemtype="http://sche', b'ma.org/WebPage" lang="ru"><head><meta content=…

response.iter_content будет автоматически декодировать сжатый ответ. Response.raw — чистый набор байтов, неизменённое содержимое ответа.

Пользовательские заголовки

Если необходимо установить заголовки в HTTP запросе, передайте словарь с ними в параметр headers. Значения заголовка должны быть типа string, bytestring или unicode. Имена заголовков не чувствительны к регистру символов.
В следующем примере мы устанавливаем информацию об используемом браузере:



import requests

response = requests.get('https://www.google.ru/', headers={'user-agent': 'unknown_browser'})
print(response.request.headers)
# Вывод:

{'user-agent': 'unknown_browser', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

Более сложные post запросы

Существует способ отправить данные так, будто это результат заполнения формы на сайте:



import requests

response = requests.post('https://httpbin.org/post', data={'foo': 'bar'})
print(response.text)
# Вывод:

{

"args": {},

"data": "",

"files": {},

"form": {

"foo": "bar"

},

"headers": {

…

Параметр data может иметь произвольное количество значений для каждого ключа. Для этого необходимо указать data в формате кортежа, либо в виде dict со списками значений.



import requests

response = requests.post('https://httpbin.org/post', data={'foo':['bar', 'eggs']})
print(response.json()['form'])
print('|'*10)
response = requests.post('https://httpbin.org/post', data=[('foo', 'bar'), ('foo', 'eggs')])
print(response.json()['form'])
# Вывод:

{'foo': ['bar', 'eggs']}

||||||||||

{'foo': ['bar', 'eggs']}

Если нужно отправить данные, не закодированные как данные формы, то передайте в запрос строку вместо словаря. Тогда данные отправятся в изначальном виде.



import requests

response = requests.post('https://httpbin.org/post', data={'foo': 'bar'})
print('URL:', response.request.url)
print('Body:', response.request.body)
print('-' * 10)
response = requests.post('https://httpbin.org/post', data='foo=bar')
print('URL:', response.request.url)
print('Body:', response.request.body)
# Вывод:

URL: https://httpbin.org/post

URL: https://httpbin.org/post

Body: foo=bar

----------

URL: https://httpbin.org/post

Body: foo=bar

Post отправка multipart encoded файла

Запросы упрощают загрузку файлов с многостраничным кодированием (Multipart-Encoded):



import requests

url = 'https://httpbin.org/post'

files = {'file': open('report.xls', 'rb')}

response = requests.post(url, files=files)

print(response.text)

# Вывод:

{

...

"files": {

"file": "<censored...binary...data>"

},

...

}

Вы можете установить имя файла, content_type и заголовки в явном виде:



import requests

url = 'https://httpbin.org/post'

files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}

response = requests.post(url, files=files)

print(response.text)

# Вывод:

{

...

"files": {

"file": "<censored...binary...data>"

},

...

}

Можете отправить строки, которые будут приняты в виде файлов:



import requests

url = 'https://httpbin.org/post'

files = {'file': ('report.csv', 'some,data,to,sendnanother,row,to,sendn')}

response = requests.post(url, files=files)

print(response.text)

# Вывод:

{

...

"files": {

"file": "some,data,to,send\nanother,row,to,send\n"

},

...

}

Коды состояния ответа

Возможно, наиболее важные данные (первые – уж точно), которые вы можете получить, используя библиотеку requests, является код состояния ответа.

Так, 200 статус означает, что запрос выполнен успешно, тогда как 404 статус означает, что ресурс не найден.

Важнее всего то, с какой цифры начинается код состояния:

  • 1XX — информация
  • 2XX — успешно
  • 3XX — перенаправление
  • 4XX — ошибка клиента (ошибка на нашей стороне)
  • 5XX — ошибка сервера (самые страшные коды для разработчика)

Используя атрибут .status_code можно получить статус, который вернул сервер:



import requests

response = requests.get('https://www.google.ru/')
print(response.status_code)

# Вывод:

200

.status_code вернул 200 — это означает, что запрос успешно выполнен и сервер вернул запрашиваемые данные.

При желании, такую информацию можно применить в Вашем Пайтон скрипте для принятия решений:



import requests

response = requests.get('https://www.google.ru/')
if response.status_code == 200:    print('Успех!')elif response.status_code == 404:    print('Страница куда-то пропала…')

# Вывод:

Успех!

Если код состояния response равен 200, то скрипт выведет «Успех!», но, если он равен 404, то скрипт вернёт «Страница куда-то пропала…».

Если применить модуль Response в условном выражении и проверить логическое значение его экземпляра (if response) то он продемонстрирует значение True, если код ответа находится в диапазоне между 200 и 400, и False во всех остальных случаях.

Упростим код из предыдущего примера:



import requests

response = requests.get('https://www.google.ru/fake/')
if response:
print('Успех!')
else:
print('Хьюстон, у нас проблемы!')
# Вывод:

Хьюстон, у нас проблемы!

Данный способ не проверяет, что код состояния равен именно 200.
Причиной этого является то, что response с кодом в диапазоне от 200 до 400, такие как 204 и 304, тоже являются успешными, ведь они возвращают обрабатываемый ответ. Следовательно, этот подход делит все запросы на успешные и неуспешные – не более того. Во многих случаях Вам потребуется более детальная обработка кодов состояния запроса.

Вы можете вызвать exception, если requests.get был неудачным. Такую конструкцию можно создать вызвав .raise_for_status() используя конструкцию try- except:



import requests

from requests.exceptions import HTTPError

for url in ['https://www.google.ru/', 'https://www.google.ru/invalid']:
try:
response = requests.get(url)

response.raise_for_status()
except HTTPError:
print(f'Возникла ошибка HTTP: {HTTPError}')
except Exception as err:
print(f'Возникла непредвиденная ошибка: {err}')
else:
print('Успех!')
# Вывод:

Успех!

Возникла ошибка HTTP: <class 'requests.exceptions.HTTPError'>

Заголовки ответов

Мы можем просматривать заголовки ответа сервера:



import requests

response = requests.get('https://www.google.ru/')
print(response.headers)
# Вывод:

{'Date': 'Sun, 27 Jun 2021 13:43:17 GMT', 'Expires': '-1', 'Cache-Control': 'private, max-age=0', 'Content-Type': 'text/html; charset=windows-1251', 'P3P': 'CP="This is not a P3P policy! See g.co/p3phelp for more info."', 'Content-Encoding': 'gzip', 'Server': 'gws', 'X-XSS-Protection': '0', 'X-Frame-Options': …

Cookies

Можно просмотреть файлы cookie, которые сервер отправляет вам обратно с помощью атрибута .cookies. Запросы также позволяют отправлять свои собственные cookie-файлы.

Чтобы добавить куки в запрос, Вы должны использовать dict, переданный в параметр cookie.



import requests

url = 'https://www.google.ru/'
headers = {'user-agent': 'your-own-user-agent/0.0.1'}
cookies = {'visit-month': 'February'}

response = requests.get(url, headers=headers, cookies=cookies)

print(response.request.headers)
# Вывод:

{'user-agent': 'your-own-user-agent/0.0.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'visit-month=February'}

Редиректы и история

По умолчанию модуль Requests выполняет редиректы для всех HTTP глаголов, кроме HEAD.

Существует возможность использовать параметр history объекта Response, чтобы отслеживать редиректы.

Например, GitHub перенаправляет все запросы HTTP на HTTPS:



import requests

response = requests.get('https://www.google.ru/')
print(response.url)
print(response.status_code)
print(response.history)
# Вывод:

https://www.google.ru/

200

[]

Тайм ауты

Так же легко можно управлять тем, сколько программа будет ждать возврат response. Время ожидания задаётся параметром timeout. Это очень важный параметр, так как, если его не использовать, написанный Вами скрипт может «зависнуть» в вечном ожидании ответа от сервера. Используем предыдущий код:



import requests


response = requests.get(‘https://www.google.ru/’, timeout=0.001)

print(response.url)

print(response.status_code)

print(response.history)

# Вывод:


…


raise ConnectTimeout(e, request=request)





  



                       
  






requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host=’www.google.ru’, port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000001E331681C70>, ‘Connection to www.google.ru timed out. (connect timeout=0.001)’))

Модуль не ждёт полной загрузки ответа. Исключение возникает, если сервер не отвечает (хотя бы один байт) за указанное время.

Ошибки и исключения

Если возникнет непредвиденная ситуация – ошибка соединения, модуль Requests выбросит эксепшн ConnectionError.

response.raise_for_status() возвращает объект HTTPError, если в процессе произошла ошибка. Его применяют для отладки модуля и, поэтому, он является неотъемлемой частью запросов Python.

Если выйдет время запроса, вызывается исключение Timeout. Если слишком много перенаправлений, то появится исключение TooManyRedirects.







Источник






 24 Дек. 2015, Python, 342273 просмотров,


Стандартная библиотека Python имеет ряд готовых модулей по работе с HTTP.


    

  • urllib
    • 

  • httplib
    • 



Если уж совсем хочется хардкора, то можно и сразу с socket поработать. Но у всех этих модулей есть один большой недостаток — неудобство работы.





  



                       
  






Во-первых, большое обилие классов и функций. Во-вторых, код получается вовсе не pythonic. Многие программисты любят Python за его элегантность и простоту, поэтому и был создан модуль, призванный решать проблему существующих и имя ему requests или HTTP For Humans. На момент написания данной заметки, последняя версия библиотеки — 2.9.1. С момента выхода Python версии 3.5 я дал себе негласное обещание писать новый код только на Py >= 3.5. Пора бы уже полностью перебираться на 3-ю ветку змеюки, поэтому в моих примерах print отныне является функцией, а не оператором  :-) 


Что же умеет requests?


Для начала хочется показать как выглядит код работы с http, используя модули из стандартной библиотеки Python и код при работе с requests. В качестве мишени для стрельбы http запросами будет использоваться очень удобный сервис httpbin.org



>>> import urllib.request
>>> response = urllib.request.urlopen('https://httpbin.org/get')
>>> print(response.read())
b'{n  "args": {}, n  "headers": {n    "Accept-Encoding": "identity", n    "Host": "httpbin.org", n    "User-Agent": "Python-urllib/3.5"n  }, n  "origin": "95.56.82.136", n  "url": "https://httpbin.org/get"n}n'
>>> print(response.getheader('Server'))
nginx
>>> print(response.getcode())
200
>>> 



Кстати, urllib.request это надстройка над «низкоуровневой» библиотекой httplib о которой я писал выше.


>>> import requests
>>> response = requests.get('https://httpbin.org/get')
>>> print(response.content)
b'{n  "args": {}, n  "headers": {n    "Accept": "*/*", n    "Accept-Encoding": "gzip, deflate", n    "Host": "httpbin.org", n    "User-Agent": "python-requests/2.9.1"n  }, n  "origin": "95.56.82.136", n  "url": "https://httpbin.org/get"n}n'
>>> response.json()
{'headers': {'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'python-requests/2.9.1', 'Host': 'httpbin.org', 'Accept': '*/*'}, 'args': {}, 'origin': '95.56.82.136', 'url': 'https://httpbin.org/get'}
>>> response.headers
{'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Server': 'nginx', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Origin': '*', 'Content-Length': '237', 'Date': 'Wed, 23 Dec 2015 17:56:46 GMT'}
>>> response.headers.get('Server')
'nginx'



В простых методах запросов значительных отличий у них не имеется. Но давайте взглянем на работы с Basic Auth:



>>> import urllib.request
>>> password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
>>> top_level_url = 'https://httpbin.org/basic-auth/user/passwd'
>>> password_mgr.add_password(None, top_level_url, 'user', 'passwd')
>>> handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
>>> opener = urllib.request.build_opener(handler)
>>> response = opener.open(top_level_url)
>>> response.getcode()
200
>>> response.read()
b'{n  "authenticated": true, n  "user": "user"n}n'




>>> import requests
>>> response = requests.get('https://httpbin.org/basic-auth/user/passwd', auth=('user', 'passwd'))
>>> print(response.content)
b'{n  "authenticated": true, n  "user": "user"n}n'
>>> print(response.json())
{'user': 'user', 'authenticated': True}



А теперь чувствуется разница между pythonic и non-pythonic? Я думаю разница на лицо. И несмотря на тот факт, что requests ничто иное как обёртка над urllib3, а последняя является надстройкой над стандартными средствами Python, удобство написания кода в большинстве случаев является приоритетом номер один.


В requests имеется:


    

  • Множество методов http аутентификации
    • 

  • Сессии с куками
    • 

  • Полноценная поддержка SSL
    • 

  • Различные методы-плюшки вроде .json(), которые вернут данные в нужном формате
    • 

  • Проксирование
    • 

  • Грамотная и логичная работа с исключениями
    • 



О последнем пункте мне бы хотелось поговорить чуточку подробнее.


Обработка исключений в requests


При работе с внешними сервисами никогда не стоит полагаться на их отказоустойчивость. Всё упадёт рано или поздно, поэтому нам, программистам, необходимо быть всегда к этому готовыми, желательно заранее и в спокойной обстановке.


Итак, как у requests дела обстоят с различными факапами в момент сетевых соединений? Для начала определим ряд проблем, которые могут возникнуть:


    

  • Хост недоступен. Обычно такого рода ошибка происходит из-за проблем конфигурирования DNS. (DNS lookup failure)
    • 

  • «Вылет» соединения по таймауту
    • 

  • Ошибки HTTP. Подробнее о HTTP кодах можно посмотреть здесь.
    • 

  • Ошибки SSL соединений (обычно при наличии проблем с SSL сертификатом: просрочен, не является доверенным и т.д.)
    • 



Базовым классом-исключением в requests является RequestException. От него наследуются все остальные





  



                       
  






    

  • HTTPError
    • 

  • ConnectionError
    • 

  • Timeout
    • 

  • SSLError
    • 

  • ProxyError
    • 



И так далее. Полный список всех исключений можно посмотреть в requests.exceptions.


Timeout


В requests имеется 2 вида таймаут-исключений:


    

  • ConnectTimeout — таймаут на соединения
    • 

  • ReadTimeout — таймаут на чтение
    • 



>>> import requests
>>> try:
...     response = requests.get('https://httpbin.org/user-agent', timeout=(0.00001, 10))
... except requests.exceptions.ConnectTimeout:
...     print('Oops. Connection timeout occured!')
...     
Oops. Connection timeout occured!
>>> try:
...     response = requests.get('https://httpbin.org/user-agent', timeout=(10, 0.0001))
... except requests.exceptions.ReadTimeout:
...     print('Oops. Read timeout occured')
... except requests.exceptions.ConnectTimeout:
...     print('Oops. Connection timeout occured!')
...     
Oops. Read timeout occured



ConnectionError



>>> import requests
>>> try:
...     response = requests.get('http://urldoesnotexistforsure.bom')
... except requests.exceptions.ConnectionError:
...     print('Seems like dns lookup failed..')
...     
Seems like dns lookup failed..


HTTPError



>>> import requests
>>> try:
...     response = requests.get('https://httpbin.org/status/500')
...     response.raise_for_status()
... except requests.exceptions.HTTPError as err:
...     print('Oops. HTTP Error occured')
...     print('Response is: {content}'.format(content=err.response.content))
...     
Oops. HTTP Error occured
Response is: b''



Я перечислил основные виды исключений, которые покрывают, пожалуй, 90% всех проблем, возникающих при работе с http. Главное помнить, что если мы действительно намерены отловить что-то и обработать, то это необходимо явно запрограммировать, если же нам неважен тип конкретного исключения, то можно отлавливать общий базовый класс RequestException и действовать уже от конкретного случая, например, залоггировать исключение и выкинуть его дальше наверх. Кстати, о логгировании я напишу отдельный подробный пост.


У блога появился свой Telegram канал, где я стараюсь делиться интересными находками из сети на тему разработки программного обеспечения. Велком, как говорится  :) 


Полезные «плюшки»


    

  • httpbin.org очень полезный сервис для тестирования http клиентов, в частности удобен для тестирования нестандартного поведения сервиса
    • 

  • httpie консольный http клиент (замена curl) написанный на Python
    • 

  • responses mock библиотека для работы с requests
    • 

  • HTTPretty mock библиотека для работы с http модулями
    • 





💌 Присоединяйтесь к рассылке


Понравился контент? Пожалуйста, подпишись на рассылку.








Источник






If you need to make HTTP requests with Python, then you may find yourself directed to the brilliant requests library. Though it’s a great library, you may have noticed that it’s not a built-in part of Python. If you prefer, for whatever reason, to limit your dependencies and stick to standard-library Python, then you can reach for urllib.request!


In this tutorial, you’ll:


    

  • Learn how to make basic HTTP requests with urllib.request
    • 

  • Dive into the nuts and bolts of an HTTP message and how urllib.request represents it 
    • 

  • Understand how to deal with character encodings of HTTP messages
    • 

  • Explore some common errors when using urllib.request and learn how to resolve them
    • 

  • Dip your toes into the world of authenticated requests with urllib.request
    • 

  • Understand why both urllib and the requests library exist and when to use one or the other
    • 



If you’ve heard of HTTP requests, including GET and POST, then you’re probably ready for this tutorial. Also, you should’ve already used Python to read and write to files, ideally with a context manager, at least once.


Ultimately, you’ll find that making a request doesn’t have to be a frustrating experience, although it does tend to have that reputation. Many of the issues that you tend to run into are due to the inherent complexity of this marvelous thing called the Internet. The good news is that the urllib.request module can help to demystify much of this complexity.




Basic HTTP GET Requests With urllib.request


Before diving into the deep end of what an HTTP request is and how it works, you’re going to get your feet wet by making a basic GET request to a sample URL. You’ll also make a GET request to a mock REST API for some JSON data. In case you’re wondering about POST Requests, you’ll be covering them later in the tutorial, once you have some more knowledge of urllib.request.





  



                       
  






To get started, you’ll make a request to www.example.com, and the server will return an HTTP message. Ensure that you’re using Python 3 or above, and then use the urlopen() function from urllib.request:




>>>


>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> body[:15]
b'<!doctype html>'





In this example, you import urlopen() from urllib.request. Using the context manager with, you make a request and receive a response with urlopen(). Then you read the body of the response and close the response object. With that, you display the first fifteen positions of the body, noting that it looks like an HTML document.


There you are! You’ve successfully made a request, and you received a response. By inspecting the content, you can tell that it’s likely an HTML document. Note that the printed output of the body is preceded by b. This indicates a bytes literal, which you may need to decode. Later in the tutorial, you’ll learn how to turn bytes into a string, write them to a file, or parse them into a dictionary.


The process is only slightly different if you want to make calls to REST APIs to get JSON data. In the following example, you’ll make a request to {JSON}Placeholder for some fake to-do data:




>>>


>>> from urllib.request import urlopen
>>> import json
>>> url = "https://jsonplaceholder.typicode.com/todos/1"
>>> with urlopen(url) as response:
...     body = response.read()
...
>>> todo_item = json.loads(body)
>>> todo_item
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}





In this example, you’re doing pretty much the same as in the previous example. But in this one, you import urllib.request and json, using the json.loads() function with body to decode and parse the returned JSON bytes into a Python dictionary. Voila!


If you’re lucky enough to be using error-free endpoints, such as the ones in these examples, then maybe the above is all that you need from urllib.request. Then again, you may find that it’s not enough.


Now, before doing some urllib.request troubleshooting, you’ll first gain an understanding of the underlying structure of HTTP messages and learn how urllib.request handles them. This understanding will provide a solid foundation for troubleshooting many different kinds of issues.






The Nuts and Bolts of HTTP Messages


To understand some of the issues that you may encounter when using urllib.request, you’ll need to examine how a response is represented by urllib.request. To do that, you’ll benefit from a high-level overview of what an HTTP message is, which is what you’ll get in this section.





  



                       
  






Before the high-level overview, a quick note on reference sources. If you want to get into the technical weeds, the Internet Engineering Task Force (IETF) has an extensive set of Request for Comments (RFC) documents. These documents end up becoming the actual specifications for things like HTTP messages. RFC 7230, part 1: Message Syntax and Routing, for example, is all about the HTTP message.


If you’re looking for some reference material that’s a bit easier to digest than RFCs, then the Mozilla Developer Network (MDN) has a great range of reference articles. For example, their article on HTTP messages, while still technical, is a lot more digestible.


Now that you know about these essential sources of reference information, in the next section you’ll get a beginner-friendly overview of HTTP messages.




Understanding What an HTTP Message Is


In a nutshell, an HTTP message can be understood as text, transmitted as a stream of bytes, structured to follow the guidelines specified by RFC 7230. A decoded HTTP message can be as simple as two lines:




GET / HTTP/1.1
Host: www.google.com





This specifies a GET request at the root (/) using the HTTP/1.1 protocol. The one and only header required is the host, www.google.com. The target server has enough information to make a response with this information.


A response is similar in structure to a request. HTTP messages have two main parts, the metadata and the body. In the request example above, the message is all metadata with no body. The response, on the other hand, does have two parts:




HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
Server: gws
(... other headers ...)

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"
...





The response starts with a status line that specifies the HTTP protocol HTTP/1.1 and the status 200 OK. After the status line, you get many key-value pairs, such as Server: gws, representing all the response headers. This is the metadata of the response.


After the metadata, there’s a blank line, which serves as the divider between the headers and the body. Everything that follows the blank line makes up the body. This is the part that gets read when you’re using urllib.request.


You can assume that all HTTP messages follow these specifications, but it’s possible that some may break these rules or follow an older specification. It’s exceptionally rare for this to cause any issues, though. So, just keep it in the back of your mind in case you run into a strange bug!


In the next section, you’ll see how urllib.request deals with raw HTTP messages.





  



                       
  










Understanding How urllib.request Represents an HTTP Message


The main representation of an HTTP message that you’ll be interacting with when using urllib.request is the HTTPResponse object. The urllib.request module itself depends on the low-level http module, which you don’t need to interact with directly. You do end up using some of the data structures that http provides, though, such as HTTPResponse and HTTPMessage.


When you make a request with urllib.request.urlopen(), you get an HTTPResponse object in return. Spend some time exploring the HTTPResponse object with pprint() and dir() to see all the different methods and properties that belong to it:




>>>


>>> from urllib.request import urlopen
>>> from pprint import pprint
>>> with urlopen("https://www.example.com") as response:
...     pprint(dir(response))
...





To reveal the output of this code snippet, click to expand the collapsible section below:




['__abstractmethods__',
 '__class__',
 '__del__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__enter__',
 '__eq__',
 '__exit__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__iter__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__next__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '_abc_impl',
 '_checkClosed',
 '_checkReadable',
 '_checkSeekable',
 '_checkWritable',
 '_check_close',
 '_close_conn',
 '_get_chunk_left',
 '_method',
 '_peek_chunked',
 '_read1_chunked',
 '_read_and_discard_trailer',
 '_read_next_chunk_size',
 '_read_status',
 '_readall_chunked',
 '_readinto_chunked',
 '_safe_read',
 '_safe_readinto',
 'begin',
 'chunk_left',
 'chunked',
 'close',
 'closed',
 'code',
 'debuglevel',
 'detach',
 'fileno',
 'flush',
 'fp',
 'getcode',
 'getheader',
 'getheaders',
 'geturl',
 'headers',
 'info',
 'isatty',
 'isclosed',
 'length',
 'msg',
 'peek',
 'read',
 'read1',
 'readable',
 'readinto',
 'readinto1',
 'readline',
 'readlines',
 'reason',
 'seek',
 'seekable',
 'status',
 'tell',
 'truncate',
 'url',
 'version',
 'will_close',
 'writable',
 'write',
 'writelines']





That’s a lot of methods and properties, but you’ll only end up using a handful of these . Apart from .read(), the important ones usually involve getting information about the headers.


One way to inspect all the headers is to access the .headers attribute of the HTTPResponse object. This will return an HTTPMessage object. Conveniently, you can treat an HTTPMessage like a dictionary by calling .items() on it to get all the headers as tuples:




>>>


>>> with urlopen("https://www.example.com") as response:
...     pass
...
>>> response.headers
<http.client.HTTPMessage object at 0x000001E029D9F4F0>
>>> pprint(response.headers.items())
[('Accept-Ranges', 'bytes'),
 ('Age', '398424'),
 ('Cache-Control', 'max-age=604800'),
 ('Content-Type', 'text/html; charset=UTF-8'),
 ('Date', 'Tue, 25 Jan 2022 12:18:53 GMT'),
 ('Etag', '"3147526947"'),
 ('Expires', 'Tue, 01 Feb 2022 12:18:53 GMT'),
 ('Last-Modified', 'Thu, 17 Oct 2019 07:18:26 GMT'),
 ('Server', 'ECS (nyb/1D16)'),
 ('Vary', 'Accept-Encoding'),
 ('X-Cache', 'HIT'),
 ('Content-Length', '1256'),
 ('Connection', 'close')]





Now you have access to all the response headers! You probably won’t need most of this information, but rest assured that some applications do use it. For example, your browser might use the headers to read the response, set cookies, and determine an appropriate cache lifetime.


There are convenience methods to get the headers from an HTTPResponse object because it’s quite a common operation. You can call .getheaders() directly on the HTTPResponse object, which will return exactly the same list of tuples as above. If you’re only interested in one header, say the Server header, then you can use the singular .getheader("Server") on HTTPResponse or use the square bracket ([]) syntax on .headers from HTTPMessage:




>>>





  



                       
  






>>> response.getheader("Server")
'ECS (nyb/1D16)'
>>> response.headers["Server"]
'ECS (nyb/1D16)'





Truth be told, you probably won’t need to interact with the headers directly like this. The information that you’re most likely to need will probably already have some built-in helper methods, but now you know, in case you ever need to dig deeper!






Closing an HTTPResponse


The HTTPResponse object has a lot in common with the file object. The HTTPResponse class inherits from the IOBase class, as do file objects, which means that you have to be mindful of opening and closing.


In simple programs, you’re not likely to notice any issues if you forget to close HTTPResponse objects. For more complex projects, though, this can significantly slow execution and cause bugs that are difficult to pinpoint.


Problems arise because input/output (I/O) streams are limited. Each HTTPResponse requires a stream to be kept clear while it’s being read. If you never close your streams, this will eventually prevent any other stream from being opened, and it might interfere with other programs or even your operating system.


So, make sure you close your HTTPResponse objects! For your convenience, you can use a context manager, as you’ve seen in the examples. You can also achieve the same result by explicitly calling .close() on the response object:




>>>


>>> from urllib.request import urlopen
>>> response = urlopen("https://www.example.com")
>>> body = response.read()
>>> response.close()





In this example, you don’t use a context manager, but instead close the response stream explicitly. The above example still has an issue, though, because an exception may be raised before the call to .close(), preventing the proper teardown. To make this call unconditional, as it should be, you can use a try … except block with both an else and a finally clause:




>>>


>>> from urllib.request import urlopen
>>> response = None
>>> try:
...     response = urlopen("https://www.example.com")
... except Exception as ex:
...     print(ex)
... else:
...     body = response.read()
... finally:
...     if response is not None:
...         response.close()





In this example, you achieve an unconditional call to .close() by using the finally block, which will always run regardless of exceptions raised. The code in the finally block first checks if the response object exists with is not None, and then closes it.


That said, this is exactly what a a context manager does, and the with syntax is generally preferred. Not only is the with syntax less verbose and more readable, but it also protects you from pesky errors of omission. Put another way, it’s a far better guard against accidentally forgetting to close the object:





  



                       
  








>>>


>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     response.read(50)
...     response.read(50)
...
b'<!doctype html>n<html>n<head>n    <title>Example D'
b'omain</title>nn    <meta charset="utf-8" />n    <m'





In this example, you import urlopen() from the urllib.request module. You use the with keyword with .urlopen() to assign the HTTPResponse object to the variable response. Then, you read the first fifty bytes of the response and then read the following fifty bytes, all within the with block. Finally, you close the with block, which executes the request and runs the lines of code within its block.


With this code, you cause two sets of fifty bytes each to be displayed. The HTTPResponse object will close once you exit the with block scope, meaning that the .read() method will only return empty bytes objects:




>>>


>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
...     response.read(50)
...
b'<!doctype html>n<html>n<head>n    <title>Example D'
>>> response.read(50)
b''





In this example, the second call to read fifty bytes is outside the with scope. Being outside the with block means that HTTPResponse is closed, even though you can still access the variable. If you try to read from HTTPResponse when it’s closed, it’ll return an empty bytes object.


Another point to note is that you can’t reread a response once you’ve read all the way to the end:




>>>


>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
...     first_read = response.read()
...     second_read = response.read()
...
>>> len(first_read)
1256
>>> len(second_read)
0





This example shows that once you’ve read a response, you can’t read it again. If you’ve fully read the response, the subsequent attempt just returns an empty bytes object even though the response isn’t closed. You’d have to make the request again.


In this regard, the response is different from a file object, because with a file object, you can read it multiple times by using the .seek() method, which HTTPResponse doesn’t support. Even after closing a response, you can still access the headers and other metadata, though.






Exploring Text, Octets, and Bits


In most of the examples so far, you read the response body from HTTPResponse, displayed the resulting data immediately, and noted that it was displayed as a bytes object. This is because text information in computers isn’t stored or transmitted as letters, but as bytes!





  



                       
  






A raw HTTP message sent over the wire is broken up into a sequence of bytes, sometimes referred to as octets. Bytes are 8-bit chunks. For example, 01010101 is a byte. To learn more about binary, bits, and bytes, check out Bitwise Operators in Python.


So how do you represent letters with bytes? A byte has 256 potential combinations, and you can assign a letter to each combination. You can assign 00000001 to A, 00000010 to B, and so on. ASCII character encoding, which is quite common, uses this type of system to encode 128 characters, which is enough for a language like English. This is particularly convenient because just one byte can represent all the characters, with space to spare.


All the standard English characters, including capitals, punctuation, and numerals, fit within ASCII. On the other hand, Japanese is thought to have around fifty thousand logographic characters, so 128 characters won’t cut it! Even the 256 characters that are theoretically available within one byte wouldn’t be nearly enough for Japanese. So, to accomodate all the world’s languages there are many different systems to encode characters.


Even though there are many systems, one thing you can rely on is the fact that they’ll always be broken up into bytes. Most servers, if they can’t resolve the MIME type and character encoding, default to application/octet-stream, which literally means a stream of bytes. Then whoever receives the message can work out the character encoding.






Dealing With Character Encodings


Problems often arise because, as you may have guessed, there are many, many different potential character encodings. The dominant character encoding today is UTF-8, which is an implementation of Unicode. Luckily, ninety-eight percent of web pages today are encoded in UTF-8!


UTF-8 is dominant because it can efficiently handle a mind-boggling number of characters. It handles all the 1,112,064 potential characters defined by Unicode, encompassing Chinese, Japanese, Arabic (with right-to-left scripts), Russian, and many more character sets, including emojis!


UTF-8 remains efficient because it uses a variable number of bytes to encode characters, which means that for many characters, it only requires one byte, while for others it can require up to four bytes.


While UTF-8 is dominant, and you usually won’t go wrong with assuming UTF-8 encodings, you’ll still run into different encodings all the time. The good news is that you don’t need to be an expert on encodings to handle them when using urllib.request.






Going From Bytes to Strings


When you use urllib.request.urlopen(), the body of the response is a bytes object. The first thing you may want to do is to convert the bytes object to a string. Perhaps you want to do some web scraping. To do this, you need to decode the bytes. To decode the bytes with Python, all you need to find out is the character encoding used. Encoding, especially when referring to character encoding, is often referred to as a character set.


As mentioned, ninety-eight percent of the time, you’ll probably be safe defaulting to UTF-8:





  



                       
  








>>>


>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> decoded_body = body.decode("utf-8")
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>





In this example, you take the bytes object returned from response.read() and decode it with the bytes object’s .decode() method, passing in utf-8 as an argument. When you print decoded_body, you can see that it’s now a string.


That said, leaving it up to chance is rarely a good strategy. Fortunately, headers are a great place to get character set information:




>>>


>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'utf-8'
>>> decoded_body = body.decode(character_set)
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>





In this example, you call .get_content_charset() on the .headers object of response and use that to decode. This is a convenience method that parses the Content-Type header so that you can painlessly decode bytes into text.






Going From Bytes to File


If you want to decode bytes into text, now you’re good to go. But what if you want to write the body of a response into a file? Well, you have two options:


    

  1. Write the bytes directly to the file
    2. 

  2. Decode the bytes into a Python string, and then encode the string back into a file
    3. 



The first method is the most straightforward, but the second method allows you to change the encoding if you want to. To learn about file manipulation in more detail, take a look at Real Python’s Reading and Writing Files in Python (Guide).


To write the bytes directly to a file without having to decode, you’ll need the built-in open() function, and you’ll need to ensure that you use write binary mode:




>>>


>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> with open("example.html", mode="wb") as html_file:
...     html_file.write(body)
...
1256





Using open() in wb mode bypasses the need to decode or encode and dumps the bytes of the HTTP message body into the example.html file. The number that’s output after the writing operation indicates the number of bytes that have been written. That’s it! You’ve written the bytes directly to a file without encoding or decoding anything.





  



                       
  






Now say you have a URL that doesn’t use UTF-8, but you want to write the contents to a file with UTF-8. For this, you’d first decode the bytes into a string and then encode the string into a file, specifying the character encoding.


Google’s home page seems to use different encodings depending on your location. In much of Europe and the US, it uses the ISO-8859-1 encoding:




>>>


>>> from urllib.request import urlopen
>>> with urlopen("https://www.google.com") as response:
...     body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'ISO-8859-1'
>>> content = body.decode(character_set)
>>> with open("google.html", encoding="utf-8", mode="w") as file:
...     file.write(content)
...
14066





In this code, you got the response character set and used it to decode the bytes object into a string. Then you wrote the string to a file, encoding it using UTF-8.


Once you’ve written to a file, you should be able to open the resulting file in your browser or text editor. Most modern text processors can detect the character encoding automatically.


If there are encoding errors and you’re using Python to read a file, then you’ll likely get an error:




>>>


>>> with open("encoding-error.html", mode="r", encoding="utf-8") as file:
...     lines = file.readlines()
...
UnicodeDecodeError:
    'utf-8' codec can't decode byte





Python explicitly stops the process and raises an exception, but in a program that displays text, such as the browser where you’re viewing this page, you may find the infamous replacement characters:


Unicode Replacement Character
A Replacement Character
The black rhombus with a white question mark (�), the square (□), and the rectangle (▯) are often used as replacements for characters which couldn’t be decoded.


Sometimes, decoding seems to work but results in unintelligible sequences, such as æ–‡å—化け., which also suggests the wrong character set was used. In Japan, they even have a word for text that’s garbled due to character encoding issues, Mojibake, because these issues plagued them at the start of the Internet age.





  



                       
  






With that, you should now be equipped to write files with the raw bytes returned from urlopen(). In the next section, you’ll learn how to parse bytes into a Python dictionary with the json module.






Going From Bytes to Dictionary


For application/json responses, you’ll often find that they don’t include any encoding information:




>>>


>>> from urllib.request import urlopen
>>> with urlopen("https://httpbin.org/json") as response:
...     body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> print(character_set)
None





In this example, you use the json endpoint of httpbin, a service that allows you to experiment with different types of requests and responses. The json endpoint simulates a typical API that returns JSON data. Note that the .get_content_charset() method returns nothing in its response.


Even though there’s no character encoding information, all is not lost. According to RFC 4627, the default encoding of UTF-8 is an absolute requirement of the application/json specification. That’s not to say that every single server plays by the rules, but generally, you can assume that if JSON is being transmitted, it’ll almost always be encoded using UTF-8.


Fortunately, json.loads() decodes byte objects under the hood and even has some leeway in terms of different encodings that it can deal with. So, json.loads() should be able to cope with most bytes objects that you throw at it, as long as they’re valid JSON:




>>>


>>> import json
>>> json.loads(body)
{'slideshow': {'author': 'Yours Truly', 'date': 'date of publication', 'slides'
: [{'title': 'Wake up to WonderWidgets!', 'type': 'all'}, {'items': ['Why <em>W
onderWidgets</em> are great', 'Who <em>buys</em> WonderWidgets'], 'title': 'Ove
rview', 'type': 'all'}], 'title': 'Sample Slide Show'}}





As you can see, the json module handles the decoding automatically and produces a Python dictionary. Almost all APIs return key-value information as JSON, although you might run into some older APIs that work with XML. For that, you might want to look into the Roadmap to XML Parsers in Python.


With that, you should know enough about bytes and encodings to be dangerous! In the next section, you’ll learn how to troubleshoot and fix a couple of common errors that you might run into when using urllib.request.








Common urllib.request Troubles


There are many kinds of issues you can run into on the world wild web, whether you’re using urllib.request or not. In this section, you’ll learn how to deal with a couple of the most common errors when getting started out: 403 errors and TLS/SSL certificate errors. Before looking at these specific errors, though, you’ll first learn how to implement error handling more generally when using urllib.request.





  



                       
  








Implementing Error Handling


Before you turn your attention to specific errors, boosting your code’s ability to gracefully deal with assorted errors will pay off. Web development is plagued with errors, and you can invest a lot of time in handling errors sensibly. Here, you’ll learn to handle HTTP, URL, and timeout errors when using urllib.request.


HTTP status codes accompany every response in the status line. If you can read a status code in the response, then the request reached its target. While this is good, you can only consider the request a complete success if the response code starts with a 2. For example, 200 and 201 represent successful requests. If the status code is 404 or 500, for example, something went wrong, and urllib.request will raise an HTTPError.


Sometimes mistakes happen, and the URL provided isn’t correct, or a connection can’t be made for another reason. In these cases, urllib.request will raise a URLError.


Finally, sometimes servers just don’t respond. Maybe your network connection is slow, the server is down, or the server is programmed to ignore specific requests. To deal with this, you can pass a timeout argument to urlopen() to raise a TimeoutError after a certain amount of time.


The first step in handling these exceptions is to catch them. You can catch errors produced within urlopen() with a try … except block, making use of the HTTPError, URLError, and TimeoutError classes:




# request.py

from urllib.error import HTTPError, URLError
from urllib.request import urlopen

def make_request(url):
    try:
        with urlopen(url, timeout=10) as response:
            print(response.status)
            return response.read(), response
    except HTTPError as error:
        print(error.status, error.reason)
    except URLError as error:
        print(error.reason)
    except TimeoutError:
        print("Request timed out")





The function make_request() takes a URL string as an argument, tries to get a response from that URL with urllib.request, and catches the HTTPError object that’s raised if an error occurs. If the URL is bad, it’ll catch a URLError. If it goes through without any errors, it’ll just print the status and return a tuple containing the body and the response. The response will close after return.


The function also calls urlopen() with a timeout argument, which will cause a TimeoutError to be raised after the seconds specified. Ten seconds is generally a good amount of time to wait for a response, though as always, much depends on the server that you need to make the request to.


Now you’re set up to gracefully handle a variety of errors, including but not limited to the errors that you’ll cover next.






Dealing With 403 Errors


You’ll now use the make_request() function to make some requests to httpstat.us, which is a mock server used for testing. This mock server will return responses that have the status code you request. If you make a request to https://httpstat.us/200, for example, you should expect a 200 response.


APIs like httpstat.us are used to ensure that your application can handle all the different status codes it might encounter. httpbin also has this functionality, but httpstat.us has a more comprehensive selection of status codes. It even has the infamous and semi-official 418 status code that returns the message I’m a teapot!





  



                       
  






To interact with the make_request() function that you wrote in the previous section, run the script in interactive mode:





  



                       
  






With the -i flag, this command will run the script in interactive mode. This means that it’ll execute the script and then open the Python REPL afterward, so you can now call the function that you just defined:




>>>


>>> make_request("https://httpstat.us/200")
200
(b'200 OK', <http.client.HTTPResponse object at 0x0000023D612660B0>)
>>> make_request("https://httpstat.us/403")
403 Forbidden





Here you tried the 200 and 403 endpoints of httpstat.us. The 200 endpoint goes through as anticipated and returns the body of the response and the response object. The 403 endpoint just printed the error message and didn’t return anything, also as expected.


The 403 status means that the server understood the request but won’t fulfill it. This is a common error that you can run into, especially while web scraping. In many cases, you can solve it by passing a User-Agent header.


One of the primary ways that servers identify who or what is making the request is by examining the User-Agent header. The raw default request sent by urllib.request is the following:




GET https://httpstat.us/403 HTTP/1.1
Accept-Encoding: identity
Host: httpstat.us
User-Agent: Python-urllib/3.10
Connection: close





Notice that User-Agent is listed as Python-urllib/3.10. You may find that some sites will try to block web scrapers, and this User-Agent is a dead giveaway. With that said, you can set your own User-Agent with urllib.request, though you’ll need to modify your function a little:




 # request.py

 from urllib.error import HTTPError, URLError
-from urllib.request import urlopen
+from urllib.request import urlopen, Request

-def make_request(url):
+def make_request(url, headers=None):
+    request = Request(url, headers=headers or {})
     try:
-        with urlopen(url, timeout=10) as response:
+        with urlopen(request, timeout=10) as response:
             print(response.status)
             return response.read(), response
     except HTTPError as error:
         print(error.status, error.reason)
     except URLError as error:
         print(error.reason)
     except TimeoutError:
         print("Request timed out")





To customize the headers that you send out with your request, you first have to instantiate a Request object with the URL. Additionally, you can pass in a keyword argument of headers, which accepts a standard dictionary representing any headers you wish to include. So, instead of passing the URL string directly into urlopen(), you pass this Request object which has been instantiated with the URL and headers.


To use this revamped function, restart the interactive session, then call make_request() with a dictionary representing the headers as an argument:




>>>


>>> body, response = make_request(
...     "https://www.httpbin.org/user-agent",
...     {"User-Agent": "Real Python"}
... )
200
>>> body
b'{n  "user-agent": "Real Python"n}n'





In this example, you make a request to httpbin. Here you use the user-agent endpoint to return the request’s User-Agent value. Because you made the request with a custom user agent of Real Python, this is what gets returned.


Some servers are strict, though, and will only accept requests from specific browsers. Luckily, it’s possible to find standard User-Agent strings on the web, including through a user agent database. They’re just strings, so all you need to do is copy the user agent string of the browser that you want to impersonate and use it as the value of the User-Agent header.






Fixing the SSL CERTIFICATE_VERIFY_FAILED Error


Another common error is due to Python not being able to access the required security certificate. To simulate this error, you can use some mock sites that have known bad SSL certificates, provided by badssl.com. You can make a request to one of them, such as superfish.badssl.com, and experience the error firsthand:




>>>


>>> from urllib.request import urlopen
>>> urlopen("https://superfish.badssl.com/")
Traceback (most recent call last):
  (...)
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  (...)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>





Here, making a request to an address with a known bad SSL certificate will result in CERTIFICATE_VERIFY_FAILED which is a type of URLError.


SSL stands for Secure Sockets Layer. This is something of a misnomer because SSL was deprecated in favor of TLS, Transport Layer Security. Sometimes old terminology just sticks! It’s a way to encrypt network traffic so that a hypothetical listener can’t eavesdrop on the information transmitted over the wire.


These days, most website addresses are preceded not by http:// but by https://, with the s standing for secure. HTTPS connections must be encrypted through the TLS. urllib.request can handle both HTTP and HTTPS connections.


The details of HTTPS are far beyond the scope of this tutorial, but you can think of an HTTPS connection as involving two stages, the handshake and the transfer of information. The handshake ensures that the connection is secure. For more information about Python and HTTPS, check out Exploring HTTPS With Python.


To establish that a particular server is secure, programs that make requests rely on a store of trusted certificates. The server’s certificate is verified during the handshake stage. Python uses the operating system’s store of certificates. If Python can’t find the system’s store of certificates, or if the store is out of date, then you’ll run into this error.


Sometimes the store of certificates that Python can access is out of date, or Python can’t reach it, for whatever reason. This is frustrating because you can sometimes visit the URL from your browser, which thinks that it’s secure, yet urllib.request still raises this error.


You may be tempted to opt out of verifying the certificate, but this will render your connection insecure and is definitely not recommended:




>>>


>>> import ssl
>>> from urllib.request import urlopen
>>> unverified_context = ssl._create_unverified_context()
>>> urlopen("https://superfish.badssl.com/", context=unverified_context)
<http.client.HTTPResponse object at 0x00000209CBE8F220>





Here you import the ssl module, which allows you to create an unverified context. You can then pass this context to urlopen() and visit a known bad SSL certificate. The connection successfully goes through because the SSL certificate isn’t checked.


Before resorting to these desperate measures, try updating your OS or updating your Python version. If that fails, then you can take a page from the requests library and install certifi:


    

  • 
    Windows
  
    • 

  • 
    Linux + macOS
  
    • 







PS> python -m venv venv
PS> .venvScriptsactivate
(venv) PS> python -m pip install certifi







$ python3 -m venv venv
$ source venv/bin/activate.sh
(venv) $ python3 -m pip install certifi







certifi is a collection of certificates that you can use instead of your system’s collection. You do this by creating an SSL context with the certifi bundle of certificates instead of the OS’s bundle:




>>>


>>> import ssl
>>> from urllib.request import urlopen
>>> import certifi
>>> certifi_context = ssl.create_default_context(cafile=certifi.where())
>>> urlopen("https://sha384.badssl.com/", context=certifi_context)
<http.client.HTTPResponse object at 0x000001C7407C3490>





In this example, you used certifi to act as your SSL certificate store, and you used it to successfully connect to a site with a known good SSL certificate. Note that instead of ._create_unverified_context(), you use .create_default_context().


This way, you can stay secure without too much trouble! In the next section, you’ll be dipping your toes into the world of authentication.








Authenticated Requests


Authentication is a vast subject, and if you’re dealing with authentication much more complicated than what’s covered here, this might be a good jumping-off point into the requests package.


In this tutorial, you’ll only cover one authentication method, which serves as an example of the type of adjustments that you have to make to authenticate your requests. urllib.request does have a lot of other functionality that helps with authentication, but that won’t be covered in this tutorial.


One of the most common authentication tools is the bearer token, specified by RFC 6750. It’s often used as part of OAuth, but can also be used in isolation. It’s also most common to see as a header, which you can use with your current make_request() function:




>>>


>>> token = "abcdefghijklmnopqrstuvwxyz"
>>> headers = {
...     "Authorization": f"Bearer {token}"
... }
>>> make_request("https://httpbin.org/bearer", headers)
200
(b'{n  "authenticated": true, n  "token": "abcdefghijklmnopqrstuvwxyz"n}n',
<http.client.HTTPResponse object at 0x0000023D612642E0>)





In this example, you make a request to the httpbin /bearer endpoint, which simulates bearer authentication. It’ll accept any string as a token. It only requires the proper format specified by RFC 6750. The name has to be Authorization, or sometimes the lowercase authorization, and the value has to be Bearer, with a single space between that and the token.


Congratulations, you’ve successfully authenticated, using a bearer token!


Another form of authentication is called Basic Access Authentication, which is a very simple method of authentication, only slightly better than sending a username and password in a header. It’s very insecure!


One of the most common protocols in use today is OAuth (Open Authorization). If you’ve ever used Google, GitHub, or Facebook to sign into another website, then you’ve used OAuth. The OAuth flow generally involves a few requests between the service that you want to interact with and an identity server, resulting in a short-lived bearer token. This bearer token can then be used for a period of time with bearer authentication.


Much of authentication comes down to understanding the specific protocol that the target server uses and reading the documentation closely to get it working.






POST Requests With urllib.request


You’ve made a lot of GET requests, but sometimes you want to send information. That’s where POST requests come in. To make POST requests with urllib.request, you don’t have to explicitly change the method. You can just pass a data object to a new Request object or directly to urlopen(). The data object must be in a special format, though. You’ll adapt your make_request() function slightly to support POST requests by adding the data parameter:




 # request.py

 from urllib.error import HTTPError, URLError
 from urllib.request import urlopen, Request

-def make_request(url, headers=None):
+def make_request(url, headers=None, data=None):

-    request = Request(url, headers=headers or {})
+    request = Request(url, headers=headers or {}, data=data)
     try:
         with urlopen(request, timeout=10) as response:
             print(response.status)
             return response.read(), response
     except HTTPError as error:
         print(error.status, error.reason)
     except URLError as error:
         print(error.reason)
     except TimeoutError:
         print("Request timed out")





Here you just modified the function to accept a data argument with a default value of None, and you passed that right into the Request instantiation. That’s not all that needs to be done, though. You can use one of two different formats to execute a POST request:


    

  1. Form Data: application/x-www-form-urlencoded
    2. 

  2. JSON: application/json
    3. 



The first format is the oldest format for POST requests and involves encoding the data with percent encoding, also known as URL encoding. You may have noticed key-value pairs URL encoded as a query string. Keys are separated from values with an equal sign (=), key-value pairs are separated with an ampersand (&), and spaces are generally suppressed but can be replaced with a plus sign (+).


If you’re starting off with a Python dictionary, to use the form data format with your make_request() function, you’ll need to encode twice:


    

  1. Once to URL encode the dictionary
    2. 

  2. Then again to encode the resulting string into bytes
    3. 



For the first stage of URL encoding, you’ll use another urllib module, urllib.parse. Remember to start your script in interactive mode so that you can use the make_request() function and play with it on the REPL:




>>>


>>> from urllib.parse import urlencode
>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> url_encoded_data = urlencode(post_dict)
>>> url_encoded_data
'Title=Hello+World&Name=Real+Python'
>>> post_data = url_encoded_data.encode("utf-8")
>>> body, response = make_request(
...     "https://httpbin.org/anything", data=post_data
... )
200
>>> print(body.decode("utf-8"))
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "Name": "Real Python",
    "Title": "Hello World"
  },
  "headers": {
    "Accept-Encoding": "identity",
    "Content-Length": "34",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/3.10",
    "X-Amzn-Trace-Id": "Root=1-61f25a81-03d2d4377f0abae95ff34096"
  },
  "json": null,
  "method": "POST",
  "origin": "86.159.145.119",
  "url": "https://httpbin.org/anything"
}





In this example, you:


    

  1. Import urlencode() from the urllib.parse module
    2. 

  2. Initialize your POST data, starting with a dictionary
    3. 

  3. Use the urlencode() function to encode the dictionary
    4. 

  4. Encode the resulting string into bytes using UTF-8 encoding
    5. 

  5. Make a request to the anything endpoint of httpbin.org
    6. 

  6. Print the UTF-8 decoded response body
    7. 



UTF-8 encoding is part of the specification for the application/x-www-form-urlencoded type. UTF-8 is used preemptively to decode the body because you already know that httpbin.org reliably uses UTF-8.


The anything endpoint from httpbin acts as a sort of echo, returning all the information it received so that you can inspect the details of the request you made. In this case, you can confirm that method is indeed POST, and you can see that the data you sent is listed under form.


To make the same request with JSON, you’ll turn a Python dictionary into a JSON string with json.dumps(), encode it with UTF-8, pass it as the data argument, and finally add a special header to indicate that the data type is JSON:




>>>


>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> import json
>>> json_string = json.dumps(post_dict)
>>> json_string
'{"Title": "Hello World", "Name": "Real Python"}'
>>> post_data = json_string.encode("utf-8")
>>> body, response = make_request(
...     "https://httpbin.org/anything",
...     data=post_data,
...     headers={"Content-Type": "application/json"},
... )
200
>>> print(body.decode("utf-8"))
{
  "args": {},
  "data": "{"Title": "Hello World", "Name": "Real Python"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "identity",
    "Content-Length": "47",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/3.10",
    "X-Amzn-Trace-Id": "Root=1-61f25a81-3e35d1c219c6b5944e2d8a52"
  },
  "json": {
    "Name": "Real Python",
    "Title": "Hello World"
  },
  "method": "POST",
  "origin": "86.159.145.119",
  "url": "https://httpbin.org/anything"
}





To serialize the dictionary this time around, you use json.dumps() instead of urlencode(). You also explicitly add the Content-Type header with a value of application/json. With this information, the httpbin server can deserialize the JSON on the receiving end. In its response, you can see the data listed under the json key.


With that, you can now start making POST requests. This tutorial won’t go into more detail about the other request methods, such as PUT. Suffice to say that you can also explicitly set the method by passing a method keyword argument to the instantiation of the Request object.






The Request Package Ecosystem


To round things out, this last section of the tutorial is dedicated to clarifying the package ecosystem around HTTP requests with Python. Because there are many packages, with no clear standard, it can be confusing. That said, there are use cases for each package, which just means more choice for you!




What Are urllib2 and urllib3?


To answer this question, you need to go back to early Python, all the way back to version 1.2, when the original urllib was introduced. Around version 1.6, a revamped urllib2 was added, which lived alongside the original urllib. When Python 3 came along, the original urllib was deprecated, and urllib2 dropped the 2, taking on the original urllib name. It also split into parts:


    

  • urllib.error
    • 

  • urllib.parse
    • 

  • urllib.request
    • 

  • urllib.response
    • 

  • urllib.robotparser
    • 



So what about urllib3? That’s a third-party library developed while urllib2 was still around. It’s not related to the standard library because it’s an independently maintained library. Interestingly, the requests library actually uses urllib3 under the hood, and so does pip!






When Should I Use requests Over urllib.request?


The main answer is ease of use and security. urllib.request is considered a low-level library, which exposes a lot of the detail about the workings of HTTP requests. The Python documentation for urllib.request makes no bones about recommending requests as a higher-level HTTP client interface.


If you interact with many different REST APIs, day in and day out, then requests is highly recommended. The requests library bills itself as “built for human beings” and has successfully created an intuitive, secure, and straightforward API around HTTP. It’s usually considered the go-to library! If you want to know more about the requests library, check out the Real Python guide to requests.


An example of how requests makes things easier is when it comes to character encoding. You’ll remember that with urllib.request, you have to be aware of encodings and take a few steps to ensure an error-free experience. The requests package abstracts that away and will resolve the encoding by using chardet, a universal character encoding detector, just in case there’s any funny business.


If your goal is to learn more about standard Python and the details of how it deals with HTTP requests, then urllib.request is a great way to get into that. You could even go further and use the very low-level http modules. On the other hand, you may just want to keep dependencies to a minimum, which urllib.request is more than capable of.






Why Is requests Not Part of the Standard Library?


Maybe you’re wondering why requests isn’t part of core Python by this point.


This is a complex issue, and there’s no hard and fast answer to it. There are many speculations as to why, but two reasons seem to stand out:


    

  1. requests has other third-party dependencies that would need to be integrated too.
    2. 

  2. requests needs to stay agile and can do this better outside the standard library.
    3. 



The requests library has third-party dependencies. Integrating requests into the standard library would mean also integrating chardet, certifi, and urllib3, among others. The alternative would be to fundamentally change requests to use only Python’s existing standard library. This is no trivial task!


Integrating requests would also mean that the existing team that develops this library would have to relinquish total control over the design and implementation, giving way to the PEP decision-making process.


HTTP specifications and recommendations change all the time, and a high-level library has to be agile enough to keep up. If there’s a security exploit to be patched, or a new workflow to add, the requests team can build and release far more quickly than they could as part of the Python release process. There have supposedly been times when they’ve released a security fix twelve hours after a vulnerability was discovered!


For an interesting overview of these issues and more, check out Adding Requests to The Standard Library, which summarizes a discussion at the Python Language Summit with Kenneth Reitz, the creator and maintainer of Requests.


Because this agility is so necessary to requests and its underlying urllib3, the paradoxical statement that requests is too important for the standard library is often used. This is because so much of the Python community depends on requests and its agility that integrating it into core Python would probably damage it and the Python community.


On the GitHub repository issues board for requests, an issue was posted, asking for the inclusion of requests in the standard library. The developers of requests and urllib3 chimed in, mainly saying they would likely lose interest in maintaining it themselves. Some even said they would fork the repositories and continue developing them for their own use cases.


With that said, note that the requests library GitHub repository is hosted under the Python Software Foundation’s account. Just because something isn’t part of the Python standard library doesn’t mean that it’s not an integral part of the ecosystem!


It seems that the current situation works for both the Python core team and the maintainers of requests. While it may be slightly confusing for newcomers, the existing structure gives the most stable experience for HTTP requests.


It’s also important to note that HTTP requests are inherently complex. urllib.request doesn’t try to sugarcoat that too much. It exposes a lot of the inner workings of HTTP requests, which is why it’s billed as a low-level module. Your choice of requests versus urllib.request really depends on your particular use case, security concerns, and preference.








Conclusion


You’re now equipped to use urllib.request to make HTTP requests. Now you can use this built-in module in your projects, keeping them dependency-free for longer. You’ve also gained the in-depth understanding of HTTP that comes from using a lower-level module, such as urllib.request.


In this tutorial, you’ve:


    

  • Learned how to make basic HTTP requests with urllib.request
    • 

  • Explored the nuts and bolts of an HTTP message and studied how it’s represented by urllib.request
    • 

  • Figured out how to deal with character encodings of HTTP messages
    • 

  • Explored some common errors when using urllib.request and learned how to resolve them
    • 

  • Dipped your toes into the world of authenticated requests with urllib.request
    • 

  • Understood why both urllib and the requests library exist and when to use one or the other
    • 



You’re now in a position to make basic HTTP requests with urllib.request, and you also have the tools to dive deeper into low-level HTTP terrain with the standard library. Finally, you can choose whether to use requests or urllib.request, depending on what you want or need. Have fun exploring the Web!






Источник






Синтаксис:




import requests

# создается в результате запроса к серверу
Response = requests.get(...)





Параметры:


    

  • нет.
    • 



Описание:


Объект requests.Response модуля requests содержит всю информацию ответа сервера на HTTP-запрос requests.get(), requests.post() и т.д.




Объект ответа сервера requests.Response генерируется после того, как библиотека requests получают ответ от сервера. Объект ответа Response содержит всю информацию, 


возвращаемую сервером


, а также 


объект запроса


, который создали изначально.




Атрибуты и методы объекта Response.


    

  • Response.apparent_encoding возвращает кодировку, угаданную chardet,
    • 

  • Response.close() освобождает соединение с пулом,
    • 

  • Response.content возвращает контент в байтах,
    • 

  • Response.cookies возвращает cookies, установленные сервером,
    • 

  • Response.elapsed возвращает время, потраченное на запрос,
    • 

  • Response.encoding устанавливает кодировку, для декодирования,
    • 

  • Response.headers возвращает заголовки сервера,
    • 

  • Response.history возвращает историю перенаправлений,
    • 

  • Response.is_permanent_redirect определение постоянных редиректов,
    • 

  • Response.is_redirect есть ли редирект,
    • 

  • Response.iter_content() перебирает данные ответа кусками,
    • 

  • Response.iter_lines() перебирает данные ответа, по одной строке,
    • 

  • Response.json() возвращает ответ в виде JSON,
    • 

  • Response.links возвращает ссылки заголовка ответа,
    • 

  • Response.next возвращает объект PreparedRequest,
    • 

  • Response.ok True, если status_code меньше 400,
    • 

  • Response.raise_for_status() вызывает исключение HTTPError,
    • 

  • Response.raw возвращает ответа в виде файлового объекта,
    • 

  • Response.reason возвращает текстовое представление ответа,
    • 

  • Response.request возвращает объект PreparedRequest запроса,
    • 

  • Response.status_code возвращает код ответа сервера,
    • 

  • Response.text возвращает контент ответа сервера в юникоде,
    • 

  • Response.url возвращает URL-адрес, после перенаправлений.
    • 

  • Пример работы с объектом ответа сервера `Response.
    • 





Response.apparent_encoding:


Атрибут Response.apparent_encoding возвращает кодировку, определенную сторонним модулем chardet.


Response.close():


Метод Response.close() освобождает соединение с пулом. Как только этот метод был вызван, базовый необработанный объект больше не будет доступен.


Примечание: обычно не нужно вызывать явно.


Response.content:


Атрибут Response.content возвращает содержание ответа сервера, представленное в байтах.


Response.cookies = None:


Атрибут Response.cookies возвращает хранилище CookieJar файлов cookie, которые сервер отправил обратно. 


Другими словами возвращает cookies, установленные сервером.


Response.elapsed = None:


Атрибут Response.elapsed возвращает время, прошедшее между отправкой запроса и получением ответа (в виде timedelta). 


Это свойство специально измеряет время, затраченное между отправкой первого байта запроса и завершением анализа заголовков. Поэтому на него не влияет потребление содержимого ответа или значения  ключевого аргумента stream.


Response.encoding = None:


Атрибут Response.encoding возвращает/устанавливает кодировку для декодирования контента при доступе к атрибуту Response.text.


Response.headers = None:


Атрибут Response.headers возвращает словарь без учета регистра, с заголовками сервера, которые он вернул во время ответа. 


Например, заголовки [‘content-encoding’] вернут значение заголовка ответа Content-Encoding.


Response.history = None:


Атрибут Response.history возвращает список объектов ответа сервера из истории запроса. Здесь окажутся все перенаправленные ответы. 


Список сортируется от самого старого до самого последнего запроса.


Response.is_permanent_redirect:


Атрибут Response.is_permanent_redirect возвращает True, если в этом ответе одна из постоянных версий перенаправления.


Response.is_redirect:


Атрибут Response.is_redirect возвращает True если этот ответ является хорошо сформированным HTTP-перенаправлением, которое могло бы быть обработано автоматически (Session.resolve_redirects).


Response.iter_content(chunk_size=1, decode_unicode=False):


Метод Response.iter_content() перебирает данные ответа. Когда в запросе установлен stream=True, то это позволяет избежать одновременного чтения содержимого в память для больших ответов. 


Размер блока chunk_size — это количество байтов, которые он должен считывать в память. Это не обязательно длина каждого возвращаемого элемента, т.к. может иметь место декодирование.


Аргумент chunk_size должен иметь тип int или None. Значение None будет функционировать по-разному в зависимости от значения stream. Если stream=True, то будет считывать данные по мере их поступления в любом размере полученных фрагментов. Если stream=False, то данные возвращаются как один фрагмент.


Если аргумент decode_unicode=True, то содержимое будет декодировано с использованием наилучшей доступной кодировки на основе ответа.


Response.iter_lines(chunk_size=512, decode_unicode=False, delimiter=None):


Метод Response.iter_lines() перебирает данные ответа, по одной строке за раз. Когда в запросе установлен stream=True, то это позволяет избежать одновременного чтения содержимого в память для больших ответов.


Обратите внимание


, что этот метод не является безопасным для повторного входа.




Response.json(**kwargs):


Метод Response.json() возвращает закодированное в json содержимое ответа, если таковое имеется.


Аргумент **kwargs это необязательные аргументы, которые принимает json.loads.


    

  • simplejson.JSONDecodeError — если тело ответа не содержит действительного json и установлен simplejson.
    • 

  • json.JSONDecodeError — если тело ответа не содержит допустимого json и simplejson не установлен.
    • 



Response.links:


Атрибут Response.links возвращает проанализированные ссылки заголовка ответа, если таковые имеются.


Response.next:


Response.ok:


Атрибут Response.ok возвращает True, если status_code меньше 400, и False, если нет.


Этот атрибут проверяет, находится ли код состояния ответа в диапазоне от 400 до 600, чтобы проверить, была ли ошибка клиента или ошибка сервера. Если код состояния находится в диапазоне от 200 до 400, то этот атрибут вернет значение True. 


Это не проверка кода ответа 200 OK.


Response.raise_for_status():


Метод Response.raise_for_status() вызывает исключение HTTPError, если таковой произошел.


Response.raw = None:


Атрибут Response.raw возвращает представление ответа в виде файлового объекта (для расширенного использования). 


Использование Response.raw требует, чтобы в запросе был установлен stream=True. Это требование не распространяется на внутреннее использование запросов.


Response.reason = None:


Атрибут Response.reason возвращает текстовое представление ответа HTTP-статуса, например 'Not Found' или 'OK'.


Response.request = None:


Response.status_code = None:


Атрибут Response.status_code целочисленный код ответа HTTP-статуса, например 404 или 200.


Response.text:


Атрибут Response.text возвращает cодержание/контент ответа сервера в юникоде.


Если Response.encoding=None, то кодировка будет угадана с помощью модуля chardet.


Кодировка содержимого ответа определяется исключительно на основе HTTP-заголовков, следующих за RFC 2616. Если вы точно знаете кодировку сайта, а Response.text возвращает «крокозябры«, то тогда сначала следует установить Response.encoding в нужную кодировку, перед тем как вызвать Response.text.


Response.url = None:


Атрибут Response.url окончательный URL-адреса ответа, после перенаправлений, которые делал сервер.


Пример работы с объектом ответа сервера Response:




import requests
resp = requests.get('https://httpbin.org/get', 
                     headers={'user-agent': 'my-agent-0.0.1'}, 
                     cookies={'one': 'true'})
if resp.ok:
    print('Заголовки ответа сервера:')
    print(resp.headers)
    print('ncookies, установленные сервером:')
    print(dict(resp.cookies))
    print('nКонтент запрашиваемой страницы:')
    print(resp.text)
    print('nДоступ к параметрам нашего запроса `resp.request`:')
    print('- заголовки, которые отправил запрос:')
    print(resp.request.headers)
    print('- метод нашего запроса:')
    print(resp.request.method)

# Заголовки ответа сервера:
# {'Date': 'Wed, 31 Mar 2021 09:32:48 GMT', 'Content-Type': 'application/json', 
# 'Content-Length': '327', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 
# 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
# 
# cookies, установленные сервером:
# {}
# 
# Контент запрашиваемой страницы:
# {
#   "args": {}, 
#   "headers": {
#     "Accept": "*/*", 
#     "Accept-Encoding": "gzip, deflate", 
#     "Cookie": "one=true", 
#     "Host": "httpbin.org", 
#     "User-Agent": "my-agent-0.0.1", 
#     "X-Amzn-Trace-Id": "Root=1-606441c0-674418047cacc3e2617e6449"
#   }, 
#   "origin": "xxx.xxx.xxx.xxx", 
#   "url": "https://httpbin.org/get"
# }
# 
# 
# Доступ к параметрам нашего запроса `resp.request`:
# - заголовки, которые отправил запрос:
# {'user-agent': 'my-agent-0.0.1', 'Accept-Encoding': 'gzip, deflate', 
# 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'one=true'}
# - метод нашего запроса:
# GET









Источник






The clear, simple syntax of Python makes it an ideal language to interact with REST APIs, and in typical Python fashion, there’s a library made specifically to provide that functionality: Requests. Python Requests is a powerful tool that provides the simple elegance of Python to make HTTP requests to any API in the world. At Nylas, we built our REST APIs for email, calendar, and contacts on Python, and we process over 500 million API requests a day, so naturally, we depend a ton on the Python Requests library.




In this guide, we’ll take a comprehensive look at making HTTP requests with Python Requests and learn how to use this functionality to integrate with REST APIs.


Contents:


    

  • The roles of HTTP, APIs, and REST
    • 

  • How to use Python Requests with REST APIs
    • 

  • How to authenticate to a REST API
    • 

  • How to handle HTTP errors with Python Requests
    • 

  • How to make robust API Requests 
    • 





The Roles of HTTP, APIs, and REST




An Application Programming Interface (API) is a web service that grants access to specific data and methods that other applications can access – and sometimes edit – via standard HTTP protocols, just like a website. This simplicity makes it easy to quickly integrate APIs into a wide variety of applications. REpresentational State Transfer (REST), is probably the most popular architectural style of APIs for web services. It consists of a set of guidelines designed to simplify client / server communication. REST APIs make data access much more straightforward and logical.


The Request


When you want to interact with data via a REST API, this is called a request. A request is made up of the following components:


Endpoint – The URL that delineates what data you are interacting with. Similar to how a web page URL is tied to a specific page, an endpoint URL is tied to a specific resource within an API.


Method – Specifies how you’re interacting with the resource located at the provided endpoint. REST APIs can provide methods to enable full Create, Read, Update, and Delete (CRUD) functionality. Here are common methods most REST APIs provide:


    

  • GET – Retrieve data
    • 

  • PUT – Replace data
    • 

  • POST – Create data
    • 

  • DELETE – Delete data
    • 



Data – If you’re using a method that involves changing data in a REST API, you’ll need to include a data payload with the request that includes all data that will be created or modified.


Headers – Contain any metadata that needs to be included with the request, such as authentication tokens, the content type that should be returned, and any caching policies.


The Response


When you perform a request, you’ll get a response from the API. Just like in the request, it’ll have a response header and response data, if applicable. The response header consists of useful metadata about the response, while the response data returns what you actually requested. This can be any sort of data, as it’s really dependent on the API. The text is usually returned as JSON, but other markdown languages like XML are also possible. 


Let’s look at a simple example of a request and a response. In the terminal, we’ll use curl to make a GET request to the Open Notify API. This is a simple, yet nifty API that has information about astronauts that are currently in space:




curl -X GET "http://api.open-notify.org/astros.json"



You should see a response in JSON format that lists data about these astronauts, at the time of this article there are three people on a historic trip to the International Space Station:




{
  "number": 3,
  "message": "success",
  "people": [
    {
      "craft": "ISS",
      "name": "Chris Cassidy"
    }, 
    {
      "craft": "ISS",
      "name": "Anatoly Ivanishin"
    }, 
    {
      "craft": "ISS",
      "name": "Ivan Vagner"
    }
  ]
}





How to Use Python Requests with REST APIs




Now, let’s take a look at what it takes to integrate with a REST API using Python Requests. First, you’ll need to have the necessary software; make sure you have Python and pip installed on your machine. Then, head over to the command line and install the python requests module with pip:




pip install requests



Now you’re ready to start using Python Requests to interact with a REST API, make sure you import the Requests library into any scripts you want to use it in:






import requests



How Request Data With GET


The GET method is used to access data for a specific resource from a REST API; Python Requests includes a function to do exactly this.




import requests
response = requests.get("http://api.open-notify.org/astros.json")
print(response)
>>>> Response<200>


The response object contains all the data sent from the server in response to your GET request, including headers and the data payload. When this code example prints the response object to the console it simply returns the name of the object’s class and the status code the request returned (more on status codes later).


While this information might be useful, you’re most likely interested in the content of the request itself, which can be accessed in a few ways:




response.content() # Return the raw bytes of the data payload
response.text() # Return a string representation of the data payload
response.json() # This method is convenient when the API returns JSON


How to Use Query Parameters 


Queries can be used to filter the data that an API returns, and these are added as query parameters that are appended to the endpoint URL. With Python Requests, this is handled via the params argument, which accepts a dictionary object; let’s see what that looks like when we use the Open Notify API to GET an estimate for when the ISS will fly over a specified point:




query = {'lat':'45', 'lon':'180'}
response = requests.get('http://api.open-notify.org/iss-pass.json', params=query)
print(response.json())


The print command would return something that looks like this:




{
  'message': 'success',
  'request': {
    'altitude': 100,
    'datetime': 1590607799,
    'latitude': 45.0,
    'longitude': 180.0,
    'passes': 5
  },
  'response': [
    {'duration': 307, 'risetime': 1590632341},
    {'duration': 627, 'risetime': 1590637934},
    {'duration': 649, 'risetime': 1590643725},
    {'duration': 624, 'risetime': 1590649575},
    {'duration': 643, 'risetime': 1590655408}
  ]
}


How to Create and Modify Data With POST and PUT


In a similar manner as the query parameters, you can use the data argument to add the associated data for PUT and POST method requests.




# Create a new resource
response = requests.post('https://httpbin.org/post', data = {'key':'value'})

# Update an existing resource
requests.put('https://httpbin.org/put', data = {'key':'value'})






  Experience Nylas REST APIs Using Python  


Get a quick demonstration on how to make your first API call in minutes with Nylas using Python. 




How to Access REST Headers


You can also retrieve metadata from the response via headers. For example, to view the date of the response, just specify that with the `headers` property:




print(response.headers["date"]) 
>>>> 'Wed, 11 June 2020 19:32:24 GMT'


For open APIs, that covers the basics. However, many APIs can’t be used by just anyone. For those, let’s go over how to authenticate to REST APIs.



How to Authenticate to a REST API




So far you’ve seen how to interact with open REST APIs that don’t require any authorization. However, many REST APIs require you to authenticate to them before you can access specific endpoints, particularly if they deal with sensitive data. 


There are a few common authentication methods for REST APIs that can be handled with Python Requests. The simplest way is to pass your username and password to the appropriate endpoint as HTTP Basic Auth; this is equivalent to typing your username and password into a website.


requests.get(
  'https://api.github.com/user', 
  auth=HTTPBasicAuth('username', 'password')
)




A more secure method is to get an access token that acts as an equivalent to a username/password combination; the method to get an access token varies widely from API to API, but the most common framework for API authentication is OAuth. Here at Nylas, we use three-legged OAuth to grant an access token for user accounts that is restricted to scopes that define the specific data and functionality that can be accessed. This process is demonstrated in the Nylas Hosted Auth service.Illustration of the Nylas Hosted Auth service that integrates with 100% of email, calendar, and contacts providers, including Google


Once you have an access token, you can provide it as a bearer token in the request header: this is the most secure way to authenticate to a REST API with an access token:


my_headers = {'Authorization' : 'Bearer {access_token}'}
response = requests.get('http://httpbin.org/headers', headers=my_headers)


There are quite a few other methods to authenticate to a REST API, including digest, Kerberos, NTLM, and AuthBase. The use of these depends on the architecture decisions of the REST API producer.


Use Sessions to Manage Access Tokens 


Session objects come in handy when working with Python Requests as a tool to persist parameters that are needed for making multiple requests within a single session, like access tokens. Also, managing session cookies can provide a nice performance increase because you don’t need to open a new connection for every request.




session = requests.Session()
session.headers.update({'Authorization': 'Bearer {access_token}'})
response = session.get('https://httpbin.org/headers')




How to Handle HTTP Errors With Python Requests




API calls don’t always go as planned, and there’s a multitude of reasons why API requests might fail that could be the fault of either the server or the client. If you’re going to use a REST API, you need to understand how to handle the errors they output when things go wrong to make your code more robust. This section covers everything you need to know about handling HTTP errors with Python Requests.


The Basics of HTTP Status Codes


Before we dive into the specifics of Python Requests, we first need to take a step back and understand what HTTP status codes are and how they relate to errors you might encounter.


All status codes fall into one of five categories. 


    

  • 1xx Informational – Indicates that a request has been received and that the client should continue to make the requests for the data payload. You likely won’t need to worry about these status codes while working with Python Requests.
    • 

  • 2xx Successful – Indicates that a requested action has been received, understood, and accepted. You can use these codes to verify the existence of data before attempting to act on it.
    • 

  • 3xx Redirection – Indicates that the client must make an additional action to complete the request like accessing the resource via a proxy or a different endpoint. You may need to make additional requests, or modify your requests to deal with these codes.
    • 

  • 4xx Client Error – Indicates problems with the client, such as a lack of authorization, forbidden access, disallowed methods, or attempts to access nonexistent resources. This usually indicates configuration errors on the client application.
    • 

  • 5xx Server Error – Indicates problems with the server that provides the API. There are a large variety of server errors and they often require the API provider to resolve.
    • 



How to Check for HTTP Errors With Python Requests


The response objects has a status_code attribute that can be used to check for any errors the API might have reported. The next example shows how to use this attribute to check for successful and 404 not found HTTP status codes, and you can use this same format for all HTTP status codes.




response = requests.get("http://api.open-notify.org/astros.json")
if (response.status_code == 200):
    print("The request was a success!")
    # Code here will only run if the request is successful
elif (response.status_code == 404:
    print("Result not found!")
    # Code here will react to failed requests


To see this in action, try removing the last letter from the URL endpoint, the API should return a 404 status code.


If you want requests to raise an exception for all error codes (4xx and 5xx), you can use the raise_for_status() function and catch specific errors using Requests built-in exceptions. This next example accomplishes the same thing as the previous code example.




try:
    response = requests.get('http://api.open-notify.org/astros.json')
    response.raise_for_status()
    # Additional code will only run if the request is successful
except requests.exceptions.HTTPError as error:
    print(error)
    # This code will run if there is a 404 error.


TooManyRedirects 


Something that is often indicated by 3xx HTTP status codes is the requirement to redirect to a different location for the resource you’re requesting. This can sometimes result in a situation where you end up with an infinite redirect loop. The Python Requests module has the TooManyRedirects error that you can use to handle this problem. To resolve this problem, it’s likely the URL you’re using to access the resource is wrong and needs to be changed.




try:
    response = requests.get('http://api.open-notify.org/astros.json')
    response.raise_for_status()
    # Code here will only run if the request is successful
except requests.exceptions.TooManyRedirects as error:
    print(error)


You can optionally use the request options to set the maximum number of redirects:




response = requests.get('http://api.open-notify.org/astros.json', max_redirects=2)


Or disable redirecting completely within your request options:




response = requests.get('http://api.open-notify.org/astros.json', allow_redirects=False)



ConnectionError


So far, we’ve only looked at errors that come from an active server. What happens if you don’t receive a response from the server at all? Connection errors can occur for many different reasons, including a DNS failure, refused connection, internet connectivity issues or latency somewhere in the network. Python Requests offers the ConnectionError exception that indicates when your client is unable to connect to the server. 




try:
    response = requests.get('http://api.open-notify.org/astros.json') 
    # Code here will only run if the request is successful
except requests.ConnectionError as error:
    print(error)


This type of error might be temporary, or permanent. In the former scenario, you should retry the request again to see if there is a different result. In the latter scenario, you should make sure you’re able to deal with a prolonged inability to access data from the API, and it might require you to investigate your own connectivity issues.


Timeout 


Timeout errors occur when you’re able to connect to the API server, but it doesn’t complete the request within the allotted amount of time. Similar to the other errors we’ve looked at, Python Requests can handle this error with a Timeout exception:   




try:
    response = requests.get('http://api.open-notify.org/astros.json', timeout=0.00001)
    # Code here will only run if the request is successful
except requests.Timeout as error:
    print(error)


In this example, the timeout was set as a fraction of a second via the request options. Most APIs are unable to respond this quickly, so the code will produce a timeout exception. You can avoid this error by setting longer timeouts for your script, optimizing your requests to be smaller, or setting up a retry loop for the request. This can also sometimes indicate a problem with the API provider. One final solution is to incorporate asynchronous API calls to  prevent your code from stopping while it waits for larger responses.



How to Make Robust API Requests




 As we’ve seen, the Requests module elegantly handles common API request errors by utilizing  exception handling in Python. If we put all of the errors we’ve talked about together, we have a rather seamless way to handle any HTTP request error that comes our way:




try:
    response = requests.get('http://api.open-notify.org/astros.json', timeout=5)
    response.raise_for_status()
    # Code here will only run if the request is successful
except requests.exceptions.HTTPError as errh:
    print(errh)
except requests.exceptions.ConnectionError as errc:
    print(errc)
except requests.exceptions.Timeout as errt:
    print(errt)
except requests.exceptions.RequestException as err:
    print(err)




Experience Nylas REST APIs Using Python


Sign up for a Nylas developer account and make your first API call in minutes using Python. 




Learn More About Python




If you’ve made it this far, congrats! You’re well on your way to becoming a Python Requests wizard for whom no REST API is too great a match. Want to keep learning? We have tons of knowledgable Python experts here at Nylas, and we have in-depth content on our blog about packaging and deploying Python code to production, and using environment variables to make your Python code more secure.




Don’t miss the action and watch our livestream Coding with Nylas:










































Источник







  



                       
  





	









    
    
    
    



    

                
Понравилась статья? Поделить с друзьями:

        
        

























    








			

  • Http response error message
    • 

  • Http response code said error
    • 

  • Http request was unsuccessful status code 500 ошибка контур
    • 

  • Http request was unsuccessful status code 500 internal server error перевод
    • 

  • Http request was unsuccessful error network error перевод

    •