If you need to make HTTP requests with Python, then you may find yourself directed to the brilliant requests library. Though it’s a great library, you may have noticed that it’s not a built-in part of Python. If you prefer, for whatever reason, to limit your dependencies and stick to standard-library Python, then you can reach for urllib.request!
In this tutorial, you’ll:
- Learn how to make basic HTTP requests with
urllib.request - Dive into the nuts and bolts of an HTTP message and how
urllib.requestrepresents it - Understand how to deal with character encodings of HTTP messages
- Explore some common errors when using
urllib.requestand learn how to resolve them - Dip your toes into the world of authenticated requests with
urllib.request - Understand why both
urlliband therequestslibrary exist and when to use one or the other
If you’ve heard of HTTP requests, including GET and POST, then you’re probably ready for this tutorial. Also, you should’ve already used Python to read and write to files, ideally with a context manager, at least once.
Ultimately, you’ll find that making a request doesn’t have to be a frustrating experience, although it does tend to have that reputation. Many of the issues that you tend to run into are due to the inherent complexity of this marvelous thing called the Internet. The good news is that the urllib.request module can help to demystify much of this complexity.
Basic HTTP GET Requests With urllib.request
Before diving into the deep end of what an HTTP request is and how it works, you’re going to get your feet wet by making a basic GET request to a sample URL. You’ll also make a GET request to a mock REST API for some JSON data. In case you’re wondering about POST Requests, you’ll be covering them later in the tutorial, once you have some more knowledge of urllib.request.
To get started, you’ll make a request to www.example.com, and the server will return an HTTP message. Ensure that you’re using Python 3 or above, and then use the urlopen() function from urllib.request:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> body[:15]
b'<!doctype html>'
In this example, you import urlopen() from urllib.request. Using the context manager with, you make a request and receive a response with urlopen(). Then you read the body of the response and close the response object. With that, you display the first fifteen positions of the body, noting that it looks like an HTML document.
There you are! You’ve successfully made a request, and you received a response. By inspecting the content, you can tell that it’s likely an HTML document. Note that the printed output of the body is preceded by b. This indicates a bytes literal, which you may need to decode. Later in the tutorial, you’ll learn how to turn bytes into a string, write them to a file, or parse them into a dictionary.
The process is only slightly different if you want to make calls to REST APIs to get JSON data. In the following example, you’ll make a request to {JSON}Placeholder for some fake to-do data:
>>>
>>> from urllib.request import urlopen
>>> import json
>>> url = "https://jsonplaceholder.typicode.com/todos/1"
>>> with urlopen(url) as response:
... body = response.read()
...
>>> todo_item = json.loads(body)
>>> todo_item
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}
In this example, you’re doing pretty much the same as in the previous example. But in this one, you import urllib.request and json, using the json.loads() function with body to decode and parse the returned JSON bytes into a Python dictionary. Voila!
If you’re lucky enough to be using error-free endpoints, such as the ones in these examples, then maybe the above is all that you need from urllib.request. Then again, you may find that it’s not enough.
Now, before doing some urllib.request troubleshooting, you’ll first gain an understanding of the underlying structure of HTTP messages and learn how urllib.request handles them. This understanding will provide a solid foundation for troubleshooting many different kinds of issues.
The Nuts and Bolts of HTTP Messages
To understand some of the issues that you may encounter when using urllib.request, you’ll need to examine how a response is represented by urllib.request. To do that, you’ll benefit from a high-level overview of what an HTTP message is, which is what you’ll get in this section.
Before the high-level overview, a quick note on reference sources. If you want to get into the technical weeds, the Internet Engineering Task Force (IETF) has an extensive set of Request for Comments (RFC) documents. These documents end up becoming the actual specifications for things like HTTP messages. RFC 7230, part 1: Message Syntax and Routing, for example, is all about the HTTP message.
If you’re looking for some reference material that’s a bit easier to digest than RFCs, then the Mozilla Developer Network (MDN) has a great range of reference articles. For example, their article on HTTP messages, while still technical, is a lot more digestible.
Now that you know about these essential sources of reference information, in the next section you’ll get a beginner-friendly overview of HTTP messages.
Understanding What an HTTP Message Is
In a nutshell, an HTTP message can be understood as text, transmitted as a stream of bytes, structured to follow the guidelines specified by RFC 7230. A decoded HTTP message can be as simple as two lines:
GET / HTTP/1.1
Host: www.google.com
This specifies a GET request at the root (/) using the HTTP/1.1 protocol. The one and only header required is the host, www.google.com. The target server has enough information to make a response with this information.
A response is similar in structure to a request. HTTP messages have two main parts, the metadata and the body. In the request example above, the message is all metadata with no body. The response, on the other hand, does have two parts:
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
Server: gws
(... other headers ...)
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"
...
The response starts with a status line that specifies the HTTP protocol HTTP/1.1 and the status 200 OK. After the status line, you get many key-value pairs, such as Server: gws, representing all the response headers. This is the metadata of the response.
After the metadata, there’s a blank line, which serves as the divider between the headers and the body. Everything that follows the blank line makes up the body. This is the part that gets read when you’re using urllib.request.
You can assume that all HTTP messages follow these specifications, but it’s possible that some may break these rules or follow an older specification. It’s exceptionally rare for this to cause any issues, though. So, just keep it in the back of your mind in case you run into a strange bug!
In the next section, you’ll see how urllib.request deals with raw HTTP messages.
Understanding How urllib.request Represents an HTTP Message
The main representation of an HTTP message that you’ll be interacting with when using urllib.request is the HTTPResponse object. The urllib.request module itself depends on the low-level http module, which you don’t need to interact with directly. You do end up using some of the data structures that http provides, though, such as HTTPResponse and HTTPMessage.
When you make a request with urllib.request.urlopen(), you get an HTTPResponse object in return. Spend some time exploring the HTTPResponse object with pprint() and dir() to see all the different methods and properties that belong to it:
>>>
>>> from urllib.request import urlopen
>>> from pprint import pprint
>>> with urlopen("https://www.example.com") as response:
... pprint(dir(response))
...
To reveal the output of this code snippet, click to expand the collapsible section below:
['__abstractmethods__',
'__class__',
'__del__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__enter__',
'__eq__',
'__exit__',
'__format__',
'__ge__',
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__iter__',
'__le__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__next__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'_abc_impl',
'_checkClosed',
'_checkReadable',
'_checkSeekable',
'_checkWritable',
'_check_close',
'_close_conn',
'_get_chunk_left',
'_method',
'_peek_chunked',
'_read1_chunked',
'_read_and_discard_trailer',
'_read_next_chunk_size',
'_read_status',
'_readall_chunked',
'_readinto_chunked',
'_safe_read',
'_safe_readinto',
'begin',
'chunk_left',
'chunked',
'close',
'closed',
'code',
'debuglevel',
'detach',
'fileno',
'flush',
'fp',
'getcode',
'getheader',
'getheaders',
'geturl',
'headers',
'info',
'isatty',
'isclosed',
'length',
'msg',
'peek',
'read',
'read1',
'readable',
'readinto',
'readinto1',
'readline',
'readlines',
'reason',
'seek',
'seekable',
'status',
'tell',
'truncate',
'url',
'version',
'will_close',
'writable',
'write',
'writelines']
That’s a lot of methods and properties, but you’ll only end up using a handful of these . Apart from .read(), the important ones usually involve getting information about the headers.
One way to inspect all the headers is to access the .headers attribute of the HTTPResponse object. This will return an HTTPMessage object. Conveniently, you can treat an HTTPMessage like a dictionary by calling .items() on it to get all the headers as tuples:
>>>
>>> with urlopen("https://www.example.com") as response:
... pass
...
>>> response.headers
<http.client.HTTPMessage object at 0x000001E029D9F4F0>
>>> pprint(response.headers.items())
[('Accept-Ranges', 'bytes'),
('Age', '398424'),
('Cache-Control', 'max-age=604800'),
('Content-Type', 'text/html; charset=UTF-8'),
('Date', 'Tue, 25 Jan 2022 12:18:53 GMT'),
('Etag', '"3147526947"'),
('Expires', 'Tue, 01 Feb 2022 12:18:53 GMT'),
('Last-Modified', 'Thu, 17 Oct 2019 07:18:26 GMT'),
('Server', 'ECS (nyb/1D16)'),
('Vary', 'Accept-Encoding'),
('X-Cache', 'HIT'),
('Content-Length', '1256'),
('Connection', 'close')]
Now you have access to all the response headers! You probably won’t need most of this information, but rest assured that some applications do use it. For example, your browser might use the headers to read the response, set cookies, and determine an appropriate cache lifetime.
There are convenience methods to get the headers from an HTTPResponse object because it’s quite a common operation. You can call .getheaders() directly on the HTTPResponse object, which will return exactly the same list of tuples as above. If you’re only interested in one header, say the Server header, then you can use the singular .getheader("Server") on HTTPResponse or use the square bracket ([]) syntax on .headers from HTTPMessage:
>>>
>>> response.getheader("Server")
'ECS (nyb/1D16)'
>>> response.headers["Server"]
'ECS (nyb/1D16)'
Truth be told, you probably won’t need to interact with the headers directly like this. The information that you’re most likely to need will probably already have some built-in helper methods, but now you know, in case you ever need to dig deeper!
Closing an HTTPResponse
The HTTPResponse object has a lot in common with the file object. The HTTPResponse class inherits from the IOBase class, as do file objects, which means that you have to be mindful of opening and closing.
In simple programs, you’re not likely to notice any issues if you forget to close HTTPResponse objects. For more complex projects, though, this can significantly slow execution and cause bugs that are difficult to pinpoint.
Problems arise because input/output (I/O) streams are limited. Each HTTPResponse requires a stream to be kept clear while it’s being read. If you never close your streams, this will eventually prevent any other stream from being opened, and it might interfere with other programs or even your operating system.
So, make sure you close your HTTPResponse objects! For your convenience, you can use a context manager, as you’ve seen in the examples. You can also achieve the same result by explicitly calling .close() on the response object:
>>>
>>> from urllib.request import urlopen
>>> response = urlopen("https://www.example.com")
>>> body = response.read()
>>> response.close()
In this example, you don’t use a context manager, but instead close the response stream explicitly. The above example still has an issue, though, because an exception may be raised before the call to .close(), preventing the proper teardown. To make this call unconditional, as it should be, you can use a try … except block with both an else and a finally clause:
>>>
>>> from urllib.request import urlopen
>>> response = None
>>> try:
... response = urlopen("https://www.example.com")
... except Exception as ex:
... print(ex)
... else:
... body = response.read()
... finally:
... if response is not None:
... response.close()
In this example, you achieve an unconditional call to .close() by using the finally block, which will always run regardless of exceptions raised. The code in the finally block first checks if the response object exists with is not None, and then closes it.
That said, this is exactly what a a context manager does, and the with syntax is generally preferred. Not only is the with syntax less verbose and more readable, but it also protects you from pesky errors of omission. Put another way, it’s a far better guard against accidentally forgetting to close the object:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... response.read(50)
... response.read(50)
...
b'<!doctype html>n<html>n<head>n <title>Example D'
b'omain</title>nn <meta charset="utf-8" />n <m'
In this example, you import urlopen() from the urllib.request module. You use the with keyword with .urlopen() to assign the HTTPResponse object to the variable response. Then, you read the first fifty bytes of the response and then read the following fifty bytes, all within the with block. Finally, you close the with block, which executes the request and runs the lines of code within its block.
With this code, you cause two sets of fifty bytes each to be displayed. The HTTPResponse object will close once you exit the with block scope, meaning that the .read() method will only return empty bytes objects:
>>>
>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
... response.read(50)
...
b'<!doctype html>n<html>n<head>n <title>Example D'
>>> response.read(50)
b''
In this example, the second call to read fifty bytes is outside the with scope. Being outside the with block means that HTTPResponse is closed, even though you can still access the variable. If you try to read from HTTPResponse when it’s closed, it’ll return an empty bytes object.
Another point to note is that you can’t reread a response once you’ve read all the way to the end:
>>>
>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
... first_read = response.read()
... second_read = response.read()
...
>>> len(first_read)
1256
>>> len(second_read)
0
This example shows that once you’ve read a response, you can’t read it again. If you’ve fully read the response, the subsequent attempt just returns an empty bytes object even though the response isn’t closed. You’d have to make the request again.
In this regard, the response is different from a file object, because with a file object, you can read it multiple times by using the .seek() method, which HTTPResponse doesn’t support. Even after closing a response, you can still access the headers and other metadata, though.
Exploring Text, Octets, and Bits
In most of the examples so far, you read the response body from HTTPResponse, displayed the resulting data immediately, and noted that it was displayed as a bytes object. This is because text information in computers isn’t stored or transmitted as letters, but as bytes!
A raw HTTP message sent over the wire is broken up into a sequence of bytes, sometimes referred to as octets. Bytes are 8-bit chunks. For example, 01010101 is a byte. To learn more about binary, bits, and bytes, check out Bitwise Operators in Python.
So how do you represent letters with bytes? A byte has 256 potential combinations, and you can assign a letter to each combination. You can assign 00000001 to A, 00000010 to B, and so on. ASCII character encoding, which is quite common, uses this type of system to encode 128 characters, which is enough for a language like English. This is particularly convenient because just one byte can represent all the characters, with space to spare.
All the standard English characters, including capitals, punctuation, and numerals, fit within ASCII. On the other hand, Japanese is thought to have around fifty thousand logographic characters, so 128 characters won’t cut it! Even the 256 characters that are theoretically available within one byte wouldn’t be nearly enough for Japanese. So, to accomodate all the world’s languages there are many different systems to encode characters.
Even though there are many systems, one thing you can rely on is the fact that they’ll always be broken up into bytes. Most servers, if they can’t resolve the MIME type and character encoding, default to application/octet-stream, which literally means a stream of bytes. Then whoever receives the message can work out the character encoding.
Dealing With Character Encodings
Problems often arise because, as you may have guessed, there are many, many different potential character encodings. The dominant character encoding today is UTF-8, which is an implementation of Unicode. Luckily, ninety-eight percent of web pages today are encoded in UTF-8!
UTF-8 is dominant because it can efficiently handle a mind-boggling number of characters. It handles all the 1,112,064 potential characters defined by Unicode, encompassing Chinese, Japanese, Arabic (with right-to-left scripts), Russian, and many more character sets, including emojis!
UTF-8 remains efficient because it uses a variable number of bytes to encode characters, which means that for many characters, it only requires one byte, while for others it can require up to four bytes.
While UTF-8 is dominant, and you usually won’t go wrong with assuming UTF-8 encodings, you’ll still run into different encodings all the time. The good news is that you don’t need to be an expert on encodings to handle them when using urllib.request.
Going From Bytes to Strings
When you use urllib.request.urlopen(), the body of the response is a bytes object. The first thing you may want to do is to convert the bytes object to a string. Perhaps you want to do some web scraping. To do this, you need to decode the bytes. To decode the bytes with Python, all you need to find out is the character encoding used. Encoding, especially when referring to character encoding, is often referred to as a character set.
As mentioned, ninety-eight percent of the time, you’ll probably be safe defaulting to UTF-8:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> decoded_body = body.decode("utf-8")
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>
In this example, you take the bytes object returned from response.read() and decode it with the bytes object’s .decode() method, passing in utf-8 as an argument. When you print decoded_body, you can see that it’s now a string.
That said, leaving it up to chance is rarely a good strategy. Fortunately, headers are a great place to get character set information:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'utf-8'
>>> decoded_body = body.decode(character_set)
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>
In this example, you call .get_content_charset() on the .headers object of response and use that to decode. This is a convenience method that parses the Content-Type header so that you can painlessly decode bytes into text.
Going From Bytes to File
If you want to decode bytes into text, now you’re good to go. But what if you want to write the body of a response into a file? Well, you have two options:
- Write the bytes directly to the file
- Decode the bytes into a Python string, and then encode the string back into a file
The first method is the most straightforward, but the second method allows you to change the encoding if you want to. To learn about file manipulation in more detail, take a look at Real Python’s Reading and Writing Files in Python (Guide).
To write the bytes directly to a file without having to decode, you’ll need the built-in open() function, and you’ll need to ensure that you use write binary mode:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> with open("example.html", mode="wb") as html_file:
... html_file.write(body)
...
1256
Using open() in wb mode bypasses the need to decode or encode and dumps the bytes of the HTTP message body into the example.html file. The number that’s output after the writing operation indicates the number of bytes that have been written. That’s it! You’ve written the bytes directly to a file without encoding or decoding anything.
Now say you have a URL that doesn’t use UTF-8, but you want to write the contents to a file with UTF-8. For this, you’d first decode the bytes into a string and then encode the string into a file, specifying the character encoding.
Google’s home page seems to use different encodings depending on your location. In much of Europe and the US, it uses the ISO-8859-1 encoding:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.google.com") as response:
... body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'ISO-8859-1'
>>> content = body.decode(character_set)
>>> with open("google.html", encoding="utf-8", mode="w") as file:
... file.write(content)
...
14066
In this code, you got the response character set and used it to decode the bytes object into a string. Then you wrote the string to a file, encoding it using UTF-8.
Once you’ve written to a file, you should be able to open the resulting file in your browser or text editor. Most modern text processors can detect the character encoding automatically.
If there are encoding errors and you’re using Python to read a file, then you’ll likely get an error:
>>>
>>> with open("encoding-error.html", mode="r", encoding="utf-8") as file:
... lines = file.readlines()
...
UnicodeDecodeError:
'utf-8' codec can't decode byte
Python explicitly stops the process and raises an exception, but in a program that displays text, such as the browser where you’re viewing this page, you may find the infamous replacement characters:
The black rhombus with a white question mark (�), the square (□), and the rectangle (▯) are often used as replacements for characters which couldn’t be decoded.
Sometimes, decoding seems to work but results in unintelligible sequences, such as æ–‡å—化ã‘., which also suggests the wrong character set was used. In Japan, they even have a word for text that’s garbled due to character encoding issues, Mojibake, because these issues plagued them at the start of the Internet age.
With that, you should now be equipped to write files with the raw bytes returned from urlopen(). In the next section, you’ll learn how to parse bytes into a Python dictionary with the json module.
Going From Bytes to Dictionary
For application/json responses, you’ll often find that they don’t include any encoding information:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://httpbin.org/json") as response:
... body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> print(character_set)
None
In this example, you use the json endpoint of httpbin, a service that allows you to experiment with different types of requests and responses. The json endpoint simulates a typical API that returns JSON data. Note that the .get_content_charset() method returns nothing in its response.
Even though there’s no character encoding information, all is not lost. According to RFC 4627, the default encoding of UTF-8 is an absolute requirement of the application/json specification. That’s not to say that every single server plays by the rules, but generally, you can assume that if JSON is being transmitted, it’ll almost always be encoded using UTF-8.
Fortunately, json.loads() decodes byte objects under the hood and even has some leeway in terms of different encodings that it can deal with. So, json.loads() should be able to cope with most bytes objects that you throw at it, as long as they’re valid JSON:
>>>
>>> import json
>>> json.loads(body)
{'slideshow': {'author': 'Yours Truly', 'date': 'date of publication', 'slides'
: [{'title': 'Wake up to WonderWidgets!', 'type': 'all'}, {'items': ['Why <em>W
onderWidgets</em> are great', 'Who <em>buys</em> WonderWidgets'], 'title': 'Ove
rview', 'type': 'all'}], 'title': 'Sample Slide Show'}}
As you can see, the json module handles the decoding automatically and produces a Python dictionary. Almost all APIs return key-value information as JSON, although you might run into some older APIs that work with XML. For that, you might want to look into the Roadmap to XML Parsers in Python.
With that, you should know enough about bytes and encodings to be dangerous! In the next section, you’ll learn how to troubleshoot and fix a couple of common errors that you might run into when using urllib.request.
Common urllib.request Troubles
There are many kinds of issues you can run into on the world wild web, whether you’re using urllib.request or not. In this section, you’ll learn how to deal with a couple of the most common errors when getting started out: 403 errors and TLS/SSL certificate errors. Before looking at these specific errors, though, you’ll first learn how to implement error handling more generally when using urllib.request.
Implementing Error Handling
Before you turn your attention to specific errors, boosting your code’s ability to gracefully deal with assorted errors will pay off. Web development is plagued with errors, and you can invest a lot of time in handling errors sensibly. Here, you’ll learn to handle HTTP, URL, and timeout errors when using urllib.request.
HTTP status codes accompany every response in the status line. If you can read a status code in the response, then the request reached its target. While this is good, you can only consider the request a complete success if the response code starts with a 2. For example, 200 and 201 represent successful requests. If the status code is 404 or 500, for example, something went wrong, and urllib.request will raise an HTTPError.
Sometimes mistakes happen, and the URL provided isn’t correct, or a connection can’t be made for another reason. In these cases, urllib.request will raise a URLError.
Finally, sometimes servers just don’t respond. Maybe your network connection is slow, the server is down, or the server is programmed to ignore specific requests. To deal with this, you can pass a timeout argument to urlopen() to raise a TimeoutError after a certain amount of time.
The first step in handling these exceptions is to catch them. You can catch errors produced within urlopen() with a try … except block, making use of the HTTPError, URLError, and TimeoutError classes:
# request.py
from urllib.error import HTTPError, URLError
from urllib.request import urlopen
def make_request(url):
try:
with urlopen(url, timeout=10) as response:
print(response.status)
return response.read(), response
except HTTPError as error:
print(error.status, error.reason)
except URLError as error:
print(error.reason)
except TimeoutError:
print("Request timed out")
The function make_request() takes a URL string as an argument, tries to get a response from that URL with urllib.request, and catches the HTTPError object that’s raised if an error occurs. If the URL is bad, it’ll catch a URLError. If it goes through without any errors, it’ll just print the status and return a tuple containing the body and the response. The response will close after return.
The function also calls urlopen() with a timeout argument, which will cause a TimeoutError to be raised after the seconds specified. Ten seconds is generally a good amount of time to wait for a response, though as always, much depends on the server that you need to make the request to.
Now you’re set up to gracefully handle a variety of errors, including but not limited to the errors that you’ll cover next.
Dealing With 403 Errors
You’ll now use the make_request() function to make some requests to httpstat.us, which is a mock server used for testing. This mock server will return responses that have the status code you request. If you make a request to https://httpstat.us/200, for example, you should expect a 200 response.
APIs like httpstat.us are used to ensure that your application can handle all the different status codes it might encounter. httpbin also has this functionality, but httpstat.us has a more comprehensive selection of status codes. It even has the infamous and semi-official 418 status code that returns the message I’m a teapot!
To interact with the make_request() function that you wrote in the previous section, run the script in interactive mode:
With the -i flag, this command will run the script in interactive mode. This means that it’ll execute the script and then open the Python REPL afterward, so you can now call the function that you just defined:
>>>
>>> make_request("https://httpstat.us/200")
200
(b'200 OK', <http.client.HTTPResponse object at 0x0000023D612660B0>)
>>> make_request("https://httpstat.us/403")
403 Forbidden
Here you tried the 200 and 403 endpoints of httpstat.us. The 200 endpoint goes through as anticipated and returns the body of the response and the response object. The 403 endpoint just printed the error message and didn’t return anything, also as expected.
The 403 status means that the server understood the request but won’t fulfill it. This is a common error that you can run into, especially while web scraping. In many cases, you can solve it by passing a User-Agent header.
One of the primary ways that servers identify who or what is making the request is by examining the User-Agent header. The raw default request sent by urllib.request is the following:
GET https://httpstat.us/403 HTTP/1.1
Accept-Encoding: identity
Host: httpstat.us
User-Agent: Python-urllib/3.10
Connection: close
Notice that User-Agent is listed as Python-urllib/3.10. You may find that some sites will try to block web scrapers, and this User-Agent is a dead giveaway. With that said, you can set your own User-Agent with urllib.request, though you’ll need to modify your function a little:
# request.py
from urllib.error import HTTPError, URLError
-from urllib.request import urlopen
+from urllib.request import urlopen, Request
-def make_request(url):
+def make_request(url, headers=None):
+ request = Request(url, headers=headers or {})
try:
- with urlopen(url, timeout=10) as response:
+ with urlopen(request, timeout=10) as response:
print(response.status)
return response.read(), response
except HTTPError as error:
print(error.status, error.reason)
except URLError as error:
print(error.reason)
except TimeoutError:
print("Request timed out")
To customize the headers that you send out with your request, you first have to instantiate a Request object with the URL. Additionally, you can pass in a keyword argument of headers, which accepts a standard dictionary representing any headers you wish to include. So, instead of passing the URL string directly into urlopen(), you pass this Request object which has been instantiated with the URL and headers.
To use this revamped function, restart the interactive session, then call make_request() with a dictionary representing the headers as an argument:
>>>
>>> body, response = make_request(
... "https://www.httpbin.org/user-agent",
... {"User-Agent": "Real Python"}
... )
200
>>> body
b'{n "user-agent": "Real Python"n}n'
In this example, you make a request to httpbin. Here you use the user-agent endpoint to return the request’s User-Agent value. Because you made the request with a custom user agent of Real Python, this is what gets returned.
Some servers are strict, though, and will only accept requests from specific browsers. Luckily, it’s possible to find standard User-Agent strings on the web, including through a user agent database. They’re just strings, so all you need to do is copy the user agent string of the browser that you want to impersonate and use it as the value of the User-Agent header.
Fixing the SSL CERTIFICATE_VERIFY_FAILED Error
Another common error is due to Python not being able to access the required security certificate. To simulate this error, you can use some mock sites that have known bad SSL certificates, provided by badssl.com. You can make a request to one of them, such as superfish.badssl.com, and experience the error firsthand:
>>>
>>> from urllib.request import urlopen
>>> urlopen("https://superfish.badssl.com/")
Traceback (most recent call last):
(...)
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
(...)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Here, making a request to an address with a known bad SSL certificate will result in CERTIFICATE_VERIFY_FAILED which is a type of URLError.
SSL stands for Secure Sockets Layer. This is something of a misnomer because SSL was deprecated in favor of TLS, Transport Layer Security. Sometimes old terminology just sticks! It’s a way to encrypt network traffic so that a hypothetical listener can’t eavesdrop on the information transmitted over the wire.
These days, most website addresses are preceded not by http:// but by https://, with the s standing for secure. HTTPS connections must be encrypted through the TLS. urllib.request can handle both HTTP and HTTPS connections.
The details of HTTPS are far beyond the scope of this tutorial, but you can think of an HTTPS connection as involving two stages, the handshake and the transfer of information. The handshake ensures that the connection is secure. For more information about Python and HTTPS, check out Exploring HTTPS With Python.
To establish that a particular server is secure, programs that make requests rely on a store of trusted certificates. The server’s certificate is verified during the handshake stage. Python uses the operating system’s store of certificates. If Python can’t find the system’s store of certificates, or if the store is out of date, then you’ll run into this error.
Sometimes the store of certificates that Python can access is out of date, or Python can’t reach it, for whatever reason. This is frustrating because you can sometimes visit the URL from your browser, which thinks that it’s secure, yet urllib.request still raises this error.
You may be tempted to opt out of verifying the certificate, but this will render your connection insecure and is definitely not recommended:
>>>
>>> import ssl
>>> from urllib.request import urlopen
>>> unverified_context = ssl._create_unverified_context()
>>> urlopen("https://superfish.badssl.com/", context=unverified_context)
<http.client.HTTPResponse object at 0x00000209CBE8F220>
Here you import the ssl module, which allows you to create an unverified context. You can then pass this context to urlopen() and visit a known bad SSL certificate. The connection successfully goes through because the SSL certificate isn’t checked.
Before resorting to these desperate measures, try updating your OS or updating your Python version. If that fails, then you can take a page from the requests library and install certifi:
- Windows
- Linux + macOS
PS> python -m venv venv
PS> .venvScriptsactivate
(venv) PS> python -m pip install certifi
$ python3 -m venv venv
$ source venv/bin/activate.sh
(venv) $ python3 -m pip install certifi
certifi is a collection of certificates that you can use instead of your system’s collection. You do this by creating an SSL context with the certifi bundle of certificates instead of the OS’s bundle:
>>>
>>> import ssl
>>> from urllib.request import urlopen
>>> import certifi
>>> certifi_context = ssl.create_default_context(cafile=certifi.where())
>>> urlopen("https://sha384.badssl.com/", context=certifi_context)
<http.client.HTTPResponse object at 0x000001C7407C3490>
In this example, you used certifi to act as your SSL certificate store, and you used it to successfully connect to a site with a known good SSL certificate. Note that instead of ._create_unverified_context(), you use .create_default_context().
This way, you can stay secure without too much trouble! In the next section, you’ll be dipping your toes into the world of authentication.
Authenticated Requests
Authentication is a vast subject, and if you’re dealing with authentication much more complicated than what’s covered here, this might be a good jumping-off point into the requests package.
In this tutorial, you’ll only cover one authentication method, which serves as an example of the type of adjustments that you have to make to authenticate your requests. urllib.request does have a lot of other functionality that helps with authentication, but that won’t be covered in this tutorial.
One of the most common authentication tools is the bearer token, specified by RFC 6750. It’s often used as part of OAuth, but can also be used in isolation. It’s also most common to see as a header, which you can use with your current make_request() function:
>>>
>>> token = "abcdefghijklmnopqrstuvwxyz"
>>> headers = {
... "Authorization": f"Bearer {token}"
... }
>>> make_request("https://httpbin.org/bearer", headers)
200
(b'{n "authenticated": true, n "token": "abcdefghijklmnopqrstuvwxyz"n}n',
<http.client.HTTPResponse object at 0x0000023D612642E0>)
In this example, you make a request to the httpbin /bearer endpoint, which simulates bearer authentication. It’ll accept any string as a token. It only requires the proper format specified by RFC 6750. The name has to be Authorization, or sometimes the lowercase authorization, and the value has to be Bearer, with a single space between that and the token.
Congratulations, you’ve successfully authenticated, using a bearer token!
Another form of authentication is called Basic Access Authentication, which is a very simple method of authentication, only slightly better than sending a username and password in a header. It’s very insecure!
One of the most common protocols in use today is OAuth (Open Authorization). If you’ve ever used Google, GitHub, or Facebook to sign into another website, then you’ve used OAuth. The OAuth flow generally involves a few requests between the service that you want to interact with and an identity server, resulting in a short-lived bearer token. This bearer token can then be used for a period of time with bearer authentication.
Much of authentication comes down to understanding the specific protocol that the target server uses and reading the documentation closely to get it working.
POST Requests With urllib.request
You’ve made a lot of GET requests, but sometimes you want to send information. That’s where POST requests come in. To make POST requests with urllib.request, you don’t have to explicitly change the method. You can just pass a data object to a new Request object or directly to urlopen(). The data object must be in a special format, though. You’ll adapt your make_request() function slightly to support POST requests by adding the data parameter:
# request.py
from urllib.error import HTTPError, URLError
from urllib.request import urlopen, Request
-def make_request(url, headers=None):
+def make_request(url, headers=None, data=None):
- request = Request(url, headers=headers or {})
+ request = Request(url, headers=headers or {}, data=data)
try:
with urlopen(request, timeout=10) as response:
print(response.status)
return response.read(), response
except HTTPError as error:
print(error.status, error.reason)
except URLError as error:
print(error.reason)
except TimeoutError:
print("Request timed out")
Here you just modified the function to accept a data argument with a default value of None, and you passed that right into the Request instantiation. That’s not all that needs to be done, though. You can use one of two different formats to execute a POST request:
- Form Data:
application/x-www-form-urlencoded - JSON:
application/json
The first format is the oldest format for POST requests and involves encoding the data with percent encoding, also known as URL encoding. You may have noticed key-value pairs URL encoded as a query string. Keys are separated from values with an equal sign (=), key-value pairs are separated with an ampersand (&), and spaces are generally suppressed but can be replaced with a plus sign (+).
If you’re starting off with a Python dictionary, to use the form data format with your make_request() function, you’ll need to encode twice:
- Once to URL encode the dictionary
- Then again to encode the resulting string into bytes
For the first stage of URL encoding, you’ll use another urllib module, urllib.parse. Remember to start your script in interactive mode so that you can use the make_request() function and play with it on the REPL:
>>>
>>> from urllib.parse import urlencode
>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> url_encoded_data = urlencode(post_dict)
>>> url_encoded_data
'Title=Hello+World&Name=Real+Python'
>>> post_data = url_encoded_data.encode("utf-8")
>>> body, response = make_request(
... "https://httpbin.org/anything", data=post_data
... )
200
>>> print(body.decode("utf-8"))
{
"args": {},
"data": "",
"files": {},
"form": {
"Name": "Real Python",
"Title": "Hello World"
},
"headers": {
"Accept-Encoding": "identity",
"Content-Length": "34",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "Python-urllib/3.10",
"X-Amzn-Trace-Id": "Root=1-61f25a81-03d2d4377f0abae95ff34096"
},
"json": null,
"method": "POST",
"origin": "86.159.145.119",
"url": "https://httpbin.org/anything"
}
In this example, you:
- Import
urlencode()from theurllib.parsemodule - Initialize your POST data, starting with a dictionary
- Use the
urlencode()function to encode the dictionary - Encode the resulting string into bytes using UTF-8 encoding
- Make a request to the
anythingendpoint ofhttpbin.org - Print the UTF-8 decoded response body
UTF-8 encoding is part of the specification for the application/x-www-form-urlencoded type. UTF-8 is used preemptively to decode the body because you already know that httpbin.org reliably uses UTF-8.
The anything endpoint from httpbin acts as a sort of echo, returning all the information it received so that you can inspect the details of the request you made. In this case, you can confirm that method is indeed POST, and you can see that the data you sent is listed under form.
To make the same request with JSON, you’ll turn a Python dictionary into a JSON string with json.dumps(), encode it with UTF-8, pass it as the data argument, and finally add a special header to indicate that the data type is JSON:
>>>
>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> import json
>>> json_string = json.dumps(post_dict)
>>> json_string
'{"Title": "Hello World", "Name": "Real Python"}'
>>> post_data = json_string.encode("utf-8")
>>> body, response = make_request(
... "https://httpbin.org/anything",
... data=post_data,
... headers={"Content-Type": "application/json"},
... )
200
>>> print(body.decode("utf-8"))
{
"args": {},
"data": "{"Title": "Hello World", "Name": "Real Python"}",
"files": {},
"form": {},
"headers": {
"Accept-Encoding": "identity",
"Content-Length": "47",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "Python-urllib/3.10",
"X-Amzn-Trace-Id": "Root=1-61f25a81-3e35d1c219c6b5944e2d8a52"
},
"json": {
"Name": "Real Python",
"Title": "Hello World"
},
"method": "POST",
"origin": "86.159.145.119",
"url": "https://httpbin.org/anything"
}
To serialize the dictionary this time around, you use json.dumps() instead of urlencode(). You also explicitly add the Content-Type header with a value of application/json. With this information, the httpbin server can deserialize the JSON on the receiving end. In its response, you can see the data listed under the json key.
With that, you can now start making POST requests. This tutorial won’t go into more detail about the other request methods, such as PUT. Suffice to say that you can also explicitly set the method by passing a method keyword argument to the instantiation of the Request object.
The Request Package Ecosystem
To round things out, this last section of the tutorial is dedicated to clarifying the package ecosystem around HTTP requests with Python. Because there are many packages, with no clear standard, it can be confusing. That said, there are use cases for each package, which just means more choice for you!
What Are urllib2 and urllib3?
To answer this question, you need to go back to early Python, all the way back to version 1.2, when the original urllib was introduced. Around version 1.6, a revamped urllib2 was added, which lived alongside the original urllib. When Python 3 came along, the original urllib was deprecated, and urllib2 dropped the 2, taking on the original urllib name. It also split into parts:
urllib.errorurllib.parseurllib.requesturllib.responseurllib.robotparser
So what about urllib3? That’s a third-party library developed while urllib2 was still around. It’s not related to the standard library because it’s an independently maintained library. Interestingly, the requests library actually uses urllib3 under the hood, and so does pip!
When Should I Use requests Over urllib.request?
The main answer is ease of use and security. urllib.request is considered a low-level library, which exposes a lot of the detail about the workings of HTTP requests. The Python documentation for urllib.request makes no bones about recommending requests as a higher-level HTTP client interface.
If you interact with many different REST APIs, day in and day out, then requests is highly recommended. The requests library bills itself as “built for human beings” and has successfully created an intuitive, secure, and straightforward API around HTTP. It’s usually considered the go-to library! If you want to know more about the requests library, check out the Real Python guide to requests.
An example of how requests makes things easier is when it comes to character encoding. You’ll remember that with urllib.request, you have to be aware of encodings and take a few steps to ensure an error-free experience. The requests package abstracts that away and will resolve the encoding by using chardet, a universal character encoding detector, just in case there’s any funny business.
If your goal is to learn more about standard Python and the details of how it deals with HTTP requests, then urllib.request is a great way to get into that. You could even go further and use the very low-level http modules. On the other hand, you may just want to keep dependencies to a minimum, which urllib.request is more than capable of.
Why Is requests Not Part of the Standard Library?
Maybe you’re wondering why requests isn’t part of core Python by this point.
This is a complex issue, and there’s no hard and fast answer to it. There are many speculations as to why, but two reasons seem to stand out:
requestshas other third-party dependencies that would need to be integrated too.requestsneeds to stay agile and can do this better outside the standard library.
The requests library has third-party dependencies. Integrating requests into the standard library would mean also integrating chardet, certifi, and urllib3, among others. The alternative would be to fundamentally change requests to use only Python’s existing standard library. This is no trivial task!
Integrating requests would also mean that the existing team that develops this library would have to relinquish total control over the design and implementation, giving way to the PEP decision-making process.
HTTP specifications and recommendations change all the time, and a high-level library has to be agile enough to keep up. If there’s a security exploit to be patched, or a new workflow to add, the requests team can build and release far more quickly than they could as part of the Python release process. There have supposedly been times when they’ve released a security fix twelve hours after a vulnerability was discovered!
For an interesting overview of these issues and more, check out Adding Requests to The Standard Library, which summarizes a discussion at the Python Language Summit with Kenneth Reitz, the creator and maintainer of Requests.
Because this agility is so necessary to requests and its underlying urllib3, the paradoxical statement that requests is too important for the standard library is often used. This is because so much of the Python community depends on requests and its agility that integrating it into core Python would probably damage it and the Python community.
On the GitHub repository issues board for requests, an issue was posted, asking for the inclusion of requests in the standard library. The developers of requests and urllib3 chimed in, mainly saying they would likely lose interest in maintaining it themselves. Some even said they would fork the repositories and continue developing them for their own use cases.
With that said, note that the requests library GitHub repository is hosted under the Python Software Foundation’s account. Just because something isn’t part of the Python standard library doesn’t mean that it’s not an integral part of the ecosystem!
It seems that the current situation works for both the Python core team and the maintainers of requests. While it may be slightly confusing for newcomers, the existing structure gives the most stable experience for HTTP requests.
It’s also important to note that HTTP requests are inherently complex. urllib.request doesn’t try to sugarcoat that too much. It exposes a lot of the inner workings of HTTP requests, which is why it’s billed as a low-level module. Your choice of requests versus urllib.request really depends on your particular use case, security concerns, and preference.
Conclusion
You’re now equipped to use urllib.request to make HTTP requests. Now you can use this built-in module in your projects, keeping them dependency-free for longer. You’ve also gained the in-depth understanding of HTTP that comes from using a lower-level module, such as urllib.request.
In this tutorial, you’ve:
- Learned how to make basic HTTP requests with
urllib.request - Explored the nuts and bolts of an HTTP message and studied how it’s represented by
urllib.request - Figured out how to deal with character encodings of HTTP messages
- Explored some common errors when using
urllib.requestand learned how to resolve them - Dipped your toes into the world of authenticated requests with
urllib.request - Understood why both
urlliband therequestslibrary exist and when to use one or the other
You’re now in a position to make basic HTTP requests with urllib.request, and you also have the tools to dive deeper into low-level HTTP terrain with the standard library. Finally, you can choose whether to use requests or urllib.request, depending on what you want or need. Have fun exploring the Web!
Today when I using pytube python package to download the YouTube video, I got the following error message:
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>I never got this error in the past, and it seems not a problem with YouTube page version. After I check on the Internet, I find maybe the version of urllib module is different from the previous version.
Simply put, the error seems an error in verifying the SSL certificate. Maybe the target website uses a certificate signed by itself, I’m not really sure.
Sample Program With Error
Let’s take an example:
import urllib.request r = urllib.request.urlopen('https://google.com')
Output:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 817, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>Solution
However, if we choose not to authenticate SSL certificate, we can make urllib work.
# coding: utf-8 import urllib.request import ssl def main(): ssl._create_default_https_context = ssl._create_unverified_context r = urllib.request.urlopen('https://google.com') print(r.status) print(r) if __name__ == '__main__': main()
Output:
200
<http.client.HTTPResponse object at 0x7fc1af051ef0>In addition to this method, if you just want a specific connection not to be verified, and the rest of the connection is to verify the SSL certificate, you can write:
# coding: utf-8 import urllib.request import ssl def main(): context = ssl._create_unverified_context() r = urllib.request.urlopen('https://google.com', context=context) print(r.status) print(r) if __name__ == '__main__': main()
Output:
200
<http.client.HTTPResponse object at 0x7fc1af051ef0>References
- https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error
- https://moreless.medium.com/how-to-fix-python-ssl-certificate-verify-failed-97772d9dd14c
Read More
- [Python] Using package pytube to download YouTube videos
- [Python] Using «GoogleNews» package to get the Google News
- Автор
-
Michael Foord
Введение¶
urllib.request — это модуль Python для получения URL (Uniform Resource Locators). Он предлагает очень простой интерфейс в виде функции urlopen. Он способен получать URL, используя множество различных протоколов. Она также предлагает немного более сложный интерфейс для обработки общих ситуаций, таких как базовая аутентификация, cookies, прокси и так далее. Эти функции предоставляются объектами, называемыми обработчиками и открывателями.
urllib.request поддерживает получение URL для многих «схем URL» (определяемых строкой перед ":" в URL — например, "ftp" является схемой URL "ftp://python.org/"), используя связанные с ними сетевые протоколы (например, FTP, HTTP). В этом учебнике рассматривается наиболее распространенный случай — HTTP.
Для простых ситуаций urlopen очень прост в использовании. Но как только вы столкнетесь с ошибками или нетривиальными случаями при открытии HTTP URL, вам понадобится некоторое понимание протокола HyperText Transfer Protocol. Наиболее полным и авторитетным справочником по HTTP является RFC 2616. Это технический документ, который не предназначен для легкого чтения. Цель этого HOWTO — проиллюстрировать использование urllib, с достаточным количеством подробностей о HTTP, чтобы помочь вам. Он не предназначен для замены документации urllib.request, а является дополнением к ней.
Получение URL-адресов¶
Самый простой способ использования urllib.request следующий:
import urllib.request with urllib.request.urlopen('http://python.org/') as response: html = response.read()
Если вы хотите получить ресурс по URL и сохранить его во временном месте, вы можете сделать это с помощью функций shutil.copyfileobj() и tempfile.NamedTemporaryFile():
import shutil import tempfile import urllib.request with urllib.request.urlopen('http://python.org/') as response: with tempfile.NamedTemporaryFile(delete=False) as tmp_file: shutil.copyfileobj(response, tmp_file) with open(tmp_file.name) as html: pass
Многие случаи использования urllib будут настолько простыми (обратите внимание, что вместо URL „http:“ мы могли бы использовать URL, начинающийся с „ftp:“, „file:“ и т.д.). Однако цель этого руководства — объяснить более сложные случаи, сосредоточившись на HTTP.
HTTP основан на запросах и ответах — клиент делает запросы, а серверы отправляют ответы. urllib.request отражает это с помощью объекта Request, который представляет HTTP-запрос, который вы делаете. В своей простейшей форме вы создаете объект Request, который определяет URL, который вы хотите получить. Вызов urlopen с этим объектом Request возвращает объект response для запрошенного URL. Этот ответ является файлоподобным объектом, что означает, что вы можете, например, вызвать .read() на ответе:
import urllib.request req = urllib.request.Request('http://www.voidspace.org.uk') with urllib.request.urlopen(req) as response: the_page = response.read()
Обратите внимание, что urllib.request использует один и тот же интерфейс Request для работы со всеми схемами URL. Например, вы можете сделать FTP-запрос следующим образом:
req = urllib.request.Request('ftp://example.com/')
В случае с HTTP есть две дополнительные вещи, которые позволяют делать объекты Request: Во-первых, вы можете передавать данные для отправки на сервер. Во-вторых, вы можете передать серверу дополнительную информацию («метаданные») о данных или о самом запросе — эта информация передается в виде «заголовков» HTTP. Давайте рассмотрим каждый из них по очереди.
Данные¶
Иногда вы хотите отправить данные на URL (часто URL ссылается на CGI (Common Gateway Interface) скрипт или другое веб-приложение). В HTTP это часто делается с помощью так называемого POST запроса. Именно так часто поступает ваш браузер, когда вы отправляете HTML-форму, заполненную в Интернете. Не все POST-запросы должны исходить от форм: вы можете использовать POST для передачи произвольных данных в ваше собственное приложение. В обычном случае HTML-формы данные должны быть закодированы стандартным образом, а затем переданы объекту Request в качестве аргумента data. Кодирование выполняется с помощью функции из библиотеки urllib.parse.
import urllib.parse import urllib.request url = 'http://www.someserver.com/cgi-bin/register.cgi' values = {'name' : 'Michael Foord', 'location' : 'Northampton', 'language' : 'Python' } data = urllib.parse.urlencode(values) data = data.encode('ascii') # data should be bytes req = urllib.request.Request(url, data) with urllib.request.urlopen(req) as response: the_page = response.read()
Обратите внимание, что иногда требуются другие кодировки (например, для загрузки файлов из HTML-форм — подробнее см. в HTML Specification, Form Submission).
Если вы не передаете аргумент data, urllib использует GET запрос. Различия между GET и POST запросами заключаются в том, что POST запросы часто имеют «побочные эффекты»: они каким-то образом изменяют состояние системы (например, размещая на сайте заказ на доставку сотни килограммов консервированного спама к вашей двери). Хотя стандарт HTTP четко указывает, что POST-запросы должны всегда вызывать побочные эффекты, а GET-запросы никогда не вызывать побочных эффектов, ничто не мешает GET-запросу иметь побочные эффекты, а POST-запросу — не иметь побочных эффектов. Данные также могут быть переданы в запросе HTTP GET путем кодирования их в самом URL.
Это делается следующим образом:
>>> import urllib.request >>> import urllib.parse >>> data = {} >>> data['name'] = 'Somebody Here' >>> data['location'] = 'Northampton' >>> data['language'] = 'Python' >>> url_values = urllib.parse.urlencode(data) >>> print(url_values) # The order may differ from below. name=Somebody+Here&language=Python&location=Northampton >>> url = 'http://www.example.com/example.cgi' >>> full_url = url + '?' + url_values >>> data = urllib.request.urlopen(full_url)
Обратите внимание, что полный URL создается путем добавления ? к URL, за которым следуют закодированные значения.
Обработка исключений¶
urlopen поднимает URLError, когда не может обработать ответ (хотя, как обычно в Python API, могут быть подняты и встроенные исключения, такие как ValueError, TypeError и т.д.).
HTTPError является подклассом URLError, возникающим в конкретном случае HTTP URL.
Классы исключений экспортируются из модуля urllib.error.
URLError¶
Часто URLError возникает из-за отсутствия сетевого соединения (нет маршрута к указанному серверу), или указанный сервер не существует. В этом случае вызванное исключение будет иметь атрибут „reason“, который представляет собой кортеж, содержащий код ошибки и текстовое сообщение об ошибке.
например
>>> req = urllib.request.Request('http://www.pretend_server.org') >>> try: urllib.request.urlopen(req) ... except urllib.error.URLError as e: ... print(e.reason) ... (4, 'getaddrinfo failed')
HTTPError¶
Каждый ответ HTTP от сервера содержит числовой «код состояния». Иногда код состояния указывает на то, что сервер не может выполнить запрос. Обработчики по умолчанию будут обрабатывать некоторые из этих ответов за вас (например, если ответ является «перенаправлением», которое просит клиента получить документ с другого URL, urllib обработает это за вас). Если urlopen не может обработать эти ответы, он выдаст ошибку HTTPError. Типичные ошибки включают „404“ (страница не найдена), „403“ (запрос запрещен) и „401“ (требуется аутентификация).
См. раздел 10 книги RFC 2616 для справки по всем кодам ошибок HTTP.
Поднятый экземпляр HTTPError будет иметь целочисленный атрибут „code“, который соответствует ошибке, отправленной сервером.
Коды ошибок¶
Поскольку обработчики по умолчанию обрабатывают перенаправления (коды в диапазоне 300), а коды в диапазоне 100-299 означают успех, вы обычно будете видеть только коды ошибок в диапазоне 400-599.
http.server.BaseHTTPRequestHandler.responses представляет собой полезный словарь кодов ответов, в котором показаны все коды ответов, используемые RFC 2616. Словарь воспроизводится здесь для удобства
# Table mapping response codes to messages; entries have the # form {code: (shortmessage, longmessage)}. responses = { 100: ('Continue', 'Request received, please continue'), 101: ('Switching Protocols', 'Switching to new protocol; obey Upgrade header'), 200: ('OK', 'Request fulfilled, document follows'), 201: ('Created', 'Document created, URL follows'), 202: ('Accepted', 'Request accepted, processing continues off-line'), 203: ('Non-Authoritative Information', 'Request fulfilled from cache'), 204: ('No Content', 'Request fulfilled, nothing follows'), 205: ('Reset Content', 'Clear input form for further input.'), 206: ('Partial Content', 'Partial content follows.'), 300: ('Multiple Choices', 'Object has several resources -- see URI list'), 301: ('Moved Permanently', 'Object moved permanently -- see URI list'), 302: ('Found', 'Object moved temporarily -- see URI list'), 303: ('See Other', 'Object moved -- see Method and URL list'), 304: ('Not Modified', 'Document has not changed since given time'), 305: ('Use Proxy', 'You must use proxy specified in Location to access this ' 'resource.'), 307: ('Temporary Redirect', 'Object moved temporarily -- see URI list'), 400: ('Bad Request', 'Bad request syntax or unsupported method'), 401: ('Unauthorized', 'No permission -- see authorization schemes'), 402: ('Payment Required', 'No payment -- see charging schemes'), 403: ('Forbidden', 'Request forbidden -- authorization will not help'), 404: ('Not Found', 'Nothing matches the given URI'), 405: ('Method Not Allowed', 'Specified method is invalid for this server.'), 406: ('Not Acceptable', 'URI not available in preferred format.'), 407: ('Proxy Authentication Required', 'You must authenticate with ' 'this proxy before proceeding.'), 408: ('Request Timeout', 'Request timed out; try again later.'), 409: ('Conflict', 'Request conflict.'), 410: ('Gone', 'URI no longer exists and has been permanently removed.'), 411: ('Length Required', 'Client must specify Content-Length.'), 412: ('Precondition Failed', 'Precondition in headers is false.'), 413: ('Request Entity Too Large', 'Entity is too large.'), 414: ('Request-URI Too Long', 'URI is too long.'), 415: ('Unsupported Media Type', 'Entity body in unsupported format.'), 416: ('Requested Range Not Satisfiable', 'Cannot satisfy request range.'), 417: ('Expectation Failed', 'Expect condition could not be satisfied.'), 500: ('Internal Server Error', 'Server got itself in trouble'), 501: ('Not Implemented', 'Server does not support this operation'), 502: ('Bad Gateway', 'Invalid responses from another server/proxy.'), 503: ('Service Unavailable', 'The server cannot process the request due to a high load'), 504: ('Gateway Timeout', 'The gateway server did not receive a timely response'), 505: ('HTTP Version Not Supported', 'Cannot fulfill request.'), }
При возникновении ошибки сервер возвращает код ошибки HTTP и страницу ошибки. Вы можете использовать экземпляр HTTPError в качестве ответа на возвращаемой странице. Это означает, что помимо атрибута code, он также имеет методы read, geturl и info, возвращаемые модулем urllib.response:
>>> req = urllib.request.Request('http://www.python.org/fish.html') >>> try: ... urllib.request.urlopen(req) ... except urllib.error.HTTPError as e: ... print(e.code) ... print(e.read()) ... 404 b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">nnn<html ... <title>Page Not Found</title>n ...
Подведение итогов¶
Итак, если вы хотите быть готовым к HTTPError или URLError, есть два основных подхода. Я предпочитаю второй подход.
Номер 1¶
from urllib.request import Request, urlopen from urllib.error import URLError, HTTPError req = Request(someurl) try: response = urlopen(req) except HTTPError as e: print('The server couldn't fulfill the request.') print('Error code: ', e.code) except URLError as e: print('We failed to reach a server.') print('Reason: ', e.reason) else: # everything is fine
Примечание
except HTTPError должен быть первым, иначе except URLError будет также ловить HTTPError.
Номер 2¶
from urllib.request import Request, urlopen from urllib.error import URLError req = Request(someurl) try: response = urlopen(req) except URLError as e: if hasattr(e, 'reason'): print('We failed to reach a server.') print('Reason: ', e.reason) elif hasattr(e, 'code'): print('The server couldn't fulfill the request.') print('Error code: ', e.code) else: # everything is fine
info и geturl¶
Ответ, возвращаемый urlopen (или экземпляр HTTPError), имеет два полезных метода info() и geturl() и определен в модуле urllib.response.
geturl — возвращает реальный URL полученной страницы. Это полезно, поскольку urlopen (или используемый объект opener) может следовать за перенаправлением. URL полученной страницы может не совпадать с запрошенным URL.
info — возвращает словареподобный объект, описывающий полученную страницу, в частности, заголовки, отправленные сервером. В настоящее время это экземпляр http.client.HTTPMessage.
Типичные заголовки включают „Content-length“, „Content-type“ и так далее. Смотрите Quick Reference to HTTP Headers для полезного списка заголовков HTTP с краткими объяснениями их значения и использования.
Открыватели и манипуляторы¶
Когда вы получаете URL, вы используете открыватель (экземпляр, возможно, сбивающего с толку названия urllib.request.OpenerDirector). Обычно мы используем открывалку по умолчанию — через urlopen, но вы можете создавать собственные открывалки. Открыватели используют обработчики. Всю «тяжелую работу» выполняют обработчики. Каждый обработчик знает, как открывать URL для определенной схемы URL (http, ftp и т.д.), или как обрабатывать тот или иной аспект открытия URL, например, HTTP-перенаправления или HTTP-куки.
Вы захотите создать открыватели, если вам нужно получить URL с установленными определенными обработчиками, например, чтобы получить открыватель, который обрабатывает cookies, или чтобы получить открыватель, который не обрабатывает перенаправления.
Чтобы создать открывалку, инстанцируйте OpenerDirector, а затем многократно вызывайте .add_handler(some_handler_instance).
В качестве альтернативы можно использовать build_opener, которая является удобной функцией для создания объектов-открывателей с помощью одного вызова функции. build_opener добавляет несколько обработчиков по умолчанию, но предоставляет быстрый способ добавить больше и/или переопределить обработчики по умолчанию.
Другие виды обработчиков, которые вам могут понадобиться, могут обрабатывать прокси, аутентификацию и другие распространенные, но немного специализированные ситуации.
install_opener можно использовать для того, чтобы сделать объект opener открывателем (глобальным) по умолчанию. Это означает, что вызовы urlopen будут использовать установленный вами открыватель.
Объекты Opener имеют метод open, который можно вызвать непосредственно для получения урлов таким же образом, как и функцию urlopen: нет необходимости вызывать install_opener, кроме как для удобства.
Базовая аутентификация¶
Для иллюстрации создания и установки обработчика мы будем использовать HTTPBasicAuthHandler. Для более подробного обсуждения этой темы — включая объяснение того, как работает базовая аутентификация — смотрите Basic Authentication Tutorial.
Когда требуется аутентификация, сервер отправляет заголовок (а также код ошибки 401) с запросом на аутентификацию. В нем указывается схема аутентификации и «область». Заголовок выглядит следующим образом: WWW-Authenticate: SCHEME realm="REALM".
например.
WWW-Authenticate: Basic realm="cPanel Users"
Затем клиент должен повторить запрос с соответствующими именем и паролем для сферы, включенными в качестве заголовка в запрос. Это и есть «базовая аутентификация». Чтобы упростить этот процесс, мы можем создать экземпляр HTTPBasicAuthHandler и открывающее устройство для использования этого обработчика.
HTTPBasicAuthHandler использует объект, называемый менеджером паролей, для обработки сопоставления URL и царств с паролями и именами пользователей. Если вы знаете, что такое царство (из заголовка аутентификации, отправленного сервером), то вы можете использовать HTTPPasswordMgr. Часто бывает так, что человеку не важно, что такое царство. В этом случае удобно использовать HTTPPasswordMgrWithDefaultRealm. Это позволяет указать имя пользователя и пароль по умолчанию для URL. Они будут заданы в том случае, если вы не укажете альтернативную комбинацию для конкретного царства. Мы указываем это, предоставляя None в качестве аргумента realm для метода add_password.
URL верхнего уровня — это первый URL, который требует аутентификации. URL «глубже», чем URL, который вы передаете в .add_password(), также будут соответствовать.
# create a password manager password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm() # Add the username and password. # If we knew the realm, we could use it instead of None. top_level_url = "http://example.com/foo/" password_mgr.add_password(None, top_level_url, username, password) handler = urllib.request.HTTPBasicAuthHandler(password_mgr) # create "opener" (OpenerDirector instance) opener = urllib.request.build_opener(handler) # use the opener to fetch a URL opener.open(a_url) # Install the opener. # Now all calls to urllib.request.urlopen use our opener. urllib.request.install_opener(opener)
Примечание
В приведенном выше примере мы только передали наш HTTPBasicAuthHandler в build_opener. По умолчанию открыватели имеют обработчики для обычных ситуаций – ProxyHandler (если установлена прокси-настройка, такая как переменная окружения http_proxy), UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, DataHandler, HTTPErrorProcessor.
top_level_url фактически является либо полным URL (включая компонент схемы „http:“, имя хоста и, опционально, номер порта), например "http://example.com/" или «авторитетом» (т.е. именем хоста, опционально включая номер порта), например "example.com" или "example.com:8080" (последний пример включает номер порта). Авторитет, если он присутствует, НЕ должен содержать компонент «userinfo» — например, "joe:password@example.com" не является корректным.
Прокси¶
urllib будет автоматически определять ваши настройки прокси и использовать их. Это происходит через ProxyHandler, который является частью обычной цепочки обработчиков при обнаружении настроек прокси. Обычно это хорошо, но бывают случаи, когда это может быть бесполезно 5. Один из способов сделать это — установить собственный ProxyHandler, без определенных прокси. Это делается с помощью шагов, аналогичных настройке обработчика Basic Authentication:
>>> proxy_support = urllib.request.ProxyHandler({}) >>> opener = urllib.request.build_opener(proxy_support) >>> urllib.request.install_opener(opener)
Примечание
В настоящее время urllib.request не поддерживает получение расположений https через прокси. Однако это можно сделать, расширив urllib.request, как показано в рецепте 6.
Примечание
HTTP_PROXY будет игнорироваться, если установлена переменная REQUEST_METHOD; см. документацию по getproxies().
Сокеты и слои¶
Поддержка Python для получения ресурсов из Интернета многоуровневая. urllib использует библиотеку http.client, которая, в свою очередь, использует библиотеку сокетов.
Начиная с версии Python 2.3 вы можете указать, как долго сокет должен ждать ответа, прежде чем завершится тайминг. Это может быть полезно в приложениях, которые должны получать веб-страницы. По умолчанию модуль сокета имеет без таймаута и может зависнуть. В настоящее время таймаут сокета не раскрывается на уровнях http.client или urllib.request. Однако вы можете установить таймаут по умолчанию глобально для всех сокетов, используя
import socket import urllib.request # timeout in seconds timeout = 10 socket.setdefaulttimeout(timeout) # this call to urllib.request.urlopen now uses the default timeout # we have set in the socket module req = urllib.request.Request('http://www.voidspace.org.uk') response = urllib.request.urlopen(req)
Сноски¶
Этот документ был проверен и отредактирован Джоном Ли.
- 1
-
Например, Google.
- 2
-
Браузерный сниффинг — это очень плохая практика для дизайна сайтов — создание сайтов с использованием веб-стандартов гораздо более разумно. К сожалению, многие сайты все еще отправляют разные версии в разные браузеры.
- 3
-
Агент пользователя для MSIE 6 — „Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)“.
- 4
-
Для получения подробной информации о других заголовках HTTP-запросов смотрите Quick Reference to HTTP Headers.
- 5
-
В моем случае я вынужден использовать прокси-сервер для доступа в Интернет на работе. Если вы пытаетесь получить URL localhost через этот прокси, он блокирует их. IE настроен на использование прокси, и urllib это замечает. Чтобы протестировать скрипты с сервером localhost, я должен запретить urllib использовать прокси.
- 6
-
urllib-открывалка для SSL-прокси (метод CONNECT): ASPN Cookbook Recipe.
Отправка HTTP POST и GET запросов. Получение данных из URL-адресов.
Модуль urllib.request определяет функции и классы, которые помогают открывать URL-адреса (в основном HTTP), имеет базовую и дайджест-аутентификацию, перенаправления, файлы cookie и многое другое.
Модуль urllib.request предлагает очень простой интерфейс в виде функции urllib.request.urlopen(), что позволяет получать URL-адреса с использованием множества различных протоколов. Также этот модуль предлагает немного более сложный интерфейс для обработки обычных ситуаций, таких как базовая аутентификация, файлы cookie, прокси и т. д. Они предоставляются объектами, называемыми обработчиками и «открывателями«.
Модуль поддерживает получение для многих схем URL-адресов (идентифицируемых строкой перед : в URL-адресе — например, ftp: — это схема URL-адресов ftp://python.org/) с использованием связанной с ними сети.
В простых ситуациях функция urllib.request.urlopen() очень проста в использовании. При появлении ошибок или нетривиальных случаев при открытии URL-адресов, необходимое некоторое понимание протокола передачи гипертекста. Цель этого вводного материала — проиллюстрировать использование модуля urllib.request с достаточно подробным описанием HTTP.
Так же, рассмотрите возможность использования в своем коде, написанного для людей, стороннего модуля HTTP запросов requests (требует дополнительной установки).
Содержание:
- Получение данных с сайта по URL-адресу;
- Отправка POST запросов модулем
urllib.request; - Отправка GET запросов модулем
urllib.request; - Отправка заголовков
Headerмодулемurllib.request; - Обработка исключений при использовании
urllib.request.
Получение данных с сайта по URL-адресу.
Самый простой способ использовать модуль urllib.request следующий:
import urllib.request with urllib.request.urlopen('http://python.org/') as resp: html = resp.read()
import shutil import tempfile import urllib.request with urllib.request.urlopen('http://python.org/') as response: with tempfile.NamedTemporaryFile(delete=False) as tmp_file: shutil.copyfileobj(response, tmp_file) with open(tmp_file.name) as html: pass
Многие варианты использования urllib.request будут такими простыми (обратите внимание, что вместо URL-адреса http: можно использовать URL-адрес, начинающийся с ftp:, file: и т. д.
HTTP основан на запросах и ответах — клиент делает запросы, а серверы отправляют ответы. Модуль urllib.request отражает это с помощью объекта Request, который представляет собой HTTP-запрос, который делается. В простейшей форме — создается объект Request с указанием URL-адреса, который надо получить получить. Вызов функции urllib.request.urlopen() с объектом Request возвращает объект ответа для запрошенного URL. Этот ответ является файловым объектом, для чтения которого можно вызвать, например, метод .read():
import urllib.request req = urllib.request.Request('http://www.voidspace.org.uk') with urllib.request.urlopen(req) as response: the_page = response.read()
, что модуль urllib.request использует один и тот же интерфейс запроса для обработки всех схем URL. Например, можно сделать такой FTP-запрос:
req = urllib.request.Request('ftp://example.com/')
В случае с HTTP есть две дополнительные вещи, которые позволяют создавать объекты Request:
- можно передавать данные для отправки на сервер.
- можно передать на сервер дополнительную информацию — метаданные о самом запросе — эта информация отправляется в виде заголовков HTTP.
Создание и отправка POST запросов модулем urllib.request.
Иногда необходимо отправить данные по URL-адресу (часто URL-адрес будет относиться к сценарию CGI или другому веб-приложению). В HTTP это делается с помощью так называемого запроса POST запроса. Это то, что делает браузер, когда отправляется HTML-форма. Не все POST запросы поступают из заполненных форм, которые заполняются пользователями на сайтах. POST запросы могут использоваться для передачи произвольных данных в собственное приложение. В общем случае HTML-форм данные должны быть закодированы стандартным способом, а затем переданы объекту Request в качестве аргумента данных. Кодирование выполняется с помощью функции из модуля urllib.parse.
import urllib.parse import urllib.request # URL - адрес, получающий данные url = 'http://www.someserver.com/cgi-bin/register.cgi' # Отправляемые данные values = {'name' : 'Michael Foord', 'location' : 'Northampton', 'language' : 'Python' } # Кодирование данных data = urllib.parse.urlencode(values) # данные должны быть байтами data = data.encode('utf-8') req = urllib.request.Request(url, data) with urllib.request.urlopen(req) as response: the_page = response.read()
Обратите внимание, что иногда, для кодирования данных требуется другая кодировка.
Создание и отправка GET запросов модулем urllib.request.
Если объект Request не получает аргумент с данными, то модуль urllib.request использует GET запрос. Одним из отличий запросов GET и POST является то, что запросы POST часто имеют побочные эффекты, они каким-то образом меняют состояние системы.
Произвольные данные также можно передать в GET-запросе, закодировав их в самом URL. Это делается следующим образом:
>>> import urllib.request >>> import urllib.parse >>> data = {} >>> data['name'] = 'Somebody Here' >>> data['location'] = 'Northampton' >>> data['language'] = 'Python' >>> url_values = urllib.parse.urlencode(data) # Порядок может отличаться от приведенного ниже. >>> print(url_values) # name=Somebody+Here&language=Python&location=Northampton >>> url = 'http://www.example.com/example.cgi' >>> full_url = '?'.join([url, url_values]) >>> data = urllib.request.urlopen(full_url)
, что полный URL-адрес создается путем добавления '?' к URL-адресу, за которым следуют закодированные значения.
Отправка заголовков HEADERS при помощи urllib.request.
Некоторым веб-сайтам не нравится, когда их просматривают различные роботы или сканеры, к тому же, по умолчанию модуль urllib.request идентифицирует себя как Python-urllib/x.y (где x и y — это старший и дополнительный номера версий версии Python, например Python-urllib/3.9). Браузер идентифицирует себя через заголовок User-Agent. При создании объекта Request, ему можно передать словарь заголовков. В следующем примере выполняется тот же запрос, что и выше, но идентифицируется как версия Internet Explorer .
import urllib.parse import urllib.request url = 'http://www.someserver.com/cgi-bin/register.cgi' user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' values = {'name': 'Michael Foord', 'location': 'Northampton', 'language': 'Python' } headers = {'User-Agent': user_agent} data = urllib.parse.urlencode(values) data = data.encode('ascii') # создание объекта Request с указанием, в качестве # аргументов, параметров запроса и заголовков req = urllib.request.Request(url, data, headers) # отправка POST запроса with urllib.request.urlopen(req) as response: the_page = response.read()
Обработка исключений при использовании urllib.request.
HTTPError — это подкласс URLError, возникающий в конкретном случае при запросе URL-адресов по HTTP протоколу.
Классы исключений экспортируются из модуля urllib.error.
Обработка ошибок URLError.
Часто URLError возникает из-за отсутствия сетевого подключения (нет маршрута к указанному серверу) или из-за того, что указанный сервер не существует. В этом случае возникшее исключение будет иметь атрибут, который представляет собой кортеж, содержащий код ошибки и текстовое сообщение об ошибке.
>>> req = urllib.request.Request('http://www.pretend_server.org') >>> try: urllib.request.urlopen(req) ... except urllib.error.URLError as e: ... print(e.reason) ... # (4, 'getaddrinfo failed')
Обработка ошибок HTTPError.
Каждый ответ HTTP от сервера содержит числовой код ответа сервера. Иногда код этот состояния указывает на то, что сервер не может выполнить запрос. Обработчики по умолчанию будут обрабатывать некоторые из этих ответов за вас (например, если ответ представляет собой перенаправление, которое запрашивает у клиента выборку документа с другого URL-адреса, то модуль urllib.request сделает это за вас). Для тех URL-адресов, с которыми модуль не может справиться, он вызывает ошибки HTTPError. Типичные ошибки включают 404 (страница не найдена), 403 (запрос запрещен) и 401 (требуется аутентификация).
Возникший экземпляр HTTPError будет иметь целочисленный атрибут code, который соответствует ошибке, отправленной сервером. Поскольку обработчики по умолчанию обрабатывают перенаправления (коды в диапазоне 300), а коды в диапазоне 100–299 указывают на успех, то необходимо обрабатывать только коды ошибок в диапазоне 400–599.
Когда возникает ошибка, сервер отвечает, возвращая код ошибки HTTP и страницу с ошибкой. Можете использовать экземпляр HTTPError в качестве ответа на возвращенной странице. Это означает, что помимо атрибута code он также имеет методы .read(), .geturl() и .info(), возвращаемые модулем urllib.response:
>>> req = urllib.request.Request('http://www.python.org/fish.html') >>> try: ... urllib.request.urlopen(req) ... except urllib.error.HTTPError as e: ... print(e.code) ... print(e.read()) ... # 404 # b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" # "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">nnn<html # ... # <title>Page Not Found</title>n # ...
Итак, если необходимо обрабатывать ошибки HTTPError или URLError, есть два основных подхода. Мы предпочитаем второй подход.
from urllib.request import Request, urlopen from urllib.error import URLError, HTTPError req = Request(someurl) try: response = urlopen(req) except HTTPError as e: print('The server couldn't fulfill the request.') print('Error code: ', e.code) except URLError as e: print('We failed to reach a server.') print('Reason: ', e.reason) else: # Все в порядке
Примечание. Исключение HTTPError должно быть первым, иначе исключение URLError также перехватит HTTPError.
from urllib.request import Request, urlopen from urllib.error import URLError req = Request(someurl) try: response = urlopen(req) except URLError as e: if hasattr(e, 'reason'): print('We failed to reach a server.') print('Reason: ', e.reason) elif hasattr(e, 'code'): print('The server couldn't fulfill the request.') print('Error code: ', e.code) else: # все в порядке