The urllib module can be used to make an HTTP request from a site, unlike the requests library, which is a built-in library. This reduces dependencies. In the following article, we will discuss why urllib.error.httperror: http error 403: forbidden occurs and how to resolve it.
What is a 403 error?
The 403 error pops up when a user tries to access a forbidden page or, in other words, the page they aren’t supposed to access. 403 is the HTTP status code that the webserver uses to denote the kind of problem that has occurred on the user or the server end. For instance, 200 is the status code for – ‘everything has worked as expected, no errors’. You can go through the other HTTP status code from here.
ModSecurity is a module that protects websites from foreign attacks. It checks whether the requests are being made from a user or from an automated bot. It blocks requests from known spider/bot agents who are trying to scrape the site. Since the urllib library uses something like python urllib/3.3.0 hence, it is easily detected as non-human and therefore gets blocked by mod security.
from urllib import request from urllib.request import Request, urlopen url = "https://www.gamefaqs.com" request_site = Request(url) webpage = urlopen(request_site).read() print(webpage[:200])
ModSecurity blocks the request and returns an HTTP error 403: forbidden error if the request was made without a valid user agent. A user-agent is a header that permits a specific string which in turn allows network protocol peers to identify the following:
- The Operating System, for instance, Windows, Linux, or macOS.
- Websserver’s browser
Moreover, the browser sends the user agent to each and every website that you get connected to. The user-Agent field is included in the HTTP header when the browser gets connected to a website. The header field data differs for each browser.
Why do sites use security that sends 403 responses?
According to a survey, more than 50% of internet traffic comes from automated sources. Automated sources can be scrapers or bots. Therefore it gets necessary to prevent these attacks. Moreover, scrapers tend to send multiple requests, and sites have some rate limits. The rate limit dictates how many requests a user can make. If the user(here scraper) exceeds it, it gets some kind of error, for instance, urllib.error.httperror: http error 403: forbidden.
Resolving urllib.error.httperror: http error 403: forbidden?
This error is caused due to mod security detecting the scraping bot of the urllib and blocking it. Therefore, in order to resolve it, we have to include user-agent/s in our scraper. This will ensure that we can safely scrape the website without getting blocked and running across an error. Let’s take a look at two ways to avoid urllib.error.httperror: http error 403: forbidden.
Method 1: using user-agent
from urllib import request
from urllib.request import Request, urlopen
url = "https://www.gamefaqs.com"
request_site = Request(url, headers={"User-Agent": "Mozilla/5.0"})
webpage = urlopen(request_site).read()
print(webpage[:500])
- In the code above, we have added a new parameter called headers which has a user-agent Mozilla/5.0. Details about the user’s device, OS, and browser are given by the webserver by the user-agent string. This prevents the bot from being blocked by the site.
- For instance, the user agent string gives information to the server that you are using Brace browser and Linux OS on your computer. Thereafter, the server accordingly sends the information.
Method 2: using Session object
There are times when even using user-agent won’t prevent the urllib.error.httperror: http error 403: forbidden. Then we can use the Session object of the request module. For instance:
from random import seed
import requests
url = "https://www.gamefaqs.com"
session_obj = requests.Session()
response = session_obj.get(url, headers={"User-Agent": "Mozilla/5.0"})
print(response.status_code)
The site could be using cookies as a defense mechanism against scraping. It’s possible the site is setting and requesting cookies to be echoed back as a defense against scraping, which might be against its policy.
The Session object is compatible with cookies.
Catching urllib.error.httperror
urllib.error.httperror can be caught using the try-except method. Try-except block can capture any exception, which can be hard to debug. For instance, it can capture exceptions like SystemExit and KeyboardInterupt. Let’s see how can we do this, for instance:
from urllib.request import Request, urlopen
from urllib.error import HTTPError
url = "https://www.gamefaqs.com"
try:
request_site = Request(url)
webpage = urlopen(request_site).read()
print(webpage[:500])
except HTTPError as e:
print("Error occured!")
print(e)
![[Resolved] NameError: Name _mysql is Not Defined](https://www.pythonpool.com/wp-content/uploads/2023/01/nameerror-name-_mysql-is-not-defined-300x157.webp)
FAQs
How to fix the 403 error in the browser?
You can try the following steps in order to resolve the 403 error in the browser try refreshing the page, rechecking the URL, clearing the browser cookies, check your user credentials.
Why do scraping modules often get 403 errors?
Scrapers often don’t use headers while requesting information. This results in their detection by the mod security. Hence, scraping modules often get 403 errors.
Conclusion
In this article, we covered why and when urllib.error.httperror: http error 403: forbidden it can occur. Moreover, we looked at the different ways to resolve the error. We also covered the handling of this error.
Trending Now
-
“Other Commands Don’t Work After on_message” in Discord Bots
●February 5, 2023
-
Botocore.Exceptions.NoCredentialsError: Unable to Locate Credentials
by Rahul Kumar Yadav●February 5, 2023
-
[Resolved] NameError: Name _mysql is Not Defined
by Rahul Kumar Yadav●February 5, 2023
-
Best Ways to Implement Regex New Line in Python
by Rahul Kumar Yadav●February 5, 2023
- the
urllibModule in Python - Check
robots.txtto PreventurllibHTTP Error 403 Forbidden Message - Adding Cookie to the Request Headers to Solve
urllibHTTP Error 403 Forbidden Message - Use Session Object to Solve
urllibHTTP Error 403 Forbidden Message
Today’s article explains how to deal with an error message (exception), urllib.error.HTTPError: HTTP Error 403: Forbidden, produced by the error class on behalf of the request classes when it faces a forbidden resource.
the urllib Module in Python
The urllib Python module handles URLs for python via different protocols. It is famous for web scrapers who want to obtain data from a particular website.
The urllib contains classes, methods, and functions that perform certain operations such as reading, parsing URLs, and robots.txt. There are four classes, request, error, parse, and robotparser.
Check robots.txt to Prevent urllib HTTP Error 403 Forbidden Message
When using the urllib module to interact with clients or servers via the request class, we might experience specific errors. One of those errors is the HTTP 403 error.
We get urllib.error.HTTPError: HTTP Error 403: Forbidden error message in urllib package while reading a URL. The HTTP 403, the Forbidden Error, is an HTTP status code that indicates that the client or server forbids access to a requested resource.
Therefore, when we see this kind of error message, urllib.error.HTTPError: HTTP Error 403: Forbidden, the server understands the request but decides not to process or authorize the request that we sent.
To understand why the website we are accessing is not processing our request, we need to check an important file, robots.txt. Before web scraping or interacting with a website, it is often advised to review this file to know what to expect and not face any further troubles.
To check it on any website, we can follow the format below.
https://<website.com>/robots.txt
For example, check YouTube, Amazon, and Google robots.txt files.
https://www.youtube.com/robots.txt
https://www.amazon.com/robots.txt
https://www.google.com/robots.txt
Checking YouTube robots.txt gives the following result.
# robots.txt file for YouTube
# Created in the distant future (the year 2000) after
# the robotic uprising of the mid-'90s wiped out all humans.
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /channel/*/community
Disallow: /comment
Disallow: /get_video
Disallow: /get_video_info
Disallow: /get_midroll_info
Disallow: /live_chat
Disallow: /login
Disallow: /results
Disallow: /signup
Disallow: /t/terms
Disallow: /timedtext_video
Disallow: /user/*/community
Disallow: /verify_age
Disallow: /watch_ajax
Disallow: /watch_fragments_ajax
Disallow: /watch_popup
Disallow: /watch_queue_ajax
Sitemap: https://www.youtube.com/sitemaps/sitemap.xml
Sitemap: https://www.youtube.com/product/sitemap.xml
We can notice a lot of Disallow tags there. This Disallow tag shows the website’s area, which is not accessible. Therefore, any request to those areas will not be processed and is forbidden.
In other robots.txt files, we might see an Allow tag. For example, http://youtube.com/comment is forbidden to any external request, even with the urllib module.
Let’s write code to scrape data from a website that returns an HTTP 403 error when accessed.
Example Code:
import urllib.request
import re
webpage = urllib.request.urlopen('https://www.cmegroup.com/markets/products.html?redirect=/trading/products/#cleared=Options&sortField=oi').read()
findrows = re.compile('<tr class="- banding(?:On|Off)>(.*?)</tr>')
findlink = re.compile('<a href =">(.*)</a>')
row_array = re.findall(findrows, webpage)
links = re.findall(findlink, webpage)
print(len(row_array))
Output:
Traceback (most recent call last):
File "c:UsersakinlDocumentsPythonindex.py", line 7, in <module>
webpage = urllib.request.urlopen('https://www.cmegroup.com/markets/products.html?redirect=/trading/products/#cleared=Options&sortField=oi').read()
File "C:Python310liburllibrequest.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "C:Python310liburllibrequest.py", line 525, in open
response = meth(req, response)
File "C:Python310liburllibrequest.py", line 634, in http_response
response = self.parent.error(
File "C:Python310liburllibrequest.py", line 563, in error
return self._call_chain(*args)
File "C:Python310liburllibrequest.py", line 496, in _call_chain
result = func(*args)
File "C:Python310liburllibrequest.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
The reason is that we are forbidden from accessing the website. However, if we check the robots.txt file, we will notice that https://www.cmegroup.com/markets/ is not with a Disallow tag. However, if we go down the robots.txt file for the website we wanted to scrape, we will find the below.
User-agent: Python-urllib/1.17
Disallow: /
The above text means that the user agent named Python-urllib is not allowed to crawl any URL within the site. That means using the Python urllib module is not allowed to crawl the site.
Therefore, check or parse the robots.txt to know what resources we have access to. we can parse robots.txt file using the robotparser class. These can prevent our code from experiencing an urllib.error.HTTPError: HTTP Error 403: Forbidden error message.
Passing a valid user agent as a header parameter will quickly fix the problem. The website may use cookies as an anti-scraping measure.
The website may set and ask for cookies to be echoed back to prevent scraping, which is maybe against its policy.
from urllib.request import Request, urlopen
def get_page_content(url, head):
req = Request(url, headers=head)
return urlopen(req)
url = 'https://example.com'
head = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding': 'none',
'Accept-Language': 'en-US,en;q=0.8',
'Connection': 'keep-alive',
'refere': 'https://example.com',
'cookie': """your cookie value ( you can get that from your web page) """
}
data = get_page_content(url, head).read()
print(data)
Output:
<!doctype html>n<html>n<head>n <title>Example Domain</title>nn <meta
'
'
'
<p><a href="https://www.iana.org/domains/example">More information...</a></p>n</div>n</body>n</html>n'
Passing a valid user agent as a header parameter will quickly fix the problem.
Use Session Object to Solve urllib HTTP Error 403 Forbidden Message
Sometimes, even using a user agent won’t stop this error from occurring. The Session object of the requests module can then be used.
from random import seed
import requests
url = "https://stackoverflow.com/search?q=html+error+403"
session_obj = requests.Session()
response = session_obj.get(url, headers={"User-Agent": "Mozilla/5.0"})
print(response.status_code)
Output:
The above article finds the cause of the urllib.error.HTTPError: HTTP Error 403: Forbidden and the solution to handle it. mod_security basically causes this error as different web pages use different security mechanisms to differentiate between human and automated computers (bots).
If you need to make HTTP requests with Python, then you may find yourself directed to the brilliant requests library. Though it’s a great library, you may have noticed that it’s not a built-in part of Python. If you prefer, for whatever reason, to limit your dependencies and stick to standard-library Python, then you can reach for urllib.request!
In this tutorial, you’ll:
- Learn how to make basic HTTP requests with
urllib.request - Dive into the nuts and bolts of an HTTP message and how
urllib.requestrepresents it - Understand how to deal with character encodings of HTTP messages
- Explore some common errors when using
urllib.requestand learn how to resolve them - Dip your toes into the world of authenticated requests with
urllib.request - Understand why both
urlliband therequestslibrary exist and when to use one or the other
If you’ve heard of HTTP requests, including GET and POST, then you’re probably ready for this tutorial. Also, you should’ve already used Python to read and write to files, ideally with a context manager, at least once.
Ultimately, you’ll find that making a request doesn’t have to be a frustrating experience, although it does tend to have that reputation. Many of the issues that you tend to run into are due to the inherent complexity of this marvelous thing called the Internet. The good news is that the urllib.request module can help to demystify much of this complexity.
Basic HTTP GET Requests With urllib.request
Before diving into the deep end of what an HTTP request is and how it works, you’re going to get your feet wet by making a basic GET request to a sample URL. You’ll also make a GET request to a mock REST API for some JSON data. In case you’re wondering about POST Requests, you’ll be covering them later in the tutorial, once you have some more knowledge of urllib.request.
To get started, you’ll make a request to www.example.com, and the server will return an HTTP message. Ensure that you’re using Python 3 or above, and then use the urlopen() function from urllib.request:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> body[:15]
b'<!doctype html>'
In this example, you import urlopen() from urllib.request. Using the context manager with, you make a request and receive a response with urlopen(). Then you read the body of the response and close the response object. With that, you display the first fifteen positions of the body, noting that it looks like an HTML document.
There you are! You’ve successfully made a request, and you received a response. By inspecting the content, you can tell that it’s likely an HTML document. Note that the printed output of the body is preceded by b. This indicates a bytes literal, which you may need to decode. Later in the tutorial, you’ll learn how to turn bytes into a string, write them to a file, or parse them into a dictionary.
The process is only slightly different if you want to make calls to REST APIs to get JSON data. In the following example, you’ll make a request to {JSON}Placeholder for some fake to-do data:
>>>
>>> from urllib.request import urlopen
>>> import json
>>> url = "https://jsonplaceholder.typicode.com/todos/1"
>>> with urlopen(url) as response:
... body = response.read()
...
>>> todo_item = json.loads(body)
>>> todo_item
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}
In this example, you’re doing pretty much the same as in the previous example. But in this one, you import urllib.request and json, using the json.loads() function with body to decode and parse the returned JSON bytes into a Python dictionary. Voila!
If you’re lucky enough to be using error-free endpoints, such as the ones in these examples, then maybe the above is all that you need from urllib.request. Then again, you may find that it’s not enough.
Now, before doing some urllib.request troubleshooting, you’ll first gain an understanding of the underlying structure of HTTP messages and learn how urllib.request handles them. This understanding will provide a solid foundation for troubleshooting many different kinds of issues.
The Nuts and Bolts of HTTP Messages
To understand some of the issues that you may encounter when using urllib.request, you’ll need to examine how a response is represented by urllib.request. To do that, you’ll benefit from a high-level overview of what an HTTP message is, which is what you’ll get in this section.
Before the high-level overview, a quick note on reference sources. If you want to get into the technical weeds, the Internet Engineering Task Force (IETF) has an extensive set of Request for Comments (RFC) documents. These documents end up becoming the actual specifications for things like HTTP messages. RFC 7230, part 1: Message Syntax and Routing, for example, is all about the HTTP message.
If you’re looking for some reference material that’s a bit easier to digest than RFCs, then the Mozilla Developer Network (MDN) has a great range of reference articles. For example, their article on HTTP messages, while still technical, is a lot more digestible.
Now that you know about these essential sources of reference information, in the next section you’ll get a beginner-friendly overview of HTTP messages.
Understanding What an HTTP Message Is
In a nutshell, an HTTP message can be understood as text, transmitted as a stream of bytes, structured to follow the guidelines specified by RFC 7230. A decoded HTTP message can be as simple as two lines:
GET / HTTP/1.1
Host: www.google.com
This specifies a GET request at the root (/) using the HTTP/1.1 protocol. The one and only header required is the host, www.google.com. The target server has enough information to make a response with this information.
A response is similar in structure to a request. HTTP messages have two main parts, the metadata and the body. In the request example above, the message is all metadata with no body. The response, on the other hand, does have two parts:
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
Server: gws
(... other headers ...)
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"
...
The response starts with a status line that specifies the HTTP protocol HTTP/1.1 and the status 200 OK. After the status line, you get many key-value pairs, such as Server: gws, representing all the response headers. This is the metadata of the response.
After the metadata, there’s a blank line, which serves as the divider between the headers and the body. Everything that follows the blank line makes up the body. This is the part that gets read when you’re using urllib.request.
You can assume that all HTTP messages follow these specifications, but it’s possible that some may break these rules or follow an older specification. It’s exceptionally rare for this to cause any issues, though. So, just keep it in the back of your mind in case you run into a strange bug!
In the next section, you’ll see how urllib.request deals with raw HTTP messages.
Understanding How urllib.request Represents an HTTP Message
The main representation of an HTTP message that you’ll be interacting with when using urllib.request is the HTTPResponse object. The urllib.request module itself depends on the low-level http module, which you don’t need to interact with directly. You do end up using some of the data structures that http provides, though, such as HTTPResponse and HTTPMessage.
When you make a request with urllib.request.urlopen(), you get an HTTPResponse object in return. Spend some time exploring the HTTPResponse object with pprint() and dir() to see all the different methods and properties that belong to it:
>>>
>>> from urllib.request import urlopen
>>> from pprint import pprint
>>> with urlopen("https://www.example.com") as response:
... pprint(dir(response))
...
To reveal the output of this code snippet, click to expand the collapsible section below:
['__abstractmethods__',
'__class__',
'__del__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__enter__',
'__eq__',
'__exit__',
'__format__',
'__ge__',
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__iter__',
'__le__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__next__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'_abc_impl',
'_checkClosed',
'_checkReadable',
'_checkSeekable',
'_checkWritable',
'_check_close',
'_close_conn',
'_get_chunk_left',
'_method',
'_peek_chunked',
'_read1_chunked',
'_read_and_discard_trailer',
'_read_next_chunk_size',
'_read_status',
'_readall_chunked',
'_readinto_chunked',
'_safe_read',
'_safe_readinto',
'begin',
'chunk_left',
'chunked',
'close',
'closed',
'code',
'debuglevel',
'detach',
'fileno',
'flush',
'fp',
'getcode',
'getheader',
'getheaders',
'geturl',
'headers',
'info',
'isatty',
'isclosed',
'length',
'msg',
'peek',
'read',
'read1',
'readable',
'readinto',
'readinto1',
'readline',
'readlines',
'reason',
'seek',
'seekable',
'status',
'tell',
'truncate',
'url',
'version',
'will_close',
'writable',
'write',
'writelines']
That’s a lot of methods and properties, but you’ll only end up using a handful of these . Apart from .read(), the important ones usually involve getting information about the headers.
One way to inspect all the headers is to access the .headers attribute of the HTTPResponse object. This will return an HTTPMessage object. Conveniently, you can treat an HTTPMessage like a dictionary by calling .items() on it to get all the headers as tuples:
>>>
>>> with urlopen("https://www.example.com") as response:
... pass
...
>>> response.headers
<http.client.HTTPMessage object at 0x000001E029D9F4F0>
>>> pprint(response.headers.items())
[('Accept-Ranges', 'bytes'),
('Age', '398424'),
('Cache-Control', 'max-age=604800'),
('Content-Type', 'text/html; charset=UTF-8'),
('Date', 'Tue, 25 Jan 2022 12:18:53 GMT'),
('Etag', '"3147526947"'),
('Expires', 'Tue, 01 Feb 2022 12:18:53 GMT'),
('Last-Modified', 'Thu, 17 Oct 2019 07:18:26 GMT'),
('Server', 'ECS (nyb/1D16)'),
('Vary', 'Accept-Encoding'),
('X-Cache', 'HIT'),
('Content-Length', '1256'),
('Connection', 'close')]
Now you have access to all the response headers! You probably won’t need most of this information, but rest assured that some applications do use it. For example, your browser might use the headers to read the response, set cookies, and determine an appropriate cache lifetime.
There are convenience methods to get the headers from an HTTPResponse object because it’s quite a common operation. You can call .getheaders() directly on the HTTPResponse object, which will return exactly the same list of tuples as above. If you’re only interested in one header, say the Server header, then you can use the singular .getheader("Server") on HTTPResponse or use the square bracket ([]) syntax on .headers from HTTPMessage:
>>>
>>> response.getheader("Server")
'ECS (nyb/1D16)'
>>> response.headers["Server"]
'ECS (nyb/1D16)'
Truth be told, you probably won’t need to interact with the headers directly like this. The information that you’re most likely to need will probably already have some built-in helper methods, but now you know, in case you ever need to dig deeper!
Closing an HTTPResponse
The HTTPResponse object has a lot in common with the file object. The HTTPResponse class inherits from the IOBase class, as do file objects, which means that you have to be mindful of opening and closing.
In simple programs, you’re not likely to notice any issues if you forget to close HTTPResponse objects. For more complex projects, though, this can significantly slow execution and cause bugs that are difficult to pinpoint.
Problems arise because input/output (I/O) streams are limited. Each HTTPResponse requires a stream to be kept clear while it’s being read. If you never close your streams, this will eventually prevent any other stream from being opened, and it might interfere with other programs or even your operating system.
So, make sure you close your HTTPResponse objects! For your convenience, you can use a context manager, as you’ve seen in the examples. You can also achieve the same result by explicitly calling .close() on the response object:
>>>
>>> from urllib.request import urlopen
>>> response = urlopen("https://www.example.com")
>>> body = response.read()
>>> response.close()
In this example, you don’t use a context manager, but instead close the response stream explicitly. The above example still has an issue, though, because an exception may be raised before the call to .close(), preventing the proper teardown. To make this call unconditional, as it should be, you can use a try … except block with both an else and a finally clause:
>>>
>>> from urllib.request import urlopen
>>> response = None
>>> try:
... response = urlopen("https://www.example.com")
... except Exception as ex:
... print(ex)
... else:
... body = response.read()
... finally:
... if response is not None:
... response.close()
In this example, you achieve an unconditional call to .close() by using the finally block, which will always run regardless of exceptions raised. The code in the finally block first checks if the response object exists with is not None, and then closes it.
That said, this is exactly what a a context manager does, and the with syntax is generally preferred. Not only is the with syntax less verbose and more readable, but it also protects you from pesky errors of omission. Put another way, it’s a far better guard against accidentally forgetting to close the object:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... response.read(50)
... response.read(50)
...
b'<!doctype html>n<html>n<head>n <title>Example D'
b'omain</title>nn <meta charset="utf-8" />n <m'
In this example, you import urlopen() from the urllib.request module. You use the with keyword with .urlopen() to assign the HTTPResponse object to the variable response. Then, you read the first fifty bytes of the response and then read the following fifty bytes, all within the with block. Finally, you close the with block, which executes the request and runs the lines of code within its block.
With this code, you cause two sets of fifty bytes each to be displayed. The HTTPResponse object will close once you exit the with block scope, meaning that the .read() method will only return empty bytes objects:
>>>
>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
... response.read(50)
...
b'<!doctype html>n<html>n<head>n <title>Example D'
>>> response.read(50)
b''
In this example, the second call to read fifty bytes is outside the with scope. Being outside the with block means that HTTPResponse is closed, even though you can still access the variable. If you try to read from HTTPResponse when it’s closed, it’ll return an empty bytes object.
Another point to note is that you can’t reread a response once you’ve read all the way to the end:
>>>
>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
... first_read = response.read()
... second_read = response.read()
...
>>> len(first_read)
1256
>>> len(second_read)
0
This example shows that once you’ve read a response, you can’t read it again. If you’ve fully read the response, the subsequent attempt just returns an empty bytes object even though the response isn’t closed. You’d have to make the request again.
In this regard, the response is different from a file object, because with a file object, you can read it multiple times by using the .seek() method, which HTTPResponse doesn’t support. Even after closing a response, you can still access the headers and other metadata, though.
Exploring Text, Octets, and Bits
In most of the examples so far, you read the response body from HTTPResponse, displayed the resulting data immediately, and noted that it was displayed as a bytes object. This is because text information in computers isn’t stored or transmitted as letters, but as bytes!
A raw HTTP message sent over the wire is broken up into a sequence of bytes, sometimes referred to as octets. Bytes are 8-bit chunks. For example, 01010101 is a byte. To learn more about binary, bits, and bytes, check out Bitwise Operators in Python.
So how do you represent letters with bytes? A byte has 256 potential combinations, and you can assign a letter to each combination. You can assign 00000001 to A, 00000010 to B, and so on. ASCII character encoding, which is quite common, uses this type of system to encode 128 characters, which is enough for a language like English. This is particularly convenient because just one byte can represent all the characters, with space to spare.
All the standard English characters, including capitals, punctuation, and numerals, fit within ASCII. On the other hand, Japanese is thought to have around fifty thousand logographic characters, so 128 characters won’t cut it! Even the 256 characters that are theoretically available within one byte wouldn’t be nearly enough for Japanese. So, to accomodate all the world’s languages there are many different systems to encode characters.
Even though there are many systems, one thing you can rely on is the fact that they’ll always be broken up into bytes. Most servers, if they can’t resolve the MIME type and character encoding, default to application/octet-stream, which literally means a stream of bytes. Then whoever receives the message can work out the character encoding.
Dealing With Character Encodings
Problems often arise because, as you may have guessed, there are many, many different potential character encodings. The dominant character encoding today is UTF-8, which is an implementation of Unicode. Luckily, ninety-eight percent of web pages today are encoded in UTF-8!
UTF-8 is dominant because it can efficiently handle a mind-boggling number of characters. It handles all the 1,112,064 potential characters defined by Unicode, encompassing Chinese, Japanese, Arabic (with right-to-left scripts), Russian, and many more character sets, including emojis!
UTF-8 remains efficient because it uses a variable number of bytes to encode characters, which means that for many characters, it only requires one byte, while for others it can require up to four bytes.
While UTF-8 is dominant, and you usually won’t go wrong with assuming UTF-8 encodings, you’ll still run into different encodings all the time. The good news is that you don’t need to be an expert on encodings to handle them when using urllib.request.
Going From Bytes to Strings
When you use urllib.request.urlopen(), the body of the response is a bytes object. The first thing you may want to do is to convert the bytes object to a string. Perhaps you want to do some web scraping. To do this, you need to decode the bytes. To decode the bytes with Python, all you need to find out is the character encoding used. Encoding, especially when referring to character encoding, is often referred to as a character set.
As mentioned, ninety-eight percent of the time, you’ll probably be safe defaulting to UTF-8:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> decoded_body = body.decode("utf-8")
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>
In this example, you take the bytes object returned from response.read() and decode it with the bytes object’s .decode() method, passing in utf-8 as an argument. When you print decoded_body, you can see that it’s now a string.
That said, leaving it up to chance is rarely a good strategy. Fortunately, headers are a great place to get character set information:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'utf-8'
>>> decoded_body = body.decode(character_set)
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>
In this example, you call .get_content_charset() on the .headers object of response and use that to decode. This is a convenience method that parses the Content-Type header so that you can painlessly decode bytes into text.
Going From Bytes to File
If you want to decode bytes into text, now you’re good to go. But what if you want to write the body of a response into a file? Well, you have two options:
- Write the bytes directly to the file
- Decode the bytes into a Python string, and then encode the string back into a file
The first method is the most straightforward, but the second method allows you to change the encoding if you want to. To learn about file manipulation in more detail, take a look at Real Python’s Reading and Writing Files in Python (Guide).
To write the bytes directly to a file without having to decode, you’ll need the built-in open() function, and you’ll need to ensure that you use write binary mode:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
... body = response.read()
...
>>> with open("example.html", mode="wb") as html_file:
... html_file.write(body)
...
1256
Using open() in wb mode bypasses the need to decode or encode and dumps the bytes of the HTTP message body into the example.html file. The number that’s output after the writing operation indicates the number of bytes that have been written. That’s it! You’ve written the bytes directly to a file without encoding or decoding anything.
Now say you have a URL that doesn’t use UTF-8, but you want to write the contents to a file with UTF-8. For this, you’d first decode the bytes into a string and then encode the string into a file, specifying the character encoding.
Google’s home page seems to use different encodings depending on your location. In much of Europe and the US, it uses the ISO-8859-1 encoding:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://www.google.com") as response:
... body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'ISO-8859-1'
>>> content = body.decode(character_set)
>>> with open("google.html", encoding="utf-8", mode="w") as file:
... file.write(content)
...
14066
In this code, you got the response character set and used it to decode the bytes object into a string. Then you wrote the string to a file, encoding it using UTF-8.
Once you’ve written to a file, you should be able to open the resulting file in your browser or text editor. Most modern text processors can detect the character encoding automatically.
If there are encoding errors and you’re using Python to read a file, then you’ll likely get an error:
>>>
>>> with open("encoding-error.html", mode="r", encoding="utf-8") as file:
... lines = file.readlines()
...
UnicodeDecodeError:
'utf-8' codec can't decode byte
Python explicitly stops the process and raises an exception, but in a program that displays text, such as the browser where you’re viewing this page, you may find the infamous replacement characters:
The black rhombus with a white question mark (�), the square (□), and the rectangle (▯) are often used as replacements for characters which couldn’t be decoded.
Sometimes, decoding seems to work but results in unintelligible sequences, such as æ–‡å—化ã‘., which also suggests the wrong character set was used. In Japan, they even have a word for text that’s garbled due to character encoding issues, Mojibake, because these issues plagued them at the start of the Internet age.
With that, you should now be equipped to write files with the raw bytes returned from urlopen(). In the next section, you’ll learn how to parse bytes into a Python dictionary with the json module.
Going From Bytes to Dictionary
For application/json responses, you’ll often find that they don’t include any encoding information:
>>>
>>> from urllib.request import urlopen
>>> with urlopen("https://httpbin.org/json") as response:
... body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> print(character_set)
None
In this example, you use the json endpoint of httpbin, a service that allows you to experiment with different types of requests and responses. The json endpoint simulates a typical API that returns JSON data. Note that the .get_content_charset() method returns nothing in its response.
Even though there’s no character encoding information, all is not lost. According to RFC 4627, the default encoding of UTF-8 is an absolute requirement of the application/json specification. That’s not to say that every single server plays by the rules, but generally, you can assume that if JSON is being transmitted, it’ll almost always be encoded using UTF-8.
Fortunately, json.loads() decodes byte objects under the hood and even has some leeway in terms of different encodings that it can deal with. So, json.loads() should be able to cope with most bytes objects that you throw at it, as long as they’re valid JSON:
>>>
>>> import json
>>> json.loads(body)
{'slideshow': {'author': 'Yours Truly', 'date': 'date of publication', 'slides'
: [{'title': 'Wake up to WonderWidgets!', 'type': 'all'}, {'items': ['Why <em>W
onderWidgets</em> are great', 'Who <em>buys</em> WonderWidgets'], 'title': 'Ove
rview', 'type': 'all'}], 'title': 'Sample Slide Show'}}
As you can see, the json module handles the decoding automatically and produces a Python dictionary. Almost all APIs return key-value information as JSON, although you might run into some older APIs that work with XML. For that, you might want to look into the Roadmap to XML Parsers in Python.
With that, you should know enough about bytes and encodings to be dangerous! In the next section, you’ll learn how to troubleshoot and fix a couple of common errors that you might run into when using urllib.request.
Common urllib.request Troubles
There are many kinds of issues you can run into on the world wild web, whether you’re using urllib.request or not. In this section, you’ll learn how to deal with a couple of the most common errors when getting started out: 403 errors and TLS/SSL certificate errors. Before looking at these specific errors, though, you’ll first learn how to implement error handling more generally when using urllib.request.
Implementing Error Handling
Before you turn your attention to specific errors, boosting your code’s ability to gracefully deal with assorted errors will pay off. Web development is plagued with errors, and you can invest a lot of time in handling errors sensibly. Here, you’ll learn to handle HTTP, URL, and timeout errors when using urllib.request.
HTTP status codes accompany every response in the status line. If you can read a status code in the response, then the request reached its target. While this is good, you can only consider the request a complete success if the response code starts with a 2. For example, 200 and 201 represent successful requests. If the status code is 404 or 500, for example, something went wrong, and urllib.request will raise an HTTPError.
Sometimes mistakes happen, and the URL provided isn’t correct, or a connection can’t be made for another reason. In these cases, urllib.request will raise a URLError.
Finally, sometimes servers just don’t respond. Maybe your network connection is slow, the server is down, or the server is programmed to ignore specific requests. To deal with this, you can pass a timeout argument to urlopen() to raise a TimeoutError after a certain amount of time.
The first step in handling these exceptions is to catch them. You can catch errors produced within urlopen() with a try … except block, making use of the HTTPError, URLError, and TimeoutError classes:
# request.py
from urllib.error import HTTPError, URLError
from urllib.request import urlopen
def make_request(url):
try:
with urlopen(url, timeout=10) as response:
print(response.status)
return response.read(), response
except HTTPError as error:
print(error.status, error.reason)
except URLError as error:
print(error.reason)
except TimeoutError:
print("Request timed out")
The function make_request() takes a URL string as an argument, tries to get a response from that URL with urllib.request, and catches the HTTPError object that’s raised if an error occurs. If the URL is bad, it’ll catch a URLError. If it goes through without any errors, it’ll just print the status and return a tuple containing the body and the response. The response will close after return.
The function also calls urlopen() with a timeout argument, which will cause a TimeoutError to be raised after the seconds specified. Ten seconds is generally a good amount of time to wait for a response, though as always, much depends on the server that you need to make the request to.
Now you’re set up to gracefully handle a variety of errors, including but not limited to the errors that you’ll cover next.
Dealing With 403 Errors
You’ll now use the make_request() function to make some requests to httpstat.us, which is a mock server used for testing. This mock server will return responses that have the status code you request. If you make a request to https://httpstat.us/200, for example, you should expect a 200 response.
APIs like httpstat.us are used to ensure that your application can handle all the different status codes it might encounter. httpbin also has this functionality, but httpstat.us has a more comprehensive selection of status codes. It even has the infamous and semi-official 418 status code that returns the message I’m a teapot!
To interact with the make_request() function that you wrote in the previous section, run the script in interactive mode:
With the -i flag, this command will run the script in interactive mode. This means that it’ll execute the script and then open the Python REPL afterward, so you can now call the function that you just defined:
>>>
>>> make_request("https://httpstat.us/200")
200
(b'200 OK', <http.client.HTTPResponse object at 0x0000023D612660B0>)
>>> make_request("https://httpstat.us/403")
403 Forbidden
Here you tried the 200 and 403 endpoints of httpstat.us. The 200 endpoint goes through as anticipated and returns the body of the response and the response object. The 403 endpoint just printed the error message and didn’t return anything, also as expected.
The 403 status means that the server understood the request but won’t fulfill it. This is a common error that you can run into, especially while web scraping. In many cases, you can solve it by passing a User-Agent header.
One of the primary ways that servers identify who or what is making the request is by examining the User-Agent header. The raw default request sent by urllib.request is the following:
GET https://httpstat.us/403 HTTP/1.1
Accept-Encoding: identity
Host: httpstat.us
User-Agent: Python-urllib/3.10
Connection: close
Notice that User-Agent is listed as Python-urllib/3.10. You may find that some sites will try to block web scrapers, and this User-Agent is a dead giveaway. With that said, you can set your own User-Agent with urllib.request, though you’ll need to modify your function a little:
# request.py
from urllib.error import HTTPError, URLError
-from urllib.request import urlopen
+from urllib.request import urlopen, Request
-def make_request(url):
+def make_request(url, headers=None):
+ request = Request(url, headers=headers or {})
try:
- with urlopen(url, timeout=10) as response:
+ with urlopen(request, timeout=10) as response:
print(response.status)
return response.read(), response
except HTTPError as error:
print(error.status, error.reason)
except URLError as error:
print(error.reason)
except TimeoutError:
print("Request timed out")
To customize the headers that you send out with your request, you first have to instantiate a Request object with the URL. Additionally, you can pass in a keyword argument of headers, which accepts a standard dictionary representing any headers you wish to include. So, instead of passing the URL string directly into urlopen(), you pass this Request object which has been instantiated with the URL and headers.
To use this revamped function, restart the interactive session, then call make_request() with a dictionary representing the headers as an argument:
>>>
>>> body, response = make_request(
... "https://www.httpbin.org/user-agent",
... {"User-Agent": "Real Python"}
... )
200
>>> body
b'{n "user-agent": "Real Python"n}n'
In this example, you make a request to httpbin. Here you use the user-agent endpoint to return the request’s User-Agent value. Because you made the request with a custom user agent of Real Python, this is what gets returned.
Some servers are strict, though, and will only accept requests from specific browsers. Luckily, it’s possible to find standard User-Agent strings on the web, including through a user agent database. They’re just strings, so all you need to do is copy the user agent string of the browser that you want to impersonate and use it as the value of the User-Agent header.
Fixing the SSL CERTIFICATE_VERIFY_FAILED Error
Another common error is due to Python not being able to access the required security certificate. To simulate this error, you can use some mock sites that have known bad SSL certificates, provided by badssl.com. You can make a request to one of them, such as superfish.badssl.com, and experience the error firsthand:
>>>
>>> from urllib.request import urlopen
>>> urlopen("https://superfish.badssl.com/")
Traceback (most recent call last):
(...)
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
(...)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Here, making a request to an address with a known bad SSL certificate will result in CERTIFICATE_VERIFY_FAILED which is a type of URLError.
SSL stands for Secure Sockets Layer. This is something of a misnomer because SSL was deprecated in favor of TLS, Transport Layer Security. Sometimes old terminology just sticks! It’s a way to encrypt network traffic so that a hypothetical listener can’t eavesdrop on the information transmitted over the wire.
These days, most website addresses are preceded not by http:// but by https://, with the s standing for secure. HTTPS connections must be encrypted through the TLS. urllib.request can handle both HTTP and HTTPS connections.
The details of HTTPS are far beyond the scope of this tutorial, but you can think of an HTTPS connection as involving two stages, the handshake and the transfer of information. The handshake ensures that the connection is secure. For more information about Python and HTTPS, check out Exploring HTTPS With Python.
To establish that a particular server is secure, programs that make requests rely on a store of trusted certificates. The server’s certificate is verified during the handshake stage. Python uses the operating system’s store of certificates. If Python can’t find the system’s store of certificates, or if the store is out of date, then you’ll run into this error.
Sometimes the store of certificates that Python can access is out of date, or Python can’t reach it, for whatever reason. This is frustrating because you can sometimes visit the URL from your browser, which thinks that it’s secure, yet urllib.request still raises this error.
You may be tempted to opt out of verifying the certificate, but this will render your connection insecure and is definitely not recommended:
>>>
>>> import ssl
>>> from urllib.request import urlopen
>>> unverified_context = ssl._create_unverified_context()
>>> urlopen("https://superfish.badssl.com/", context=unverified_context)
<http.client.HTTPResponse object at 0x00000209CBE8F220>
Here you import the ssl module, which allows you to create an unverified context. You can then pass this context to urlopen() and visit a known bad SSL certificate. The connection successfully goes through because the SSL certificate isn’t checked.
Before resorting to these desperate measures, try updating your OS or updating your Python version. If that fails, then you can take a page from the requests library and install certifi:
- Windows
- Linux + macOS
PS> python -m venv venv
PS> .venvScriptsactivate
(venv) PS> python -m pip install certifi
$ python3 -m venv venv
$ source venv/bin/activate.sh
(venv) $ python3 -m pip install certifi
certifi is a collection of certificates that you can use instead of your system’s collection. You do this by creating an SSL context with the certifi bundle of certificates instead of the OS’s bundle:
>>>
>>> import ssl
>>> from urllib.request import urlopen
>>> import certifi
>>> certifi_context = ssl.create_default_context(cafile=certifi.where())
>>> urlopen("https://sha384.badssl.com/", context=certifi_context)
<http.client.HTTPResponse object at 0x000001C7407C3490>
In this example, you used certifi to act as your SSL certificate store, and you used it to successfully connect to a site with a known good SSL certificate. Note that instead of ._create_unverified_context(), you use .create_default_context().
This way, you can stay secure without too much trouble! In the next section, you’ll be dipping your toes into the world of authentication.
Authenticated Requests
Authentication is a vast subject, and if you’re dealing with authentication much more complicated than what’s covered here, this might be a good jumping-off point into the requests package.
In this tutorial, you’ll only cover one authentication method, which serves as an example of the type of adjustments that you have to make to authenticate your requests. urllib.request does have a lot of other functionality that helps with authentication, but that won’t be covered in this tutorial.
One of the most common authentication tools is the bearer token, specified by RFC 6750. It’s often used as part of OAuth, but can also be used in isolation. It’s also most common to see as a header, which you can use with your current make_request() function:
>>>
>>> token = "abcdefghijklmnopqrstuvwxyz"
>>> headers = {
... "Authorization": f"Bearer {token}"
... }
>>> make_request("https://httpbin.org/bearer", headers)
200
(b'{n "authenticated": true, n "token": "abcdefghijklmnopqrstuvwxyz"n}n',
<http.client.HTTPResponse object at 0x0000023D612642E0>)
In this example, you make a request to the httpbin /bearer endpoint, which simulates bearer authentication. It’ll accept any string as a token. It only requires the proper format specified by RFC 6750. The name has to be Authorization, or sometimes the lowercase authorization, and the value has to be Bearer, with a single space between that and the token.
Congratulations, you’ve successfully authenticated, using a bearer token!
Another form of authentication is called Basic Access Authentication, which is a very simple method of authentication, only slightly better than sending a username and password in a header. It’s very insecure!
One of the most common protocols in use today is OAuth (Open Authorization). If you’ve ever used Google, GitHub, or Facebook to sign into another website, then you’ve used OAuth. The OAuth flow generally involves a few requests between the service that you want to interact with and an identity server, resulting in a short-lived bearer token. This bearer token can then be used for a period of time with bearer authentication.
Much of authentication comes down to understanding the specific protocol that the target server uses and reading the documentation closely to get it working.
POST Requests With urllib.request
You’ve made a lot of GET requests, but sometimes you want to send information. That’s where POST requests come in. To make POST requests with urllib.request, you don’t have to explicitly change the method. You can just pass a data object to a new Request object or directly to urlopen(). The data object must be in a special format, though. You’ll adapt your make_request() function slightly to support POST requests by adding the data parameter:
# request.py
from urllib.error import HTTPError, URLError
from urllib.request import urlopen, Request
-def make_request(url, headers=None):
+def make_request(url, headers=None, data=None):
- request = Request(url, headers=headers or {})
+ request = Request(url, headers=headers or {}, data=data)
try:
with urlopen(request, timeout=10) as response:
print(response.status)
return response.read(), response
except HTTPError as error:
print(error.status, error.reason)
except URLError as error:
print(error.reason)
except TimeoutError:
print("Request timed out")
Here you just modified the function to accept a data argument with a default value of None, and you passed that right into the Request instantiation. That’s not all that needs to be done, though. You can use one of two different formats to execute a POST request:
- Form Data:
application/x-www-form-urlencoded - JSON:
application/json
The first format is the oldest format for POST requests and involves encoding the data with percent encoding, also known as URL encoding. You may have noticed key-value pairs URL encoded as a query string. Keys are separated from values with an equal sign (=), key-value pairs are separated with an ampersand (&), and spaces are generally suppressed but can be replaced with a plus sign (+).
If you’re starting off with a Python dictionary, to use the form data format with your make_request() function, you’ll need to encode twice:
- Once to URL encode the dictionary
- Then again to encode the resulting string into bytes
For the first stage of URL encoding, you’ll use another urllib module, urllib.parse. Remember to start your script in interactive mode so that you can use the make_request() function and play with it on the REPL:
>>>
>>> from urllib.parse import urlencode
>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> url_encoded_data = urlencode(post_dict)
>>> url_encoded_data
'Title=Hello+World&Name=Real+Python'
>>> post_data = url_encoded_data.encode("utf-8")
>>> body, response = make_request(
... "https://httpbin.org/anything", data=post_data
... )
200
>>> print(body.decode("utf-8"))
{
"args": {},
"data": "",
"files": {},
"form": {
"Name": "Real Python",
"Title": "Hello World"
},
"headers": {
"Accept-Encoding": "identity",
"Content-Length": "34",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "Python-urllib/3.10",
"X-Amzn-Trace-Id": "Root=1-61f25a81-03d2d4377f0abae95ff34096"
},
"json": null,
"method": "POST",
"origin": "86.159.145.119",
"url": "https://httpbin.org/anything"
}
In this example, you:
- Import
urlencode()from theurllib.parsemodule - Initialize your POST data, starting with a dictionary
- Use the
urlencode()function to encode the dictionary - Encode the resulting string into bytes using UTF-8 encoding
- Make a request to the
anythingendpoint ofhttpbin.org - Print the UTF-8 decoded response body
UTF-8 encoding is part of the specification for the application/x-www-form-urlencoded type. UTF-8 is used preemptively to decode the body because you already know that httpbin.org reliably uses UTF-8.
The anything endpoint from httpbin acts as a sort of echo, returning all the information it received so that you can inspect the details of the request you made. In this case, you can confirm that method is indeed POST, and you can see that the data you sent is listed under form.
To make the same request with JSON, you’ll turn a Python dictionary into a JSON string with json.dumps(), encode it with UTF-8, pass it as the data argument, and finally add a special header to indicate that the data type is JSON:
>>>
>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> import json
>>> json_string = json.dumps(post_dict)
>>> json_string
'{"Title": "Hello World", "Name": "Real Python"}'
>>> post_data = json_string.encode("utf-8")
>>> body, response = make_request(
... "https://httpbin.org/anything",
... data=post_data,
... headers={"Content-Type": "application/json"},
... )
200
>>> print(body.decode("utf-8"))
{
"args": {},
"data": "{"Title": "Hello World", "Name": "Real Python"}",
"files": {},
"form": {},
"headers": {
"Accept-Encoding": "identity",
"Content-Length": "47",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "Python-urllib/3.10",
"X-Amzn-Trace-Id": "Root=1-61f25a81-3e35d1c219c6b5944e2d8a52"
},
"json": {
"Name": "Real Python",
"Title": "Hello World"
},
"method": "POST",
"origin": "86.159.145.119",
"url": "https://httpbin.org/anything"
}
To serialize the dictionary this time around, you use json.dumps() instead of urlencode(). You also explicitly add the Content-Type header with a value of application/json. With this information, the httpbin server can deserialize the JSON on the receiving end. In its response, you can see the data listed under the json key.
With that, you can now start making POST requests. This tutorial won’t go into more detail about the other request methods, such as PUT. Suffice to say that you can also explicitly set the method by passing a method keyword argument to the instantiation of the Request object.
The Request Package Ecosystem
To round things out, this last section of the tutorial is dedicated to clarifying the package ecosystem around HTTP requests with Python. Because there are many packages, with no clear standard, it can be confusing. That said, there are use cases for each package, which just means more choice for you!
What Are urllib2 and urllib3?
To answer this question, you need to go back to early Python, all the way back to version 1.2, when the original urllib was introduced. Around version 1.6, a revamped urllib2 was added, which lived alongside the original urllib. When Python 3 came along, the original urllib was deprecated, and urllib2 dropped the 2, taking on the original urllib name. It also split into parts:
urllib.errorurllib.parseurllib.requesturllib.responseurllib.robotparser
So what about urllib3? That’s a third-party library developed while urllib2 was still around. It’s not related to the standard library because it’s an independently maintained library. Interestingly, the requests library actually uses urllib3 under the hood, and so does pip!
When Should I Use requests Over urllib.request?
The main answer is ease of use and security. urllib.request is considered a low-level library, which exposes a lot of the detail about the workings of HTTP requests. The Python documentation for urllib.request makes no bones about recommending requests as a higher-level HTTP client interface.
If you interact with many different REST APIs, day in and day out, then requests is highly recommended. The requests library bills itself as “built for human beings” and has successfully created an intuitive, secure, and straightforward API around HTTP. It’s usually considered the go-to library! If you want to know more about the requests library, check out the Real Python guide to requests.
An example of how requests makes things easier is when it comes to character encoding. You’ll remember that with urllib.request, you have to be aware of encodings and take a few steps to ensure an error-free experience. The requests package abstracts that away and will resolve the encoding by using chardet, a universal character encoding detector, just in case there’s any funny business.
If your goal is to learn more about standard Python and the details of how it deals with HTTP requests, then urllib.request is a great way to get into that. You could even go further and use the very low-level http modules. On the other hand, you may just want to keep dependencies to a minimum, which urllib.request is more than capable of.
Why Is requests Not Part of the Standard Library?
Maybe you’re wondering why requests isn’t part of core Python by this point.
This is a complex issue, and there’s no hard and fast answer to it. There are many speculations as to why, but two reasons seem to stand out:
requestshas other third-party dependencies that would need to be integrated too.requestsneeds to stay agile and can do this better outside the standard library.
The requests library has third-party dependencies. Integrating requests into the standard library would mean also integrating chardet, certifi, and urllib3, among others. The alternative would be to fundamentally change requests to use only Python’s existing standard library. This is no trivial task!
Integrating requests would also mean that the existing team that develops this library would have to relinquish total control over the design and implementation, giving way to the PEP decision-making process.
HTTP specifications and recommendations change all the time, and a high-level library has to be agile enough to keep up. If there’s a security exploit to be patched, or a new workflow to add, the requests team can build and release far more quickly than they could as part of the Python release process. There have supposedly been times when they’ve released a security fix twelve hours after a vulnerability was discovered!
For an interesting overview of these issues and more, check out Adding Requests to The Standard Library, which summarizes a discussion at the Python Language Summit with Kenneth Reitz, the creator and maintainer of Requests.
Because this agility is so necessary to requests and its underlying urllib3, the paradoxical statement that requests is too important for the standard library is often used. This is because so much of the Python community depends on requests and its agility that integrating it into core Python would probably damage it and the Python community.
On the GitHub repository issues board for requests, an issue was posted, asking for the inclusion of requests in the standard library. The developers of requests and urllib3 chimed in, mainly saying they would likely lose interest in maintaining it themselves. Some even said they would fork the repositories and continue developing them for their own use cases.
With that said, note that the requests library GitHub repository is hosted under the Python Software Foundation’s account. Just because something isn’t part of the Python standard library doesn’t mean that it’s not an integral part of the ecosystem!
It seems that the current situation works for both the Python core team and the maintainers of requests. While it may be slightly confusing for newcomers, the existing structure gives the most stable experience for HTTP requests.
It’s also important to note that HTTP requests are inherently complex. urllib.request doesn’t try to sugarcoat that too much. It exposes a lot of the inner workings of HTTP requests, which is why it’s billed as a low-level module. Your choice of requests versus urllib.request really depends on your particular use case, security concerns, and preference.
Conclusion
You’re now equipped to use urllib.request to make HTTP requests. Now you can use this built-in module in your projects, keeping them dependency-free for longer. You’ve also gained the in-depth understanding of HTTP that comes from using a lower-level module, such as urllib.request.
In this tutorial, you’ve:
- Learned how to make basic HTTP requests with
urllib.request - Explored the nuts and bolts of an HTTP message and studied how it’s represented by
urllib.request - Figured out how to deal with character encodings of HTTP messages
- Explored some common errors when using
urllib.requestand learned how to resolve them - Dipped your toes into the world of authenticated requests with
urllib.request - Understood why both
urlliband therequestslibrary exist and when to use one or the other
You’re now in a position to make basic HTTP requests with urllib.request, and you also have the tools to dive deeper into low-level HTTP terrain with the standard library. Finally, you can choose whether to use requests or urllib.request, depending on what you want or need. Have fun exploring the Web!
The urllib.error.httperror: http error 403: forbidden occurs when you try to scrap a webpage using urllib.request module and the mod_security blocks the request. There are several reasons why you get this error. Let’s take a look at each of the use cases in detail.
Usually, the websites are protected with App Gateway, WAF rules, etc., which monitor whether the requests are from the actual users or triggered through the automated bot system. The mod_security or the WAF rule will block these requests treating them as spider/bot requests. These security features are the most standard ones to prevent DDOS attacks on the server.
Now coming back to the error when you make a request to any site using urllib.request basically, you will not set any user-agents and headers and by default the urllib sets something like python urllib/3.3.0, which is easily detected by the mod_security.
The mod_security is usually configured in such a way that if any requests happen without a valid user-agent header(browser user-agent), the mod_security will block the request and return the urllib.error.httperror: http error 403: forbidden
Example of 403 forbidden error
from urllib.request import Request, urlopen
req = Request('http://www.cmegroup.com/')
webpage = urlopen(req).read()Output
File "C:UsersuserAppDataLocalProgramsPythonPython39liburllibrequest.py", line 494, in _call_chain
result = func(*args)
urllib.error.HTTPError: HTTP Error 403: Forbidden
PS C:ProjectsTryouts> from urllib.request import Request, urlopenThe easy way to resolve the error is by passing a valid user-agent as a header parameter, as shown below.
from urllib.request import Request, urlopen
req = Request('https://www.yahoo.com', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
Alternatively, you can even set a timeout if you are not getting the response from the website. Python will raise a socket exception if the website doesn’t respond within the mentioned timeout period.
from urllib.request import Request, urlopen
req = Request('http://www.cmegroup.com/', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req,timeout=10).read()
In some cases, like getting a real-time bitcoin or stock market value, you will send requests every second, and the servers can block if there are too many requests coming from the same IP address and throws 403 security error.
If you get this error because of too many requests, consider adding delay between each request to resolve the error.
Srinivas Ramakrishna is a Solution Architect and has 14+ Years of Experience in the Software Industry. He has published many articles on Medium, Hackernoon, dev.to and solved many problems in StackOverflow. He has core expertise in various technologies such as Microsoft .NET Core, Python, Node.JS, JavaScript, Cloud (Azure), RDBMS (MSSQL), React, Powershell, etc.
Sign Up for Our Newsletters
Subscribe to get notified of the latest articles. We will never spam you. Be a part of our ever-growing community.
By checking this box, you confirm that you have read and are agreeing to our terms of use regarding the storage of the data submitted through this form.
|
olm18 0 / 0 / 0 Регистрация: 15.12.2016 Сообщений: 3 |
||||
|
1 |
||||
|
18.03.2019, 22:28. Показов 17895. Ответов 3 Метки 403, get запрос, requests (Все метки)
Делаю запрос. Статус запроса 403. Через браузер запрос выполнить получается. Везде пишут, что в такой ситуации надо указать User-Agent. Я указал, но никакого результата. Пробовал добавлять различные параметры в headers, но та же ошибка 403. Подскажите, что я делаю не так?
Миниатюры
__________________
0 |
|
5403 / 3827 / 1214 Регистрация: 28.10.2013 Сообщений: 9,554 Записей в блоге: 1 |
|
|
18.03.2019, 22:49 |
2 |
|
Сайту нужна кука — поэтому используйте сессию. В requests сессия есть.
0 |
|
olm18 0 / 0 / 0 Регистрация: 15.12.2016 Сообщений: 3 |
||||||||
|
19.03.2019, 00:24 [ТС] |
3 |
|||||||
|
Делаю так. В результате всё равно ошибка 403
Добавлено через 59 минут
0 |
|
Garry Galler
5403 / 3827 / 1214 Регистрация: 28.10.2013 Сообщений: 9,554 Записей в блоге: 1 |
||||
|
19.03.2019, 00:33 |
4 |
|||
|
Решение
Очень странно, но этот параметр из кук (несмотря на предварительный запрос) почему-то в сессионных куках отсутствует, и поэтому без него ничего не работает.
0 |
Сообщение было отмечено olm18 как решение
Содержание
- HOWTO Fetch Internet Resources Using The urllib Package¶
- Introduction¶
- Fetching URLs¶
- Data¶
- Headers¶
- Handling Exceptions¶
- URLError¶
- HTTPError¶
- Error Codes¶
- Wrapping it Up¶
- Number 1В¶
- Number 2В¶
- info and geturl¶
- Openers and Handlers¶
- Basic Authentication¶
- Proxies¶
- Sockets and Layers¶
- Footnotes¶
HOWTO Fetch Internet Resources Using The urllib Package¶
There is a French translation of an earlier revision of this HOWTO, available at urllib2 — Le Manuel manquant.
Introduction¶
You may also find useful the following article on fetching web resources with Python:
A tutorial on Basic Authentication, with examples in Python.
urllib.request is a Python module for fetching URLs (Uniform Resource Locators). It offers a very simple interface, in the form of the urlopen function. This is capable of fetching URLs using a variety of different protocols. It also offers a slightly more complex interface for handling common situations — like basic authentication, cookies, proxies and so on. These are provided by objects called handlers and openers.
urllib.request supports fetching URLs for many “URL schemes” (identified by the string before the «:» in URL — for example «ftp» is the URL scheme of «ftp://python.org/» ) using their associated network protocols (e.g. FTP, HTTP). This tutorial focuses on the most common case, HTTP.
For straightforward situations urlopen is very easy to use. But as soon as you encounter errors or non-trivial cases when opening HTTP URLs, you will need some understanding of the HyperText Transfer Protocol. The most comprehensive and authoritative reference to HTTP is RFC 2616. This is a technical document and not intended to be easy to read. This HOWTO aims to illustrate using urllib, with enough detail about HTTP to help you through. It is not intended to replace the urllib.request docs, but is supplementary to them.
Fetching URLs¶
The simplest way to use urllib.request is as follows:
If you wish to retrieve a resource via URL and store it in a temporary location, you can do so via the shutil.copyfileobj() and tempfile.NamedTemporaryFile() functions:
Many uses of urllib will be that simple (note that instead of an вЂhttp:’ URL we could have used a URL starting with вЂftp:’, вЂfile:’, etc.). However, it’s the purpose of this tutorial to explain the more complicated cases, concentrating on HTTP.
HTTP is based on requests and responses — the client makes requests and servers send responses. urllib.request mirrors this with a Request object which represents the HTTP request you are making. In its simplest form you create a Request object that specifies the URL you want to fetch. Calling urlopen with this Request object returns a response object for the URL requested. This response is a file-like object, which means you can for example call .read() on the response:
Note that urllib.request makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so:
In the case of HTTP, there are two extra things that Request objects allow you to do: First, you can pass data to be sent to the server. Second, you can pass extra information (“metadata”) about the data or about the request itself, to the server — this information is sent as HTTP “headers”. Let’s look at each of these in turn.
Data¶
Sometimes you want to send data to a URL (often the URL will refer to a CGI (Common Gateway Interface) script or other web application). With HTTP, this is often done using what’s known as a POST request. This is often what your browser does when you submit a HTML form that you filled in on the web. Not all POSTs have to come from forms: you can use a POST to transmit arbitrary data to your own application. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the data argument. The encoding is done using a function from the urllib.parse library.
Note that other encodings are sometimes required (e.g. for file upload from HTML forms — see HTML Specification, Form Submission for more details).
If you do not pass the data argument, urllib uses a GET request. One way in which GET and POST requests differ is that POST requests often have “side-effects”: they change the state of the system in some way (for example by placing an order with the website for a hundredweight of tinned spam to be delivered to your door). Though the HTTP standard makes it clear that POSTs are intended to always cause side-effects, and GET requests never to cause side-effects, nothing prevents a GET request from having side-effects, nor a POST requests from having no side-effects. Data can also be passed in an HTTP GET request by encoding it in the URL itself.
This is done as follows:
Notice that the full URL is created by adding a ? to the URL, followed by the encoded values.
We’ll discuss here one particular HTTP header, to illustrate how to add headers to your HTTP request.
Some websites 1 dislike being browsed by programs, or send different versions to different browsers 2. By default urllib identifies itself as Python-urllib/x.y (where x and y are the major and minor version numbers of the Python release, e.g. Python-urllib/2.5 ), which may confuse the site, or just plain not work. The way a browser identifies itself is through the User-Agent header 3. When you create a Request object you can pass a dictionary of headers in. The following example makes the same request as above, but identifies itself as a version of Internet Explorer 4.
The response also has two useful methods. See the section on info and geturl which comes after we have a look at what happens when things go wrong.
Handling Exceptions¶
urlopen raises URLError when it cannot handle a response (though as usual with Python APIs, built-in exceptions such as ValueError , TypeError etc. may also be raised).
HTTPError is the subclass of URLError raised in the specific case of HTTP URLs.
The exception classes are exported from the urllib.error module.
URLError¶
Often, URLError is raised because there is no network connection (no route to the specified server), or the specified server doesn’t exist. In this case, the exception raised will have a вЂreason’ attribute, which is a tuple containing an error code and a text error message.
HTTPError¶
Every HTTP response from the server contains a numeric “status code”. Sometimes the status code indicates that the server is unable to fulfil the request. The default handlers will handle some of these responses for you (for example, if the response is a “redirection” that requests the client fetch the document from a different URL, urllib will handle that for you). For those it can’t handle, urlopen will raise an HTTPError . Typical errors include вЂ404’ (page not found), вЂ403’ (request forbidden), and вЂ401’ (authentication required).
See section 10 of RFC 2616 for a reference on all the HTTP error codes.
The HTTPError instance raised will have an integer вЂcode’ attribute, which corresponds to the error sent by the server.
Error Codes¶
Because the default handlers handle redirects (codes in the 300 range), and codes in the 100–299 range indicate success, you will usually only see error codes in the 400–599 range.
http.server.BaseHTTPRequestHandler.responses is a useful dictionary of response codes in that shows all the response codes used by RFC 2616. The dictionary is reproduced here for convenience
When an error is raised the server responds by returning an HTTP error code and an error page. You can use the HTTPError instance as a response on the page returned. This means that as well as the code attribute, it also has read, geturl, and info, methods as returned by the urllib.response module:
Wrapping it Up¶
So if you want to be prepared for HTTPError or URLError there are two basic approaches. I prefer the second approach.
Number 1В¶
The except HTTPError must come first, otherwise except URLError will also catch an HTTPError .
Number 2В¶
info and geturl¶
The response returned by urlopen (or the HTTPError instance) has two useful methods info() and geturl() and is defined in the module urllib.response ..
geturl — this returns the real URL of the page fetched. This is useful because urlopen (or the opener object used) may have followed a redirect. The URL of the page fetched may not be the same as the URL requested.
info — this returns a dictionary-like object that describes the page fetched, particularly the headers sent by the server. It is currently an http.client.HTTPMessage instance.
Typical headers include вЂContent-length’, вЂContent-type’, and so on. See the Quick Reference to HTTP Headers for a useful listing of HTTP headers with brief explanations of their meaning and use.
Openers and Handlers¶
When you fetch a URL you use an opener (an instance of the perhaps confusingly named urllib.request.OpenerDirector ). Normally we have been using the default opener — via urlopen — but you can create custom openers. Openers use handlers. All the “heavy lifting” is done by the handlers. Each handler knows how to open URLs for a particular URL scheme (http, ftp, etc.), or how to handle an aspect of URL opening, for example HTTP redirections or HTTP cookies.
You will want to create openers if you want to fetch URLs with specific handlers installed, for example to get an opener that handles cookies, or to get an opener that does not handle redirections.
To create an opener, instantiate an OpenerDirector , and then call .add_handler(some_handler_instance) repeatedly.
Alternatively, you can use build_opener , which is a convenience function for creating opener objects with a single function call. build_opener adds several handlers by default, but provides a quick way to add more and/or override the default handlers.
Other sorts of handlers you might want to can handle proxies, authentication, and other common but slightly specialised situations.
install_opener can be used to make an opener object the (global) default opener. This means that calls to urlopen will use the opener you have installed.
Opener objects have an open method, which can be called directly to fetch urls in the same way as the urlopen function: there’s no need to call install_opener , except as a convenience.
Basic Authentication¶
To illustrate creating and installing a handler we will use the HTTPBasicAuthHandler . For a more detailed discussion of this subject – including an explanation of how Basic Authentication works — see the Basic Authentication Tutorial.
When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme and a вЂrealm’. The header looks like: WWW-Authenticate: SCHEME realm=»REALM» .
The client should then retry the request with the appropriate name and password for the realm included as a header in the request. This is вЂbasic authentication’. In order to simplify this process we can create an instance of HTTPBasicAuthHandler and an opener to use this handler.
The HTTPBasicAuthHandler uses an object called a password manager to handle the mapping of URLs and realms to passwords and usernames. If you know what the realm is (from the authentication header sent by the server), then you can use a HTTPPasswordMgr . Frequently one doesn’t care what the realm is. In that case, it is convenient to use HTTPPasswordMgrWithDefaultRealm . This allows you to specify a default username and password for a URL. This will be supplied in the absence of you providing an alternative combination for a specific realm. We indicate this by providing None as the realm argument to the add_password method.
The top-level URL is the first URL that requires authentication. URLs “deeper” than the URL you pass to .add_password() will also match.
In the above example we only supplied our HTTPBasicAuthHandler to build_opener . By default openers have the handlers for normal situations – ProxyHandler (if a proxy setting such as an http_proxy environment variable is set), UnknownHandler , HTTPHandler , HTTPDefaultErrorHandler , HTTPRedirectHandler , FTPHandler , FileHandler , DataHandler , HTTPErrorProcessor .
top_level_url is in fact either a full URL (including the вЂhttp:’ scheme component and the hostname and optionally the port number) e.g. «http://example.com/» or an “authority” (i.e. the hostname, optionally including the port number) e.g. «example.com» or «example.com:8080» (the latter example includes a port number). The authority, if present, must NOT contain the “userinfo” component — for example «joe:password@example.com» is not correct.
Proxies¶
urllib will auto-detect your proxy settings and use those. This is through the ProxyHandler , which is part of the normal handler chain when a proxy setting is detected. Normally that’s a good thing, but there are occasions when it may not be helpful 5. One way to do this is to setup our own ProxyHandler , with no proxies defined. This is done using similar steps to setting up a Basic Authentication handler:
Currently urllib.request does not support fetching of https locations through a proxy. However, this can be enabled by extending urllib.request as shown in the recipe 6.
HTTP_PROXY will be ignored if a variable REQUEST_METHOD is set; see the documentation on getproxies() .
Sockets and Layers¶
The Python support for fetching resources from the web is layered. urllib uses the http.client library, which in turn uses the socket library.
As of Python 2.3 you can specify how long a socket should wait for a response before timing out. This can be useful in applications which have to fetch web pages. By default the socket module has no timeout and can hang. Currently, the socket timeout is not exposed at the http.client or urllib.request levels. However, you can set the default timeout globally for all sockets using
This document was reviewed and revised by John Lee.
Google for example.
Browser sniffing is a very bad practice for website design — building sites using web standards is much more sensible. Unfortunately a lot of sites still send different versions to different browsers.
The user agent for MSIE 6 is вЂMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)’
For details of more HTTP request headers, see Quick Reference to HTTP Headers.
In my case I have to use a proxy to access the internet at work. If you attempt to fetch localhost URLs through this proxy it blocks them. IE is set to use the proxy, which urllib picks up on. In order to test scripts with a localhost server, I have to prevent urllib from using the proxy.
urllib opener for SSL proxy (CONNECT method): ASPN Cookbook Recipe.
Источник