Urlopen error timed out

Содержание

urllib.error.URLError: #2115
Comments
Environment
urllib.error.URLError: #128
Comments
Footer
Urlopen error timed out
Mr Onion
averyfreeman
urlopen error time out #15
Comments
Issue: Running into URLError when uploading CSV file through API #739
Comments
Description
What type of issue is this? (place an x in one of the [ ] )
Requirements (place an x in each of the [ ] )
Bug Report
Reproducible in:
Steps to reproduce:
Expected result:
Actual result:

urllib.error.URLError: #2115

COMPONENT NAME

bigip_pool,
bigip_pool_member,
bigip_virtual_server:

Environment

ANSIBLE VERSION

ansible [core 2.11.6]
config file = /ansible/ansible.cfg
configured module search path = [‘/root/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python3/dist-packages/ansible
ansible collection location = /ansible/collections
executable location = /usr/bin/ansible
python version = 3.8.10 (default, Jun 2 2021, 10:49:15) [GCC 9.4.0]
jinja version = 2.10.1
libyaml = True

BIGIP VERSION

BIG-IP 15.1.0.2 Build 0.0.9 Point Release 2

CONFIGURATION

[defaults]
inventory = ./hosts
host_key_checking = false
gathering_facts = false
timeout = 5
collections_paths = ./collections

OS / ENVIRONMENT

SUMMARY

I have a test linux(ansible ctrl node) make ansible scripts to automate virtual F5, I have both control node and virtual f5 in intranet their IPs respectively 10.21.74.163 & 10.21.12.250. When i try to run basic playbook from F5 site «https://clouddocs.f5.com/products/orchestration/ansible/devel/usage/playbook_tutorial.html» I’m getting urllib.error.URLError: . my inventory files has just localhost in it. and i have added my config file on top. i also searched some of previous bugs and added no_teem_f5 : yes to the provider yet no diff from the error which I’m getting.

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

after using -vvv i get the following error

root && sleep 0′ EXEC /bin/sh -c ‘( umask 77 && mkdir -p «` echo /root/.ansible/tmp `»&& mkdir «` echo /root/.ansible/tmp/ansible-tmp-1634632478.9153419-61922-38643095072116 `» && echo ansible-tmp-1634632478.9153419-61922-38643095072116=»` echo /root/.ansible/tmp/ansible-tmp-1634632478.9153419-61922-38643095072116 `» ) && sleep 0’ redirecting (type: modules) ansible.builtin.bigip_pool to f5networks.f5_modules.bigip_pool Using module file /ansible/collections/ansible_collections/f5networks/f5_modules/plugins/modules/bigip_pool.py PUT /root/.ansible/tmp/ansible-local-61832jy6dh6nd/tmpffb91jvd TO /root/.ansible/tmp/ansible-tmp-1634632478.9153419-61922-38643095072116/AnsiballZ_bigip_pool.py EXEC /bin/sh -c ‘chmod u+x /root/.ansible/tmp/ansible-tmp-1634632478.9153419-61922-38643095072116/ /root/.ansible/tmp/ansible-tmp-1634632478.9153419-61922-38643095072116/AnsiballZ_bigip_pool.py && sleep 0’ EXEC /bin/sh -c ‘/usr/bin/python /root/.ansible/tmp/ansible-tmp-1634632478.9153419-61922-38643095072116/AnsiballZ_bigip_pool.py && sleep 0’ EXEC /bin/sh -c ‘rm -f -r /root/.ansible/tmp/ansible-tmp-1634632478.9153419-61922-38643095072116/ > /dev/null 2>&1 && sleep 0’ The full traceback is: Traceback (most recent call last): File «/usr/lib/python3.8/urllib/request.py», line 1354, in do_open h.request(req.get_method(), req.selector, req.data, headers, File «/usr/lib/python3.8/http/client.py», line 1252, in request self._send_request(method, url, body, headers, encode_chunked) File «/usr/lib/python3.8/http/client.py», line 1298, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File «/usr/lib/python3.8/http/client.py», line 1247, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File «/usr/lib/python3.8/http/client.py», line 1007, in _send_output self.send(msg) File «/usr/lib/python3.8/http/client.py», line 947, in send self.connect() File «/usr/lib/python3.8/http/client.py», line 1414, in connect super().connect() File «/usr/lib/python3.8/http/client.py», line 918, in connect self.sock = self._create_connection( File «/usr/lib/python3.8/socket.py», line 808, in create_connection raise err File «/usr/lib/python3.8/socket.py», line 796, in create_connection sock.connect(sa) socket.timeout: timed out»>

During handling of the above exception, another exception occurred:

The text was updated successfully, but these errors were encountered:

Источник

urllib.error.URLError: #128

Hi Team,
I am running the bingads API call in the server.
After loading the credentials it is trying to open the URL (https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc?singleWsdl)
after some time it is throwing a urllib.error.URLError:
can someone help with this?
FYI, I am attaching the log.

The text was updated successfully, but these errors were encountered:

Hi, seems you are using python 3.6. Could you please try below and let me know if it works for you?

Besides, are you using proxy?

hi qitia,
thanks for the reply
I ran the code which you have given using a proxy

it is throwing urlopen error timed out error
error msg:

and next, I tried to run the same using curl at that time it has given the response

I think proxy is not stopping the URL to hit the server
can you please help with this

thanks for the info @dheemanth1434. From this we can tell that 1) there is no issue from the service(api) side as you can access it with curl. 2) it is not the SDK(the python lib) issue since you repro it without using SDK.

I am thinking there is some network setting issue from your local. What is the behavior when try the first approach without proxy? something like below

hi @qitia,
I ran the code

and its showing the urlopen timed out error

hi @dheemanth1434 it seems to me neither the microsoft ads api issue nor the SDK problem. Could you please try url «https://graphical.weather.gov/xml/SOAP_server/ndfdXMLserver.php?wsdl». If this does not work either, I think you should check from you local settings.

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

Источник

Now this is really weirder. I booted up the system this morning, having done nothing to any of the the settings since the screenshots above. The log in page asked me to update my password (never seen that before) and now I can view all of the discs in the freenas control panel and I am currently running the update.

Updated perfectly, I can check for updates without any error and see all of the discs.

I know I am loosing my memory, but freenas shouldn’t

I have no idea why it stopped working (It had been like that for several weeks) and then suddenly started working, so I can give no advice if anyone else has a similar problem.

Mr Onion

Dabbler

. but now I can’t access files on the freenas server, nor can I check/adjust the sharing settings. Reboot of the server makes no difference. I have also manually updated, with no difference noted.

averyfreeman

Contributor

I’m having a similar issue. I just upgraded from 11-U1 to 11-U2 when I was receiving a traceback error trying to update via the web GUI.

I noticed there was a bug fix for update errors like this in U1, so I just updated to 11-U2 (e417d8aa5) using the U2 ISO and now I’m receiving a different error trying to update using Web GUI «Check Now».

I usually have the FreeNAS server connected to a domain controller/DNS server, but have also tried updating from the GUI with DNS set to 8.8.8.8 (google DNS) and either way I get the same error:

Is there a problem with the FreeNAS update server, or is this yet another bug?

Источник

urlopen error time out #15

I noticed an error message when using scrapy inside the VM. I think it has no consequence for the examples.
The error is link to urllib2 shooting a time out.
Is this a normal behavior and will be fix in later version ?

Feel free to close this if it is not relevant. Thanks !

# scrapy shell http://web:9312/properties/property_000000.html > 2016-09-23 11:04:19 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot) > 2016-09-23 11:04:19 [scrapy] INFO: Optional features available: ssl, http11, boto > 2016-09-23 11:04:19 [scrapy] INFO: Overridden settings: <‘LOGSTATS_INTERVAL’: 0>> 2016-09-23 11:04:19 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState > 2016-09-23 11:04:19 [boto] DEBUG: Retrieving credentials from metadata server. > 2016-09-23 11:04:20 [boto] ERROR: Caught exception reading instance data > Traceback (most recent call last): > File «/usr/local/lib/python2.7/dist-packages/boto/utils.py», line 210, in retry_url > r = opener.open(req, timeout=timeout) > File «/usr/lib/python2.7/urllib2.py», line 404, in open > response = self._open(req, data) > File «/usr/lib/python2.7/urllib2.py», line 422, in _open > ‘_open’, req) > File «/usr/lib/python2.7/urllib2.py», line 382, in _call_chain > result = func(*args) > File «/usr/lib/python2.7/urllib2.py», line 1214, in http_open > return self.do_open(httplib.HTTPConnection, req) > File «/usr/lib/python2.7/urllib2.py», line 1184, in do_open > raise URLError(err) > URLError: > 2016-09-23 11:04:20 [boto] ERROR: Unable to read instance data, giving up > 2016-09-23 11:04:20 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats > 2016-09-23 11:04:20 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware > 2016-09-23 11:04:20 [scrapy] INFO: Enabled item pipelines: > 2016-09-23 11:04:20 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 > 2016-09-23 11:04:20 [scrapy] INFO: Spider opened > 2016-09-23 11:04:20 [scrapy] DEBUG: Crawled (200) (referer: None) > [s] Available Scrapy objects: > [s] crawler > [s] item <> > [s] request > [s] response > [s] settings > [s] spider > [s] Useful shortcuts: > [s] shelp() Shell help (print this help) > [s] fetch(req_or_url) Fetch request (or URL) and update local objects > [s] view(response) View response in a browser»>

The text was updated successfully, but these errors were encountered:

Источник

Issue: Running into URLError when uploading CSV file through API #739

Description

Hi Folks, I am running into URLError when uploading CSV file of large size (

200MB). The file is eventually uploaded into slack, yet the code breaks with that error. Can anyone explain why that’s happening?

What type of issue is this? (place an x in one of the [ ] )

bug
enhancement (feature request)
question
documentation related
testing related
discussion

Requirements (place an x in each of the [ ] )

I’ve read and understood the Contributing guidelines and have done my best effort to follow them.
I’ve read and agree to the Code of Conduct.
I’ve searched for any related issues and avoided creating a duplicate issue.

Bug Report

Filling out the following details about bugs will help us solve your issue sooner.

Reproducible in:

slackclient version: 2.7.1

python version: 3.6.5

OS version(s): macOS 10.14.6

Steps to reproduce:

Create a webclient with the necessary permissions
Create/Load a dataframe with size

200MB as df (the one I tested with had a shape of (5400000, 6))

use the files_upload method to load files using the command as shown web_client.files_upload(channels = ‘CXXXXXXXXXX’, initial_comment=»something», content=df.to_csv(index=False), filename=»test.csv»)

Expected result:

What you expected to happen: File sent with response ‘ok’:true

Actual result:

What actually happened:

/anaconda3/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked) 1238 «»»Send a complete request to the server.»»» -> 1239 self._send_request(method, url, body, headers, encode_chunked) 1240

/anaconda3/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked) 1284 body = _encode(body, ‘body’) -> 1285 self.endheaders(body, encode_chunked=encode_chunked) 1286

/anaconda3/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked) 1233 raise CannotSendHeader() -> 1234 self._send_output(message_body, encode_chunked=encode_chunked) 1235

/anaconda3/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked) 1064 + b’rn’ -> 1065 self.send(chunk) 1066

/anaconda3/lib/python3.6/http/client.py in send(self, data) 985 try: —> 986 self.sock.sendall(data) 987 except TypeError:

/anaconda3/lib/python3.6/ssl.py in sendall(self, data, flags) 971 while count 972 v = self.send(byte_view[count:]) 973 count += v

/anaconda3/lib/python3.6/ssl.py in send(self, data, flags) 940 self.__class__) —> 941 return self._sslobj.write(data) 942 else:

/anaconda3/lib/python3.6/ssl.py in write(self, data) 641 «»» —> 642 return self._sslobj.write(data) 643 timeout: The write operation timed out During handling of the above exception, another exception occurred: [74/639] URLError Traceback (most recent call last) in —-> 1 web_client.files_upload(channels = ‘CXXXXXXXXXX’, initial_comment=»something», content=df.to_csv(index=False), filename=»test.csv»)

/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/client.py in files_upload(self, file, content, **kwargs) 1217 data = kwargs.copy() 1218 data.update(<«content»: content>) -> 1219 return self.api_call(«files.upload», data=data) 1220 1221 def groups_archive(self, *, channel: str, **kwargs) -> Union[Future, SlackResponse]:

/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in api_call(self, api_method, http_verb, files, data, params, json, headers, auth) 213 return self._event_loop.run_until_complete(future) 214 else: —> 215 return self._sync_send(api_url=api_url, req_args=req_args) 216 217 def _get_url(self, api_method):

/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _sync_send(self, api_url, req_args) 346 files=files, 347 json_body=_json, —> 348 additional_headers=headers, 349 ) 350

/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _urllib_api_call(self, token, url, query_params, json_body, body_params, files, additional_headers) 448 url = f»&» if «?» in url else f»?» 449 —> 450 response = self._perform_urllib_http_request(url=url, args=request_args) 451 if response.get(«body», None): 452 response_body_data: dict = json.loads(response[«body»])

/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args) 579 except Exception as err: 580 self._logger.error(f»Failed to send a request to Slack API server: «) —> 581 raise err 582 583 def _build_urllib_request_headers(

/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args) 560 561 resp: HTTPResponse = urlopen( —> 562 req, context=self.ssl, timeout=self.timeout 563 ) 564 charset = resp.headers.get_content_charset()

/anaconda3/lib/python3.6/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context) 221 else: 222 opener = _opener —> 223 return opener.open(url, data, timeout) 224 225 def install_opener(opener):

/anaconda3/lib/python3.6/urllib/request.py in open(self, fullurl, data, timeout) 524 req = meth(req) 525 —> 526 response = self._open(req, data) 527 528 # post-process response

/anaconda3/lib/python3.6/urllib/request.py in _open(self, req, data) 542 protocol = req.type 543 result = self._call_chain(self.handle_open, protocol, protocol + —> 544 ‘_open’, req) 545 if result: 546 return result

/anaconda3/lib/python3.6/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args) 502 for handler in handlers: 503 func = getattr(handler, meth_name) —> 504 result = func(*args) 505 if result is not None: 506 return result

/anaconda3/lib/python3.6/urllib/request.py in https_open(self, req) 1359 def https_open(self, req): 1360 return self.do_open(http.client.HTTPSConnection, req, -> 1361 context=self._context, check_hostname=self._check_hostname) 1362 1363 https_request = AbstractHTTPHandler.do_request_

/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args) 1318 encode_chunked=req.has_header(‘Transfer-encoding’)) 1319 except OSError as err: # timeout error -> 1320 raise URLError(err) 1321 r = h.getresponse() 1322 except: URLError: «>

The text was updated successfully, but these errors were encountered:

Seemingly the uploading request took more than 30 seconds. The default timeout is the period of time. You can customize the value by timeout arg in the constructor.

So now it throws a URLError:
Is there a limit on file size?

Источник

If you need to make HTTP requests with Python, then you may find yourself directed to the brilliant requests library. Though it’s a great library, you may have noticed that it’s not a built-in part of Python. If you prefer, for whatever reason, to limit your dependencies and stick to standard-library Python, then you can reach for urllib.request!

In this tutorial, you’ll:

Learn how to make basic HTTP requests with urllib.request
Dive into the nuts and bolts of an HTTP message and how urllib.request represents it
Understand how to deal with character encodings of HTTP messages
Explore some common errors when using urllib.request and learn how to resolve them
Dip your toes into the world of authenticated requests with urllib.request
Understand why both urllib and the requests library exist and when to use one or the other

If you’ve heard of HTTP requests, including GET and POST, then you’re probably ready for this tutorial. Also, you should’ve already used Python to read and write to files, ideally with a context manager, at least once.

Ultimately, you’ll find that making a request doesn’t have to be a frustrating experience, although it does tend to have that reputation. Many of the issues that you tend to run into are due to the inherent complexity of this marvelous thing called the Internet. The good news is that the urllib.request module can help to demystify much of this complexity.

Basic HTTP GET Requests With `urllib.request`

Before diving into the deep end of what an HTTP request is and how it works, you’re going to get your feet wet by making a basic GET request to a sample URL. You’ll also make a GET request to a mock REST API for some JSON data. In case you’re wondering about POST Requests, you’ll be covering them later in the tutorial, once you have some more knowledge of urllib.request.

To get started, you’ll make a request to www.example.com, and the server will return an HTTP message. Ensure that you’re using Python 3 or above, and then use the urlopen() function from urllib.request:

>>>

>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> body[:15]
b'<!doctype html>'

In this example, you import urlopen() from urllib.request. Using the context manager with, you make a request and receive a response with urlopen(). Then you read the body of the response and close the response object. With that, you display the first fifteen positions of the body, noting that it looks like an HTML document.

There you are! You’ve successfully made a request, and you received a response. By inspecting the content, you can tell that it’s likely an HTML document. Note that the printed output of the body is preceded by b. This indicates a bytes literal, which you may need to decode. Later in the tutorial, you’ll learn how to turn bytes into a string, write them to a file, or parse them into a dictionary.

The process is only slightly different if you want to make calls to REST APIs to get JSON data. In the following example, you’ll make a request to {JSON}Placeholder for some fake to-do data:

>>>

>>> from urllib.request import urlopen
>>> import json
>>> url = "https://jsonplaceholder.typicode.com/todos/1"
>>> with urlopen(url) as response:
...     body = response.read()
...
>>> todo_item = json.loads(body)
>>> todo_item
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

In this example, you’re doing pretty much the same as in the previous example. But in this one, you import urllib.request and json, using the json.loads() function with body to decode and parse the returned JSON bytes into a Python dictionary. Voila!

If you’re lucky enough to be using error-free endpoints, such as the ones in these examples, then maybe the above is all that you need from urllib.request. Then again, you may find that it’s not enough.

Now, before doing some urllib.request troubleshooting, you’ll first gain an understanding of the underlying structure of HTTP messages and learn how urllib.request handles them. This understanding will provide a solid foundation for troubleshooting many different kinds of issues.

The Nuts and Bolts of HTTP Messages

To understand some of the issues that you may encounter when using urllib.request, you’ll need to examine how a response is represented by urllib.request. To do that, you’ll benefit from a high-level overview of what an HTTP message is, which is what you’ll get in this section.

Before the high-level overview, a quick note on reference sources. If you want to get into the technical weeds, the Internet Engineering Task Force (IETF) has an extensive set of Request for Comments (RFC) documents. These documents end up becoming the actual specifications for things like HTTP messages. RFC 7230, part 1: Message Syntax and Routing, for example, is all about the HTTP message.

If you’re looking for some reference material that’s a bit easier to digest than RFCs, then the Mozilla Developer Network (MDN) has a great range of reference articles. For example, their article on HTTP messages, while still technical, is a lot more digestible.

Now that you know about these essential sources of reference information, in the next section you’ll get a beginner-friendly overview of HTTP messages.

Understanding What an HTTP Message Is

In a nutshell, an HTTP message can be understood as text, transmitted as a stream of bytes, structured to follow the guidelines specified by RFC 7230. A decoded HTTP message can be as simple as two lines:

GET / HTTP/1.1
Host: www.google.com

This specifies a GET request at the root (/) using the HTTP/1.1 protocol. The one and only header required is the host, www.google.com. The target server has enough information to make a response with this information.

A response is similar in structure to a request. HTTP messages have two main parts, the metadata and the body. In the request example above, the message is all metadata with no body. The response, on the other hand, does have two parts:

HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
Server: gws
(... other headers ...)

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"
...

The response starts with a status line that specifies the HTTP protocol HTTP/1.1 and the status 200 OK. After the status line, you get many key-value pairs, such as Server: gws, representing all the response headers. This is the metadata of the response.

After the metadata, there’s a blank line, which serves as the divider between the headers and the body. Everything that follows the blank line makes up the body. This is the part that gets read when you’re using urllib.request.

You can assume that all HTTP messages follow these specifications, but it’s possible that some may break these rules or follow an older specification. It’s exceptionally rare for this to cause any issues, though. So, just keep it in the back of your mind in case you run into a strange bug!

In the next section, you’ll see how urllib.request deals with raw HTTP messages.

Understanding How `urllib.request` Represents an HTTP Message

The main representation of an HTTP message that you’ll be interacting with when using urllib.request is the HTTPResponse object. The urllib.request module itself depends on the low-level http module, which you don’t need to interact with directly. You do end up using some of the data structures that http provides, though, such as HTTPResponse and HTTPMessage.

When you make a request with urllib.request.urlopen(), you get an HTTPResponse object in return. Spend some time exploring the HTTPResponse object with pprint() and dir() to see all the different methods and properties that belong to it:

>>>

>>> from urllib.request import urlopen
>>> from pprint import pprint
>>> with urlopen("https://www.example.com") as response:
...     pprint(dir(response))
...

To reveal the output of this code snippet, click to expand the collapsible section below:

['__abstractmethods__',
 '__class__',
 '__del__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__enter__',
 '__eq__',
 '__exit__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__iter__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__next__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '_abc_impl',
 '_checkClosed',
 '_checkReadable',
 '_checkSeekable',
 '_checkWritable',
 '_check_close',
 '_close_conn',
 '_get_chunk_left',
 '_method',
 '_peek_chunked',
 '_read1_chunked',
 '_read_and_discard_trailer',
 '_read_next_chunk_size',
 '_read_status',
 '_readall_chunked',
 '_readinto_chunked',
 '_safe_read',
 '_safe_readinto',
 'begin',
 'chunk_left',
 'chunked',
 'close',
 'closed',
 'code',
 'debuglevel',
 'detach',
 'fileno',
 'flush',
 'fp',
 'getcode',
 'getheader',
 'getheaders',
 'geturl',
 'headers',
 'info',
 'isatty',
 'isclosed',
 'length',
 'msg',
 'peek',
 'read',
 'read1',
 'readable',
 'readinto',
 'readinto1',
 'readline',
 'readlines',
 'reason',
 'seek',
 'seekable',
 'status',
 'tell',
 'truncate',
 'url',
 'version',
 'will_close',
 'writable',
 'write',
 'writelines']

That’s a lot of methods and properties, but you’ll only end up using a handful of these . Apart from .read(), the important ones usually involve getting information about the headers.

One way to inspect all the headers is to access the .headers attribute of the HTTPResponse object. This will return an HTTPMessage object. Conveniently, you can treat an HTTPMessage like a dictionary by calling .items() on it to get all the headers as tuples:

>>>

>>> with urlopen("https://www.example.com") as response:
...     pass
...
>>> response.headers
<http.client.HTTPMessage object at 0x000001E029D9F4F0>
>>> pprint(response.headers.items())
[('Accept-Ranges', 'bytes'),
 ('Age', '398424'),
 ('Cache-Control', 'max-age=604800'),
 ('Content-Type', 'text/html; charset=UTF-8'),
 ('Date', 'Tue, 25 Jan 2022 12:18:53 GMT'),
 ('Etag', '"3147526947"'),
 ('Expires', 'Tue, 01 Feb 2022 12:18:53 GMT'),
 ('Last-Modified', 'Thu, 17 Oct 2019 07:18:26 GMT'),
 ('Server', 'ECS (nyb/1D16)'),
 ('Vary', 'Accept-Encoding'),
 ('X-Cache', 'HIT'),
 ('Content-Length', '1256'),
 ('Connection', 'close')]

Now you have access to all the response headers! You probably won’t need most of this information, but rest assured that some applications do use it. For example, your browser might use the headers to read the response, set cookies, and determine an appropriate cache lifetime.

There are convenience methods to get the headers from an HTTPResponse object because it’s quite a common operation. You can call .getheaders() directly on the HTTPResponse object, which will return exactly the same list of tuples as above. If you’re only interested in one header, say the Server header, then you can use the singular .getheader("Server") on HTTPResponse or use the square bracket ([]) syntax on .headers from HTTPMessage:

>>>

>>> response.getheader("Server")
'ECS (nyb/1D16)'
>>> response.headers["Server"]
'ECS (nyb/1D16)'

Truth be told, you probably won’t need to interact with the headers directly like this. The information that you’re most likely to need will probably already have some built-in helper methods, but now you know, in case you ever need to dig deeper!

Closing an `HTTPResponse`

The HTTPResponse object has a lot in common with the file object. The HTTPResponse class inherits from the IOBase class, as do file objects, which means that you have to be mindful of opening and closing.

In simple programs, you’re not likely to notice any issues if you forget to close HTTPResponse objects. For more complex projects, though, this can significantly slow execution and cause bugs that are difficult to pinpoint.

Problems arise because input/output (I/O) streams are limited. Each HTTPResponse requires a stream to be kept clear while it’s being read. If you never close your streams, this will eventually prevent any other stream from being opened, and it might interfere with other programs or even your operating system.

So, make sure you close your HTTPResponse objects! For your convenience, you can use a context manager, as you’ve seen in the examples. You can also achieve the same result by explicitly calling .close() on the response object:

>>>

>>> from urllib.request import urlopen
>>> response = urlopen("https://www.example.com")
>>> body = response.read()
>>> response.close()

In this example, you don’t use a context manager, but instead close the response stream explicitly. The above example still has an issue, though, because an exception may be raised before the call to .close(), preventing the proper teardown. To make this call unconditional, as it should be, you can use a try … except block with both an else and a finally clause:

>>>

>>> from urllib.request import urlopen
>>> response = None
>>> try:
...     response = urlopen("https://www.example.com")
... except Exception as ex:
...     print(ex)
... else:
...     body = response.read()
... finally:
...     if response is not None:
...         response.close()

In this example, you achieve an unconditional call to .close() by using the finally block, which will always run regardless of exceptions raised. The code in the finally block first checks if the response object exists with is not None, and then closes it.

That said, this is exactly what a a context manager does, and the with syntax is generally preferred. Not only is the with syntax less verbose and more readable, but it also protects you from pesky errors of omission. Put another way, it’s a far better guard against accidentally forgetting to close the object:

>>>

>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     response.read(50)
...     response.read(50)
...
b'<!doctype html>n<html>n<head>n    <title>Example D'
b'omain</title>nn    <meta charset="utf-8" />n    <m'

In this example, you import urlopen() from the urllib.request module. You use the with keyword with .urlopen() to assign the HTTPResponse object to the variable response. Then, you read the first fifty bytes of the response and then read the following fifty bytes, all within the with block. Finally, you close the with block, which executes the request and runs the lines of code within its block.

With this code, you cause two sets of fifty bytes each to be displayed. The HTTPResponse object will close once you exit the with block scope, meaning that the .read() method will only return empty bytes objects:

>>>

>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
...     response.read(50)
...
b'<!doctype html>n<html>n<head>n    <title>Example D'
>>> response.read(50)
b''

In this example, the second call to read fifty bytes is outside the with scope. Being outside the with block means that HTTPResponse is closed, even though you can still access the variable. If you try to read from HTTPResponse when it’s closed, it’ll return an empty bytes object.

Another point to note is that you can’t reread a response once you’ve read all the way to the end:

>>>

>>> import urllib.request
>>> with urllib.request.urlopen("https://www.example.com") as response:
...     first_read = response.read()
...     second_read = response.read()
...
>>> len(first_read)
1256
>>> len(second_read)
0

This example shows that once you’ve read a response, you can’t read it again. If you’ve fully read the response, the subsequent attempt just returns an empty bytes object even though the response isn’t closed. You’d have to make the request again.

In this regard, the response is different from a file object, because with a file object, you can read it multiple times by using the .seek() method, which HTTPResponse doesn’t support. Even after closing a response, you can still access the headers and other metadata, though.

Exploring Text, Octets, and Bits

In most of the examples so far, you read the response body from HTTPResponse, displayed the resulting data immediately, and noted that it was displayed as a bytes object. This is because text information in computers isn’t stored or transmitted as letters, but as bytes!

A raw HTTP message sent over the wire is broken up into a sequence of bytes, sometimes referred to as octets. Bytes are 8-bit chunks. For example, 01010101 is a byte. To learn more about binary, bits, and bytes, check out Bitwise Operators in Python.

So how do you represent letters with bytes? A byte has 256 potential combinations, and you can assign a letter to each combination. You can assign 00000001 to A, 00000010 to B, and so on. ASCII character encoding, which is quite common, uses this type of system to encode 128 characters, which is enough for a language like English. This is particularly convenient because just one byte can represent all the characters, with space to spare.

All the standard English characters, including capitals, punctuation, and numerals, fit within ASCII. On the other hand, Japanese is thought to have around fifty thousand logographic characters, so 128 characters won’t cut it! Even the 256 characters that are theoretically available within one byte wouldn’t be nearly enough for Japanese. So, to accomodate all the world’s languages there are many different systems to encode characters.

Even though there are many systems, one thing you can rely on is the fact that they’ll always be broken up into bytes. Most servers, if they can’t resolve the MIME type and character encoding, default to application/octet-stream, which literally means a stream of bytes. Then whoever receives the message can work out the character encoding.

Dealing With Character Encodings

Problems often arise because, as you may have guessed, there are many, many different potential character encodings. The dominant character encoding today is UTF-8, which is an implementation of Unicode. Luckily, ninety-eight percent of web pages today are encoded in UTF-8!

UTF-8 is dominant because it can efficiently handle a mind-boggling number of characters. It handles all the 1,112,064 potential characters defined by Unicode, encompassing Chinese, Japanese, Arabic (with right-to-left scripts), Russian, and many more character sets, including emojis!

UTF-8 remains efficient because it uses a variable number of bytes to encode characters, which means that for many characters, it only requires one byte, while for others it can require up to four bytes.

While UTF-8 is dominant, and you usually won’t go wrong with assuming UTF-8 encodings, you’ll still run into different encodings all the time. The good news is that you don’t need to be an expert on encodings to handle them when using urllib.request.

Going From Bytes to Strings

When you use urllib.request.urlopen(), the body of the response is a bytes object. The first thing you may want to do is to convert the bytes object to a string. Perhaps you want to do some web scraping. To do this, you need to decode the bytes. To decode the bytes with Python, all you need to find out is the character encoding used. Encoding, especially when referring to character encoding, is often referred to as a character set.

As mentioned, ninety-eight percent of the time, you’ll probably be safe defaulting to UTF-8:

>>>

>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> decoded_body = body.decode("utf-8")
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>

In this example, you take the bytes object returned from response.read() and decode it with the bytes object’s .decode() method, passing in utf-8 as an argument. When you print decoded_body, you can see that it’s now a string.

That said, leaving it up to chance is rarely a good strategy. Fortunately, headers are a great place to get character set information:

>>>

>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'utf-8'
>>> decoded_body = body.decode(character_set)
>>> print(decoded_body[:30])
<!doctype html>
<html>
<head>

In this example, you call .get_content_charset() on the .headers object of response and use that to decode. This is a convenience method that parses the Content-Type header so that you can painlessly decode bytes into text.

Going From Bytes to File

If you want to decode bytes into text, now you’re good to go. But what if you want to write the body of a response into a file? Well, you have two options:

Write the bytes directly to the file
Decode the bytes into a Python string, and then encode the string back into a file

The first method is the most straightforward, but the second method allows you to change the encoding if you want to. To learn about file manipulation in more detail, take a look at Real Python’s Reading and Writing Files in Python (Guide).

To write the bytes directly to a file without having to decode, you’ll need the built-in open() function, and you’ll need to ensure that you use write binary mode:

>>>

>>> from urllib.request import urlopen
>>> with urlopen("https://www.example.com") as response:
...     body = response.read()
...
>>> with open("example.html", mode="wb") as html_file:
...     html_file.write(body)
...
1256

Using open() in wb mode bypasses the need to decode or encode and dumps the bytes of the HTTP message body into the example.html file. The number that’s output after the writing operation indicates the number of bytes that have been written. That’s it! You’ve written the bytes directly to a file without encoding or decoding anything.

Now say you have a URL that doesn’t use UTF-8, but you want to write the contents to a file with UTF-8. For this, you’d first decode the bytes into a string and then encode the string into a file, specifying the character encoding.

Google’s home page seems to use different encodings depending on your location. In much of Europe and the US, it uses the ISO-8859-1 encoding:

>>>

>>> from urllib.request import urlopen
>>> with urlopen("https://www.google.com") as response:
...     body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> character_set
'ISO-8859-1'
>>> content = body.decode(character_set)
>>> with open("google.html", encoding="utf-8", mode="w") as file:
...     file.write(content)
...
14066

In this code, you got the response character set and used it to decode the bytes object into a string. Then you wrote the string to a file, encoding it using UTF-8.

Once you’ve written to a file, you should be able to open the resulting file in your browser or text editor. Most modern text processors can detect the character encoding automatically.

If there are encoding errors and you’re using Python to read a file, then you’ll likely get an error:

>>>

>>> with open("encoding-error.html", mode="r", encoding="utf-8") as file:
...     lines = file.readlines()
...
UnicodeDecodeError:
    'utf-8' codec can't decode byte

Python explicitly stops the process and raises an exception, but in a program that displays text, such as the browser where you’re viewing this page, you may find the infamous replacement characters:

A Replacement Character

The black rhombus with a white question mark (�), the square (□), and the rectangle (▯) are often used as replacements for characters which couldn’t be decoded.

Sometimes, decoding seems to work but results in unintelligible sequences, such as æ–‡å—åŒ–ã‘., which also suggests the wrong character set was used. In Japan, they even have a word for text that’s garbled due to character encoding issues, Mojibake, because these issues plagued them at the start of the Internet age.

With that, you should now be equipped to write files with the raw bytes returned from urlopen(). In the next section, you’ll learn how to parse bytes into a Python dictionary with the json module.

Going From Bytes to Dictionary

For application/json responses, you’ll often find that they don’t include any encoding information:

>>>

>>> from urllib.request import urlopen
>>> with urlopen("https://httpbin.org/json") as response:
...     body = response.read()
...
>>> character_set = response.headers.get_content_charset()
>>> print(character_set)
None

In this example, you use the json endpoint of httpbin, a service that allows you to experiment with different types of requests and responses. The json endpoint simulates a typical API that returns JSON data. Note that the .get_content_charset() method returns nothing in its response.

Even though there’s no character encoding information, all is not lost. According to RFC 4627, the default encoding of UTF-8 is an absolute requirement of the application/json specification. That’s not to say that every single server plays by the rules, but generally, you can assume that if JSON is being transmitted, it’ll almost always be encoded using UTF-8.

Fortunately, json.loads() decodes byte objects under the hood and even has some leeway in terms of different encodings that it can deal with. So, json.loads() should be able to cope with most bytes objects that you throw at it, as long as they’re valid JSON:

>>>

>>> import json
>>> json.loads(body)
{'slideshow': {'author': 'Yours Truly', 'date': 'date of publication', 'slides'
: [{'title': 'Wake up to WonderWidgets!', 'type': 'all'}, {'items': ['Why <em>W
onderWidgets</em> are great', 'Who <em>buys</em> WonderWidgets'], 'title': 'Ove
rview', 'type': 'all'}], 'title': 'Sample Slide Show'}}

As you can see, the json module handles the decoding automatically and produces a Python dictionary. Almost all APIs return key-value information as JSON, although you might run into some older APIs that work with XML. For that, you might want to look into the Roadmap to XML Parsers in Python.

With that, you should know enough about bytes and encodings to be dangerous! In the next section, you’ll learn how to troubleshoot and fix a couple of common errors that you might run into when using urllib.request.

Common `urllib.request` Troubles

There are many kinds of issues you can run into on the world wild web, whether you’re using urllib.request or not. In this section, you’ll learn how to deal with a couple of the most common errors when getting started out: 403 errors and TLS/SSL certificate errors. Before looking at these specific errors, though, you’ll first learn how to implement error handling more generally when using urllib.request.

Implementing Error Handling

Before you turn your attention to specific errors, boosting your code’s ability to gracefully deal with assorted errors will pay off. Web development is plagued with errors, and you can invest a lot of time in handling errors sensibly. Here, you’ll learn to handle HTTP, URL, and timeout errors when using urllib.request.

HTTP status codes accompany every response in the status line. If you can read a status code in the response, then the request reached its target. While this is good, you can only consider the request a complete success if the response code starts with a 2. For example, 200 and 201 represent successful requests. If the status code is 404 or 500, for example, something went wrong, and urllib.request will raise an HTTPError.

Sometimes mistakes happen, and the URL provided isn’t correct, or a connection can’t be made for another reason. In these cases, urllib.request will raise a URLError.

Finally, sometimes servers just don’t respond. Maybe your network connection is slow, the server is down, or the server is programmed to ignore specific requests. To deal with this, you can pass a timeout argument to urlopen() to raise a TimeoutError after a certain amount of time.

The first step in handling these exceptions is to catch them. You can catch errors produced within urlopen() with a try … except block, making use of the HTTPError, URLError, and TimeoutError classes:

# request.py

from urllib.error import HTTPError, URLError
from urllib.request import urlopen

def make_request(url):
    try:
        with urlopen(url, timeout=10) as response:
            print(response.status)
            return response.read(), response
    except HTTPError as error:
        print(error.status, error.reason)
    except URLError as error:
        print(error.reason)
    except TimeoutError:
        print("Request timed out")

The function make_request() takes a URL string as an argument, tries to get a response from that URL with urllib.request, and catches the HTTPError object that’s raised if an error occurs. If the URL is bad, it’ll catch a URLError. If it goes through without any errors, it’ll just print the status and return a tuple containing the body and the response. The response will close after return.

The function also calls urlopen() with a timeout argument, which will cause a TimeoutError to be raised after the seconds specified. Ten seconds is generally a good amount of time to wait for a response, though as always, much depends on the server that you need to make the request to.

Now you’re set up to gracefully handle a variety of errors, including but not limited to the errors that you’ll cover next.

Dealing With `403` Errors

You’ll now use the make_request() function to make some requests to httpstat.us, which is a mock server used for testing. This mock server will return responses that have the status code you request. If you make a request to https://httpstat.us/200, for example, you should expect a 200 response.

APIs like httpstat.us are used to ensure that your application can handle all the different status codes it might encounter. httpbin also has this functionality, but httpstat.us has a more comprehensive selection of status codes. It even has the infamous and semi-official 418 status code that returns the message I’m a teapot!

To interact with the make_request() function that you wrote in the previous section, run the script in interactive mode:

With the -i flag, this command will run the script in interactive mode. This means that it’ll execute the script and then open the Python REPL afterward, so you can now call the function that you just defined:

>>>

>>> make_request("https://httpstat.us/200")
200
(b'200 OK', <http.client.HTTPResponse object at 0x0000023D612660B0>)
>>> make_request("https://httpstat.us/403")
403 Forbidden

Here you tried the 200 and 403 endpoints of httpstat.us. The 200 endpoint goes through as anticipated and returns the body of the response and the response object. The 403 endpoint just printed the error message and didn’t return anything, also as expected.

The 403 status means that the server understood the request but won’t fulfill it. This is a common error that you can run into, especially while web scraping. In many cases, you can solve it by passing a User-Agent header.

One of the primary ways that servers identify who or what is making the request is by examining the User-Agent header. The raw default request sent by urllib.request is the following:

GET https://httpstat.us/403 HTTP/1.1
Accept-Encoding: identity
Host: httpstat.us
User-Agent: Python-urllib/3.10
Connection: close

Notice that User-Agent is listed as Python-urllib/3.10. You may find that some sites will try to block web scrapers, and this User-Agent is a dead giveaway. With that said, you can set your own User-Agent with urllib.request, though you’ll need to modify your function a little:

 # request.py

 from urllib.error import HTTPError, URLError
-from urllib.request import urlopen
+from urllib.request import urlopen, Request

-def make_request(url):
+def make_request(url, headers=None):
+    request = Request(url, headers=headers or {})
     try:
-        with urlopen(url, timeout=10) as response:
+        with urlopen(request, timeout=10) as response:
             print(response.status)
             return response.read(), response
     except HTTPError as error:
         print(error.status, error.reason)
     except URLError as error:
         print(error.reason)
     except TimeoutError:
         print("Request timed out")

To customize the headers that you send out with your request, you first have to instantiate a Request object with the URL. Additionally, you can pass in a keyword argument of headers, which accepts a standard dictionary representing any headers you wish to include. So, instead of passing the URL string directly into urlopen(), you pass this Request object which has been instantiated with the URL and headers.

To use this revamped function, restart the interactive session, then call make_request() with a dictionary representing the headers as an argument:

>>>

>>> body, response = make_request(
...     "https://www.httpbin.org/user-agent",
...     {"User-Agent": "Real Python"}
... )
200
>>> body
b'{n  "user-agent": "Real Python"n}n'

In this example, you make a request to httpbin. Here you use the user-agent endpoint to return the request’s User-Agent value. Because you made the request with a custom user agent of Real Python, this is what gets returned.

Some servers are strict, though, and will only accept requests from specific browsers. Luckily, it’s possible to find standard User-Agent strings on the web, including through a user agent database. They’re just strings, so all you need to do is copy the user agent string of the browser that you want to impersonate and use it as the value of the User-Agent header.

Fixing the SSL `CERTIFICATE_VERIFY_FAILED` Error

Another common error is due to Python not being able to access the required security certificate. To simulate this error, you can use some mock sites that have known bad SSL certificates, provided by badssl.com. You can make a request to one of them, such as superfish.badssl.com, and experience the error firsthand:

>>>

>>> from urllib.request import urlopen
>>> urlopen("https://superfish.badssl.com/")
Traceback (most recent call last):
  (...)
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  (...)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

Here, making a request to an address with a known bad SSL certificate will result in CERTIFICATE_VERIFY_FAILED which is a type of URLError.

SSL stands for Secure Sockets Layer. This is something of a misnomer because SSL was deprecated in favor of TLS, Transport Layer Security. Sometimes old terminology just sticks! It’s a way to encrypt network traffic so that a hypothetical listener can’t eavesdrop on the information transmitted over the wire.

These days, most website addresses are preceded not by http:// but by https://, with the s standing for secure. HTTPS connections must be encrypted through the TLS. urllib.request can handle both HTTP and HTTPS connections.

The details of HTTPS are far beyond the scope of this tutorial, but you can think of an HTTPS connection as involving two stages, the handshake and the transfer of information. The handshake ensures that the connection is secure. For more information about Python and HTTPS, check out Exploring HTTPS With Python.

To establish that a particular server is secure, programs that make requests rely on a store of trusted certificates. The server’s certificate is verified during the handshake stage. Python uses the operating system’s store of certificates. If Python can’t find the system’s store of certificates, or if the store is out of date, then you’ll run into this error.

Sometimes the store of certificates that Python can access is out of date, or Python can’t reach it, for whatever reason. This is frustrating because you can sometimes visit the URL from your browser, which thinks that it’s secure, yet urllib.request still raises this error.

You may be tempted to opt out of verifying the certificate, but this will render your connection insecure and is definitely not recommended:

>>>

>>> import ssl
>>> from urllib.request import urlopen
>>> unverified_context = ssl._create_unverified_context()
>>> urlopen("https://superfish.badssl.com/", context=unverified_context)
<http.client.HTTPResponse object at 0x00000209CBE8F220>

Here you import the ssl module, which allows you to create an unverified context. You can then pass this context to urlopen() and visit a known bad SSL certificate. The connection successfully goes through because the SSL certificate isn’t checked.

Before resorting to these desperate measures, try updating your OS or updating your Python version. If that fails, then you can take a page from the requests library and install certifi:

Windows
Linux + macOS

PS> python -m venv venv
PS> .venvScriptsactivate
(venv) PS> python -m pip install certifi

$ python3 -m venv venv
$ source venv/bin/activate.sh
(venv) $ python3 -m pip install certifi

certifi is a collection of certificates that you can use instead of your system’s collection. You do this by creating an SSL context with the certifi bundle of certificates instead of the OS’s bundle:

>>>

>>> import ssl
>>> from urllib.request import urlopen
>>> import certifi
>>> certifi_context = ssl.create_default_context(cafile=certifi.where())
>>> urlopen("https://sha384.badssl.com/", context=certifi_context)
<http.client.HTTPResponse object at 0x000001C7407C3490>

In this example, you used certifi to act as your SSL certificate store, and you used it to successfully connect to a site with a known good SSL certificate. Note that instead of ._create_unverified_context(), you use .create_default_context().

This way, you can stay secure without too much trouble! In the next section, you’ll be dipping your toes into the world of authentication.

Authenticated Requests

Authentication is a vast subject, and if you’re dealing with authentication much more complicated than what’s covered here, this might be a good jumping-off point into the requests package.

In this tutorial, you’ll only cover one authentication method, which serves as an example of the type of adjustments that you have to make to authenticate your requests. urllib.request does have a lot of other functionality that helps with authentication, but that won’t be covered in this tutorial.

One of the most common authentication tools is the bearer token, specified by RFC 6750. It’s often used as part of OAuth, but can also be used in isolation. It’s also most common to see as a header, which you can use with your current make_request() function:

>>>

>>> token = "abcdefghijklmnopqrstuvwxyz"
>>> headers = {
...     "Authorization": f"Bearer {token}"
... }
>>> make_request("https://httpbin.org/bearer", headers)
200
(b'{n  "authenticated": true, n  "token": "abcdefghijklmnopqrstuvwxyz"n}n',
<http.client.HTTPResponse object at 0x0000023D612642E0>)

In this example, you make a request to the httpbin /bearer endpoint, which simulates bearer authentication. It’ll accept any string as a token. It only requires the proper format specified by RFC 6750. The name has to be Authorization, or sometimes the lowercase authorization, and the value has to be Bearer, with a single space between that and the token.

Congratulations, you’ve successfully authenticated, using a bearer token!

Another form of authentication is called Basic Access Authentication, which is a very simple method of authentication, only slightly better than sending a username and password in a header. It’s very insecure!

One of the most common protocols in use today is OAuth (Open Authorization). If you’ve ever used Google, GitHub, or Facebook to sign into another website, then you’ve used OAuth. The OAuth flow generally involves a few requests between the service that you want to interact with and an identity server, resulting in a short-lived bearer token. This bearer token can then be used for a period of time with bearer authentication.

Much of authentication comes down to understanding the specific protocol that the target server uses and reading the documentation closely to get it working.

POST Requests With `urllib.request`

You’ve made a lot of GET requests, but sometimes you want to send information. That’s where POST requests come in. To make POST requests with urllib.request, you don’t have to explicitly change the method. You can just pass a data object to a new Request object or directly to urlopen(). The data object must be in a special format, though. You’ll adapt your make_request() function slightly to support POST requests by adding the data parameter:

 # request.py

 from urllib.error import HTTPError, URLError
 from urllib.request import urlopen, Request

-def make_request(url, headers=None):
+def make_request(url, headers=None, data=None):

-    request = Request(url, headers=headers or {})
+    request = Request(url, headers=headers or {}, data=data)
     try:
         with urlopen(request, timeout=10) as response:
             print(response.status)
             return response.read(), response
     except HTTPError as error:
         print(error.status, error.reason)
     except URLError as error:
         print(error.reason)
     except TimeoutError:
         print("Request timed out")

Here you just modified the function to accept a data argument with a default value of None, and you passed that right into the Request instantiation. That’s not all that needs to be done, though. You can use one of two different formats to execute a POST request:

Form Data: application/x-www-form-urlencoded
JSON: application/json

The first format is the oldest format for POST requests and involves encoding the data with percent encoding, also known as URL encoding. You may have noticed key-value pairs URL encoded as a query string. Keys are separated from values with an equal sign (=), key-value pairs are separated with an ampersand (&), and spaces are generally suppressed but can be replaced with a plus sign (+).

If you’re starting off with a Python dictionary, to use the form data format with your make_request() function, you’ll need to encode twice:

Once to URL encode the dictionary
Then again to encode the resulting string into bytes

For the first stage of URL encoding, you’ll use another urllib module, urllib.parse. Remember to start your script in interactive mode so that you can use the make_request() function and play with it on the REPL:

>>>

>>> from urllib.parse import urlencode
>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> url_encoded_data = urlencode(post_dict)
>>> url_encoded_data
'Title=Hello+World&Name=Real+Python'
>>> post_data = url_encoded_data.encode("utf-8")
>>> body, response = make_request(
...     "https://httpbin.org/anything", data=post_data
... )
200
>>> print(body.decode("utf-8"))
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "Name": "Real Python",
    "Title": "Hello World"
  },
  "headers": {
    "Accept-Encoding": "identity",
    "Content-Length": "34",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/3.10",
    "X-Amzn-Trace-Id": "Root=1-61f25a81-03d2d4377f0abae95ff34096"
  },
  "json": null,
  "method": "POST",
  "origin": "86.159.145.119",
  "url": "https://httpbin.org/anything"
}

In this example, you:

Import urlencode() from the urllib.parse module
Initialize your POST data, starting with a dictionary
Use the urlencode() function to encode the dictionary
Encode the resulting string into bytes using UTF-8 encoding
Make a request to the anything endpoint of httpbin.org
Print the UTF-8 decoded response body

UTF-8 encoding is part of the specification for the application/x-www-form-urlencoded type. UTF-8 is used preemptively to decode the body because you already know that httpbin.org reliably uses UTF-8.

The anything endpoint from httpbin acts as a sort of echo, returning all the information it received so that you can inspect the details of the request you made. In this case, you can confirm that method is indeed POST, and you can see that the data you sent is listed under form.

To make the same request with JSON, you’ll turn a Python dictionary into a JSON string with json.dumps(), encode it with UTF-8, pass it as the data argument, and finally add a special header to indicate that the data type is JSON:

>>>

>>> post_dict = {"Title": "Hello World", "Name": "Real Python"}
>>> import json
>>> json_string = json.dumps(post_dict)
>>> json_string
'{"Title": "Hello World", "Name": "Real Python"}'
>>> post_data = json_string.encode("utf-8")
>>> body, response = make_request(
...     "https://httpbin.org/anything",
...     data=post_data,
...     headers={"Content-Type": "application/json"},
... )
200
>>> print(body.decode("utf-8"))
{
  "args": {},
  "data": "{"Title": "Hello World", "Name": "Real Python"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "identity",
    "Content-Length": "47",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/3.10",
    "X-Amzn-Trace-Id": "Root=1-61f25a81-3e35d1c219c6b5944e2d8a52"
  },
  "json": {
    "Name": "Real Python",
    "Title": "Hello World"
  },
  "method": "POST",
  "origin": "86.159.145.119",
  "url": "https://httpbin.org/anything"
}

To serialize the dictionary this time around, you use json.dumps() instead of urlencode(). You also explicitly add the Content-Type header with a value of application/json. With this information, the httpbin server can deserialize the JSON on the receiving end. In its response, you can see the data listed under the json key.

With that, you can now start making POST requests. This tutorial won’t go into more detail about the other request methods, such as PUT. Suffice to say that you can also explicitly set the method by passing a method keyword argument to the instantiation of the Request object.

The Request Package Ecosystem

To round things out, this last section of the tutorial is dedicated to clarifying the package ecosystem around HTTP requests with Python. Because there are many packages, with no clear standard, it can be confusing. That said, there are use cases for each package, which just means more choice for you!

What Are `urllib2` and `urllib3`?

To answer this question, you need to go back to early Python, all the way back to version 1.2, when the original urllib was introduced. Around version 1.6, a revamped urllib2 was added, which lived alongside the original urllib. When Python 3 came along, the original urllib was deprecated, and urllib2 dropped the 2, taking on the original urllib name. It also split into parts:

urllib.error
urllib.parse
urllib.request
urllib.response
urllib.robotparser

So what about urllib3? That’s a third-party library developed while urllib2 was still around. It’s not related to the standard library because it’s an independently maintained library. Interestingly, the requests library actually uses urllib3 under the hood, and so does pip!

When Should I Use `requests` Over `urllib.request`?

The main answer is ease of use and security. urllib.request is considered a low-level library, which exposes a lot of the detail about the workings of HTTP requests. The Python documentation for urllib.request makes no bones about recommending requests as a higher-level HTTP client interface.

If you interact with many different REST APIs, day in and day out, then requests is highly recommended. The requests library bills itself as “built for human beings” and has successfully created an intuitive, secure, and straightforward API around HTTP. It’s usually considered the go-to library! If you want to know more about the requests library, check out the Real Python guide to requests.

An example of how requests makes things easier is when it comes to character encoding. You’ll remember that with urllib.request, you have to be aware of encodings and take a few steps to ensure an error-free experience. The requests package abstracts that away and will resolve the encoding by using chardet, a universal character encoding detector, just in case there’s any funny business.

If your goal is to learn more about standard Python and the details of how it deals with HTTP requests, then urllib.request is a great way to get into that. You could even go further and use the very low-level http modules. On the other hand, you may just want to keep dependencies to a minimum, which urllib.request is more than capable of.

Why Is `requests` Not Part of the Standard Library?

Maybe you’re wondering why requests isn’t part of core Python by this point.

This is a complex issue, and there’s no hard and fast answer to it. There are many speculations as to why, but two reasons seem to stand out:

requests has other third-party dependencies that would need to be integrated too.
requests needs to stay agile and can do this better outside the standard library.

The requests library has third-party dependencies. Integrating requests into the standard library would mean also integrating chardet, certifi, and urllib3, among others. The alternative would be to fundamentally change requests to use only Python’s existing standard library. This is no trivial task!

Integrating requests would also mean that the existing team that develops this library would have to relinquish total control over the design and implementation, giving way to the PEP decision-making process.

HTTP specifications and recommendations change all the time, and a high-level library has to be agile enough to keep up. If there’s a security exploit to be patched, or a new workflow to add, the requests team can build and release far more quickly than they could as part of the Python release process. There have supposedly been times when they’ve released a security fix twelve hours after a vulnerability was discovered!

For an interesting overview of these issues and more, check out Adding Requests to The Standard Library, which summarizes a discussion at the Python Language Summit with Kenneth Reitz, the creator and maintainer of Requests.

Because this agility is so necessary to requests and its underlying urllib3, the paradoxical statement that requests is too important for the standard library is often used. This is because so much of the Python community depends on requests and its agility that integrating it into core Python would probably damage it and the Python community.

On the GitHub repository issues board for requests, an issue was posted, asking for the inclusion of requests in the standard library. The developers of requests and urllib3 chimed in, mainly saying they would likely lose interest in maintaining it themselves. Some even said they would fork the repositories and continue developing them for their own use cases.

With that said, note that the requests library GitHub repository is hosted under the Python Software Foundation’s account. Just because something isn’t part of the Python standard library doesn’t mean that it’s not an integral part of the ecosystem!

It seems that the current situation works for both the Python core team and the maintainers of requests. While it may be slightly confusing for newcomers, the existing structure gives the most stable experience for HTTP requests.

It’s also important to note that HTTP requests are inherently complex. urllib.request doesn’t try to sugarcoat that too much. It exposes a lot of the inner workings of HTTP requests, which is why it’s billed as a low-level module. Your choice of requests versus urllib.request really depends on your particular use case, security concerns, and preference.

Conclusion

You’re now equipped to use urllib.request to make HTTP requests. Now you can use this built-in module in your projects, keeping them dependency-free for longer. You’ve also gained the in-depth understanding of HTTP that comes from using a lower-level module, such as urllib.request.

In this tutorial, you’ve:

Learned how to make basic HTTP requests with urllib.request
Explored the nuts and bolts of an HTTP message and studied how it’s represented by urllib.request
Figured out how to deal with character encodings of HTTP messages
Explored some common errors when using urllib.request and learned how to resolve them
Dipped your toes into the world of authenticated requests with urllib.request
Understood why both urllib and the requests library exist and when to use one or the other

You’re now in a position to make basic HTTP requests with urllib.request, and you also have the tools to dive deeper into low-level HTTP terrain with the standard library. Finally, you can choose whether to use requests or urllib.request, depending on what you want or need. Have fun exploring the Web!

Источник

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

Closed

praveen2593 opened this issue

Jul 1, 2020

· 10 comments

Comments

Description

Hi Folks, I am running into URLError <urlopen error The write operation timed out> when uploading CSV file of large size (~200MB). The file is eventually uploaded into slack, yet the code breaks with that error. Can anyone explain why that’s happening?

What type of issue is this? (place an `x` in one of the `[ ]`)

bug
enhancement (feature request)
question
documentation related
testing related
discussion

Requirements (place an `x` in each of the `[ ]`)

I’ve read and understood the Contributing guidelines and have done my best effort to follow them.
I’ve read and agree to the Code of Conduct.
I’ve searched for any related issues and avoided creating a duplicate issue.

Bug Report

Filling out the following details about bugs will help us solve your issue sooner.

Reproducible in:

slackclient version: 2.7.1

python version: 3.6.5

OS version(s): macOS 10.14.6

Steps to reproduce:

Create a webclient with the necessary permissions
Create/Load a dataframe with size ~200MB as df (the one I tested with had a shape of (5400000, 6))
use the files_upload method to load files using the command as shown web_client.files_upload(channels = 'CXXXXXXXXXX', initial_comment="something", content=df.to_csv(index=False), filename="test.csv")

Expected result:

What you expected to happen: File sent with response ‘ok’:true

Actual result:

What actually happened:

---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1317                 h.request(req.get_method(), req.selector, req.data, headers,
-> 1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error

~/anaconda3/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1238         """Send a complete request to the server."""
-> 1239         self._send_request(method, url, body, headers, encode_chunked)
   1240

~/anaconda3/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1284             body = _encode(body, 'body')
-> 1285         self.endheaders(body, encode_chunked=encode_chunked)
   1286

~/anaconda3/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1233             raise CannotSendHeader()
-> 1234         self._send_output(message_body, encode_chunked=encode_chunked)
   1235

~/anaconda3/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked)
   1064                         + b'rn'
-> 1065                 self.send(chunk)
   1066

~/anaconda3/lib/python3.6/http/client.py in send(self, data)
    985         try:
--> 986             self.sock.sendall(data)
    987         except TypeError:

~/anaconda3/lib/python3.6/ssl.py in sendall(self, data, flags)
    971                 while count < amount:
--> 972                     v = self.send(byte_view[count:])
    973                     count += v

~/anaconda3/lib/python3.6/ssl.py in send(self, data, flags)
    940                     self.__class__)
--> 941             return self._sslobj.write(data)
    942         else:

~/anaconda3/lib/python3.6/ssl.py in write(self, data)
    641         """
--> 642         return self._sslobj.write(data)
    643

timeout: The write operation timed out
During handling of the above exception, another exception occurred:                                                                                                                                                                                                     [74/639]

URLError                                  Traceback (most recent call last)
<ipython-input-16-e4c541ebf2f5> in <module>
----> 1 web_client.files_upload(channels = 'CXXXXXXXXXX', initial_comment="something", content=df.to_csv(index=False), filename="test.csv")

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/client.py in files_upload(self, file, content, **kwargs)
   1217         data = kwargs.copy()
   1218         data.update({"content": content})
-> 1219         return self.api_call("files.upload", data=data)
   1220
   1221     def groups_archive(self, *, channel: str, **kwargs) -> Union[Future, SlackResponse]:

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in api_call(self, api_method, http_verb, files, data, params, json, headers, auth)
    213                 return self._event_loop.run_until_complete(future)
    214         else:
--> 215             return self._sync_send(api_url=api_url, req_args=req_args)
    216
    217     def _get_url(self, api_method):

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _sync_send(self, api_url, req_args)
    346             files=files,
    347             json_body=_json,
--> 348             additional_headers=headers,
    349         )
    350

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _urllib_api_call(self, token, url, query_params, json_body, body_params, files, additional_headers)
    448                 url = f"{url}&{q}" if "?" in url else f"{url}?{q}"
    449
--> 450             response = self._perform_urllib_http_request(url=url, args=request_args)
    451             if response.get("body", None):
    452                 response_body_data: dict = json.loads(response["body"])

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    579         except Exception as err:
    580             self._logger.error(f"Failed to send a request to Slack API server: {err}")
--> 581             raise err
    582
    583     def _build_urllib_request_headers(

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    560
    561                 resp: HTTPResponse = urlopen(
--> 562                     req, context=self.ssl, timeout=self.timeout
    563                 )
    564                 charset = resp.headers.get_content_charset()

~/anaconda3/lib/python3.6/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    221     else:
    222         opener = _opener
--> 223     return opener.open(url, data, timeout)
    224
    225 def install_opener(opener):

~/anaconda3/lib/python3.6/urllib/request.py in open(self, fullurl, data, timeout)
    524             req = meth(req)
    525
--> 526         response = self._open(req, data)
    527
    528         # post-process response
~/anaconda3/lib/python3.6/urllib/request.py in _open(self, req, data)
    542         protocol = req.type
    543         result = self._call_chain(self.handle_open, protocol, protocol +
--> 544                                   '_open', req)
    545         if result:
    546             return result

~/anaconda3/lib/python3.6/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    502         for handler in handlers:
    503             func = getattr(handler, meth_name)
--> 504             result = func(*args)
    505             if result is not None:
    506                 return result

~/anaconda3/lib/python3.6/urllib/request.py in https_open(self, req)
   1359         def https_open(self, req):
   1360             return self.do_open(http.client.HTTPSConnection, req,
-> 1361                 context=self._context, check_hostname=self._check_hostname)
   1362
   1363         https_request = AbstractHTTPHandler.do_request_

~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error
-> 1320                 raise URLError(err)
   1321             r = h.getresponse()
   1322         except:

URLError: <urlopen error The write operation timed out>

So now it throws a URLError: <urlopen error [Errno 32] Broken pipe>
Is there a limit on file size?

---------------------------------------------------------------------------
BrokenPipeError                           Traceback (most recent call last)
~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1317                 h.request(req.get_method(), req.selector, req.data, headers,
-> 1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error

~/anaconda3/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1238         """Send a complete request to the server."""
-> 1239         self._send_request(method, url, body, headers, encode_chunked)
   1240

~/anaconda3/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1284             body = _encode(body, 'body')
-> 1285         self.endheaders(body, encode_chunked=encode_chunked)
   1286

~/anaconda3/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1233             raise CannotSendHeader()
-> 1234         self._send_output(message_body, encode_chunked=encode_chunked)
   1235

~/anaconda3/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked)
   1064                         + b'rn'
-> 1065                 self.send(chunk)
   1066

~/anaconda3/lib/python3.6/http/client.py in send(self, data)
    985         try:
--> 986             self.sock.sendall(data)
    987         except TypeError:

~/anaconda3/lib/python3.6/ssl.py in sendall(self, data, flags)
    971                 while count < amount:
--> 972                     v = self.send(byte_view[count:])
    973                     count += v

~/anaconda3/lib/python3.6/ssl.py in send(self, data, flags)
    940                     self.__class__)
--> 941             return self._sslobj.write(data)
    942         else:

~/anaconda3/lib/python3.6/ssl.py in write(self, data)
    641         """
--> 642         return self._sslobj.write(data)
    643

BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
<ipython-input-8-75bdb47149f2> in <module>
----> 1 test_slacker.upload_dataframe(data=df, filename="check_result.csv", msg="Adding Large DataFrame as CSV")

~/Documents/scratch_folder/slack_testing.py in upload_dataframe(self, data, msg, filename, filepath)
     98                     content=data.to_csv(index=False),
     99                     msg=msg,
--> 100                     filename=filename,
    101                 )
    102                 # TODO: Log this

~/Documents/scratch_folder/slack_testing.py in _upload_content(self, content, msg, filename)
     79         except Exception as err:
     80             # TODO: Log this error
---> 81             raise err
     82         # TODO: Log this
     83         print("File Uploaded")

~/Documents/scratch_folder/slack_testing.py in _upload_content(self, content, msg, filename)
     75                 initial_comment=msg,
     76                 content=content,
---> 77                 filename=filename,
     78             )
     79         except Exception as err:

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/client.py in files_upload(self, file, content, **kwargs)
   1217         data = kwargs.copy()
   1218         data.update({"content": content})
-> 1219         return self.api_call("files.upload", data=data)
   1220
   1221     def groups_archive(self, *, channel: str, **kwargs) -> Union[Future, SlackResponse]:

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in api_call(self, api_method, http_verb, files, data, params, json, headers, auth)
    213                 return self._event_loop.run_until_complete(future)
    214         else:
--> 215             return self._sync_send(api_url=api_url, req_args=req_args)
    216
    217     def _get_url(self, api_method):

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _sync_send(self, api_url, req_args)
    346             files=files,
    347             json_body=_json,
--> 348             additional_headers=headers,
    349         )
    350

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _urllib_api_call(self, token, url, query_params, json_body, body_params, files, additional_headers)
    448                 url = f"{url}&{q}" if "?" in url else f"{url}?{q}"
    449
--> 450             response = self._perform_urllib_http_request(url=url, args=request_args)
    451             if response.get("body", None):
    452                 response_body_data: dict = json.loads(response["body"])

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    579         except Exception as err:
    580             self._logger.error(f"Failed to send a request to Slack API server: {err}")
--> 581             raise err
    582
    583     def _build_urllib_request_headers(

~/.virtualenvs/scratch/lib/python3.6/site-packages/slack/web/base_client.py in _perform_urllib_http_request(self, url, args)
    560
    561                 resp: HTTPResponse = urlopen(
--> 562                     req, context=self.ssl, timeout=self.timeout
    563                 )
    564                 charset = resp.headers.get_content_charset()
~/anaconda3/lib/python3.6/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    221     else:
    222         opener = _opener
--> 223     return opener.open(url, data, timeout)
    224
    225 def install_opener(opener):

~/anaconda3/lib/python3.6/urllib/request.py in open(self, fullurl, data, timeout)
    524             req = meth(req)
    525
--> 526         response = self._open(req, data)
    527
    528         # post-process response

~/anaconda3/lib/python3.6/urllib/request.py in _open(self, req, data)
    542         protocol = req.type
    543         result = self._call_chain(self.handle_open, protocol, protocol +
--> 544                                   '_open', req)
    545         if result:
    546             return result

~/anaconda3/lib/python3.6/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    502         for handler in handlers:
    503             func = getattr(handler, meth_name)
--> 504             result = func(*args)
    505             if result is not None:
    506                 return result

~/anaconda3/lib/python3.6/urllib/request.py in https_open(self, req)
   1359         def https_open(self, req):
   1360             return self.do_open(http.client.HTTPSConnection, req,
-> 1361                 context=self._context, check_hostname=self._check_hostname)
   1362
   1363         https_request = AbstractHTTPHandler.do_request_

~/anaconda3/lib/python3.6/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1318                           encode_chunked=req.has_header('Transfer-encoding'))
   1319             except OSError as err: # timeout error
-> 1320                 raise URLError(err)
   1321             r = h.getresponse()
   1322         except:

URLError: <urlopen error [Errno 32] Broken pipe>

Is there a limit on file size?

It’s up to 1Gb in size but, when your operation to upload a file takes long, timeouts or network issues between Slack API and your end may occur much more often.

If you set run_async=True in the WebClient constructor, this API client internally uses aiohttp. In this mode, it runs asynchronously (you need to use async/await in your code). If the async way works more stably for your use case, switching to the async mode may be an option.

As your file size is huge, another way to handle would be to utilize yet another online storage platform (e.g., Google Drive, Box, etc.) and share the permalink of the file in Slack.

If you use files.remote.add API for sharing a file link in Slack, you can customize the appearance by adding a preview image in Slack (preview_image) and also can register text data (indexable_file_contents) for the file into Slack’s search engine index.

I’ll try the async mode, to see if that works. And about the remote sharing, is it possible to use aws s3 platform here (not sure if it’s considered as an online storage platform)?

You can use S3 as long as the files are accessible to your end-users. You don’t need to give a permission to Slack API servers.

@seratch is it possible to set the timeout to be «indefinite» i.e never timeout? If I use timeout=0 I get urllib.error.URLError: <urlopen error [Errno 115] Operation now in progress>, which can be resolved by setting timeout=0 to any other number besides 0, but I’d rather have an «indefinite» timeout where the upload would retry until it succeeds.

@Arszilla We’re not planning to implement such. Please set an extremely large timeout and implement your own retry logic.

Understandable. Right now I have a 600 second timeout but was curious if an «indefinite» setting was to be implemented.

Cheers @seratch

@Arszilla I’m glad to know you’ve found your solution around this issue. Can we close this issue now as it seems we don’t have anything further to discuss here?

@seratch Thanks. We can close it. I’m good on my end!! 👍

Источник

I have a problem where I can not connect to the update server. I haven’t changed any settings either to my router or freenas (that I am aware of anyway) and looking through the docs everything seems to be correct (but obviously isn’t) I can access the freenas control panel and the files without error.

The traceback is:-

Code:

Traceback (most recent call last):
File "/usr/local/www/freenasUI/../freenasUI/system/views.py", line 1448, in update_check
train=updateobj.get_train(),
File "/usr/local/www/freenasUI/../freenasUI/system/models.py", line 573, in get_train
trains = conf.AvailableTrains() or []
File "/usr/local/lib/freenasOS/Configuration.py", line 891, in AvailableTrains
fileref = self.TryGetNetworkFile(file=TRAIN_FILE, reason="FetchTrains")
File "/usr/local/lib/freenasOS/Configuration.py", line 670, in TryGetNetworkFile
raise url_exc
URLError: <urlopen error timed out>

Freenas 9.10.1-U4

TiA

dlavigne

Guest

Do you mean this?

My router is on 192.168.0.1

dlavigne

Guest

Yes. Can you resolve a name (eg ping google.com? If not, try using 8.8.8.8 as the nameserver.

Code:

[root@freenas ~]# ping google.com											   
PING google.com (213.105.64.217): 56 data bytes

dlavigne

Guest

It’s resolving. Is it also pinging? (your output is truncated after the resolution line)

Thats all I am getting, so I assume not

Changed nameserver to 8.8.8.8 and its still erroring when I check for updates

Code:

Traceback (most recent call last):
File "/usr/local/www/freenasUI/../freenasUI/system/views.py", line 1448, in update_check
train=updateobj.get_train(),
File "/usr/local/www/freenasUI/../freenasUI/system/models.py", line 573, in get_train
trains = conf.AvailableTrains() or []
File "/usr/local/lib/freenasOS/Configuration.py", line 952, in AvailableTrains
fileref = self.TryGetNetworkFile(file=TRAIN_FILE, reason="FetchTrains")
File "/usr/local/lib/freenasOS/Configuration.py", line 720, in TryGetNetworkFile
raise url_exc
URLError: <urlopen error [Errno 8] hostname nor servname provided, or not known>

dlavigne

Guest

If you’re not getting ping feedback when using 8.8.8.8, check the settings on the router. Also, check for firewall settings on the router and the client operating system. Something is blocking packets.

I turned off the router firewall, and there was no difference in the ‘ping’ results on either nameserver.

What other router settings need to be checked?

I will be away from the PC for the next couple of hours, so any apologies for any deley in getting back to you.

UPDATE:-

I can ping correctly from the router it self

Code:

PING 172.217.23.14 (172.217.23.14): 64 data bytes
Reply from 172.217.23.14: Bytes 64 Time=18.257 ms
Reply from 172.217.23.14: Bytes 64 Time=19.113 ms
Reply from 172.217.23.14: Bytes 64 Time=21.102 ms
Reply from 172.217.23.14: Bytes 64 Time=19.072 ms
Reply from 172.217.23.14: Bytes 64 Time=20.458 ms
Reply from 172.217.23.14: Bytes 64 Time=21.591 ms
Reply from 172.217.23.14: Bytes 64 Time=19.053 ms
Reply from 172.217.23.14: Bytes 64 Time=18.944 ms
Reply from 172.217.23.14: Bytes 64 Time=20.505 ms
Reply from 172.217.23.14: Bytes 64 Time=20.391 ms

--- 172.217.23.14 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 18.257/19.848/21.591 ms

so something is wrong with the freenas settings, but I haven’t a clue what or where.

Looking around the control panel I have found other errors. eg, there are ‘no discs’ on the freenas

but I can access the discs from the computer without any problem

What the *beep* has happened?

Looks like you are simply not routing. And with the other confusions, I’m curious how your system got broken.

Updated perfectly, I can check for updates without any error and see all of the discs.

I know I am loosing my memory, but freenas shouldn’t :confused:

I have no idea why it stopped working (It had been like that for several weeks) and then suddenly started working, so I can give no advice if anyone else has a similar problem.

… but now I can’t access files on the freenas server, nor can I check/adjust the sharing settings. Reboot of the server makes no difference. I have also manually updated, with no difference noted.

I’m having a similar issue. I just upgraded from 11-U1 to 11-U2 when I was receiving a traceback error trying to update via the web GUI.

I usually have the FreeNAS server connected to a domain controller/DNS server, but have also tried updating from the GUI with DNS set to 8.8.8.8 (google DNS) and either way I get the same error:

Code:

Update server could not be reached

<urlopen error timed out>


Traceback
Traceback (most recent call last):
File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.6/socket.py", line 722, in create_connection
raise err
File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "./freenasUI/system/views.py", line 1424, in update_check
train=updateobj.get_train(),
File "./freenasUI/system/models.py", line 616, in get_train
trains = conf.AvailableTrains() or []
File "/usr/local/lib/freenasOS/Configuration.py", line 1064, in AvailableTrains
fileref = self.TryGetNetworkFile(file=TRAIN_FILE, reason="FetchTrains")
File "/usr/local/lib/freenasOS/Configuration.py", line 787, in TryGetNetworkFile
raise url_exc
File "/usr/local/lib/freenasOS/Configuration.py", line 761, in TryGetNetworkFile
furl = opener.open(req, timeout=30)
File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 1346, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error timed out>

Is there a problem with the FreeNAS update server, or is this yet another bug?

Thanks,
Avery

When in doubt, run freenas-update -v check from the shell. If that fails, then you can try doing fetch http://update-master.ixsystems.com/FreeNAS/`manifest_util train`/LATEST and see how that does.

We had a problem, which was fixed a week or two ago, that resulted in the non-CDN server being the only one with the updates, and then it was timing out due to the load.

When in doubt, run freenas-update -v check from the shell. If that fails, then you can try doing fetch http://update-master.ixsystems.com/FreeNAS/`manifest_util train`/LATEST and see how that does.

Tried both these and both timed out — the first one with the urlopen error timed out error and the 2nd one with «Operation timed out»

Tried both of these using local DNS and Google DNS.

Thanks for the recommendations though.

I’m having this problem too… webGUI check for updates causes the following error:

Code:

Update server could not be reached

<urlopen error timed out>


Traceback
Traceback (most recent call last):
File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.6/socket.py", line 722, in create_connection
raise err
File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./freenasUI/system/views.py", line 1424, in update_check
train=updateobj.get_train(),
File "./freenasUI/system/models.py", line 625, in get_train
trains = conf.AvailableTrains() or []
File "/usr/local/lib/freenasOS/Configuration.py", line 1064, in AvailableTrains
fileref = self.TryGetNetworkFile(file=TRAIN_FILE, reason="FetchTrains")
File "/usr/local/lib/freenasOS/Configuration.py", line 787, in TryGetNetworkFile
raise url_exc
File "/usr/local/lib/freenasOS/Configuration.py", line 761, in TryGetNetworkFile
furl = opener.open(req, timeout=30)
File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 1346, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error timed out>

I tried upgrading from Freenas-11-U2 to Freenas-11-U3 via the cli command «freenas-update -v update» and then restarted and the update went smoothly. Still getting this problem. I don’t seem to have problems with dns resolution from the CLI, but I can’t figure out what the heck is causing this. Also can’t view the available plugins via the webGUI, where I note this error:

Code:

September 22 18:44:23 freenas.local collectd[8054]: aggregation plugin: Unable to read the current rate of "freenas.local/cpu-2/cpu-system".
September 22 18:44:23 freenas.local collectd[8054]: utils_vl_lookup: The user object callback failed with status 2.

My gut says they’re related, but I have no idea what the problem is. Any ideas?

I’ve been having all sorts of issues with U3, I wish I had stayed at U2. My guess is the developers are focusing all their resources on the new GUI in the 11.1 release… can’t say I blame them. Hope this stuff gets more reliable along with it.

Источник

Открывает URL-адрес и читает ответ данные.

Синтаксис:

from urllib.request import urlopen

resp = urlopen(url, data=None, [timeout, ]*, 
               cafile=None, capath=None, 
               cadefault=False, context=None)

Параметры:

url — URL-адрес,
data=None — дополнительные данные для отправки на сервер,
timeout — указывает тайм-аут в секундах для блокирующих операций,
cafile=None — файл, содержащий набор сертификатов для HTTPS запроса,
capath=None — каталог хешированных файлов сертификатов,
cadefault=False — игнорируется,
context=None — экземпляр ssl.SSLContext, описывающий различные параметры SSL.

Возвращаемое значение:

объект Response.

Описание:

Аргумент data должен быть объектом, определяющим дополнительные данные для отправки на сервер, или None, если такие данные не нужны. Смотрите более подробно в описании класса Request.

Модуль urllib.request использует HTTP/1.1 и включает заголовок Connection: close в свои HTTP-запросы.

Необязательный параметр тайм-аута timeout указывает тайм-аут в секундах для блокирующих операций, таких как попытка подключения (если не указан, будет использоваться глобальная настройка тайм-аута по умолчанию). На самом деле это работает только для HTTP, HTTPS и FTP-соединений.

Если указан аргумент context, это должен быть экземпляр ssl.SSLContext, описывающий различные параметры SSL. Для получения дополнительных сведений смотрите описание http.client.HTTPSConnection.

Необязательные параметры cafile и capath определяют набор доверенных сертификатов CA для запросов HTTPS. Аргумент cafile должен указывать на один файл, содержащий набор сертификатов CA, тогда как capath должен указывать на каталог хешированных файлов сертификатов.

Параметр cadefault игнорируется.

Функция urllib.request.urlopen всегда возвращает объект, который может работать как менеджер контекста и имеет свойства url, заголовки и статус. Для получения дополнительных сведений об этих свойствах смотрите описание объекта urllib.response.addinfourl.

Для URL-адресов HTTP и HTTPS эта функция возвращает слегка измененный объект http.client.HTTPResponse. В дополнение к трем новым методам, атрибут msg содержит ту же информацию, что и атрибут reason возвращаемый сервером, вместо заголовков ответа, как указано в документации для HTTPResponse.

Для URL-адресов FTP, файлов и данных и запросов, явно обрабатываемых устаревшими классами URLopener и FancyURLopener, эта функция возвращает объект urllib.response.addinfourl.

Функция urllib.request.urlopen вызывает URLError при ошибках протокола.

Обратите внимание

, что может быть возвращено None, если никакой обработчик не обрабатывает запрос (хотя установленный по умолчанию глобальный OpenerDirector использует UnknownHandler, чтобы этого никогда не происходило).

Кроме того, если обнаружены параметры прокси-сервера (например, когда установлена переменная среды *_proxy, например http_proxy), urllib.request.ProxyHandler устанавливается по умолчанию и обеспечивает обработку запросов через прокси.

Примеры открытия URL-адресов:

>>> import urllib.request
>>> response = urllib.request.urlopen('http://www.python.org/')
>>> print(response.read(100).decode('utf-8'))
# <!doctype html>
# <!--[if lt IE 7]>   <html class="no-js ie6 lt-ie7 lt-ie8 lt-ie9">   <![endif]-->
>>> response.status
# 200
>>> dict(response.headers)
# {'Connection': 'close', 'Content-Length': '49562', 'Server': 'nginx', 
# 'Content-Type': 'text/html; charset=utf-8', 'X-Frame-Options': 'DENY', 
# ...
# 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains'}
>>> response.headers['Content-Type']
# 'text/html; charset=utf-8'
>>> response.url
# 'https://www.python.org/'

В следующем примере используется метод POST.

Обратите внимание

, что вывод params из urlencode кодируется в байты, перед отправкой в функцию urlopen() как аргумент data:

>>> import urllib.request
>>> import urllib.parse
>>> data = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> data = data.encode('ascii')
>>> with urllib.request.urlopen("http://requestb.in/xrbl82xr", data) as f:
...     print(f.read().decode('utf-8'))

Вот пример запроса, в котором используется метод GET для получения URL-адреса, содержащего параметры:

>>> import urllib.request
>>> import urllib.parse
>>> params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> url = f"http://www.musi-cal.com/cgi-bin/query?{params}"
>>> with urllib.request.urlopen(url) as f:
...     print(f.read().decode('utf-8'))

Как обработать тайм-аут `urllib.request.urlopen()` в Python?

import urllib.request, socket, urllib.error

url = 'https://www.usembassy.gov/'

try:
    response = urllib.request.urlopen(url, timeout=0.05)
except (urllib.error.HTTPError, urllib.error.URLError) as error:
    print(f'Data of not retrieved because {error}nURL: {url}')
except socket.timeout:
    print(f'socket timed out - URL {url}')
else:
    print(response.read().decode('utf-8'))

Источник

I am trying to install CDH5.3.1 offline in Oracle Enterprise Linux6u5. I have downloaded the parcels from archive.cloudera.com as following:

CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel

CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel.sha1 (and I rename it as CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel.sha)

manifest.json

then I changed it owner to be cloudera-scm:cloudera-scm.

After I logged in to the Cloudera Manager (version 5.4.0, which I installed by using tarball ) console, the server found the CDH parcels and showed

the download status as 100%, but when it started to distribute, error occurred as » <urlopen timed out>»

Here is the log of agent:

[28/Sep/2015 15:15:28 +0000] 5713 Thread-13 downloader INFO Starting download of: http://myhostname:7180/cmf/parcel/download/CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel
[28/Sep/2015 15:15:28 +0000] 5713 Thread-13 downloader INFO /opt/cloudera/parcel-cache/CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel.part exists, add Range header: bytes=0-
[28/Sep/2015 15:15:49 +0000] 5713 CP Server Thread-4 _cplogging INFO 10.182.xxx.xxx — — [28/Sep/2015:15:15:49] «GET /heartbeat HTTP/1.1» 200 2 «» «NING/1.0»
[28/Sep/2015 15:15:49 +0000] 5713 MainThread downloader INFO Aborting download request for http://myhostname:7180/cmf/parcel/download/CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel
[28/Sep/2015 15:15:58 +0000] 5713 Thread-13 downloader ERROR Unexpected exception during download
Traceback (most recent call last):
File «/opt/cloudera-manager/cm-5.4.0/lib64/cmf/agent/src/cmf/downloader.py», line 275, in _download
response = self.opener.fetch_url(request)
File «/opt/cloudera-manager/cm-5.4.0/lib64/cmf/agent/src/cmf/downloader.py», line 73, in fetch_url
return urllib2.urlopen(request)
File «/usr/lib64/python2.6/urllib2.py», line 126, in urlopen
return _opener.open(url, data, timeout)
File «/usr/lib64/python2.6/urllib2.py», line 391, in open
response = self._open(req, data)
File «/usr/lib64/python2.6/urllib2.py», line 409, in _open
‘_open’, req)
File «/usr/lib64/python2.6/urllib2.py», line 369, in _call_chain
result = func(*args)
File «/usr/lib64/python2.6/urllib2.py», line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File «/usr/lib64/python2.6/urllib2.py», line 1165, in do_open
raise URLError(err)
URLError: <urlopen error timed out>
[28/Sep/2015 15:15:58 +0000] 5713 Thread-13 parcel_cache INFO Distribution aborted during download, skipping unpack.

here is the disk and memory info:

[root@hostname init.d]# df -h /opt/cloudera/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg01-lv_root 71G 12G 56G 18% /

[root@hostname parcel-repo]# free
total used free shared buffers cached
Mem: 11782728 7458676 4324052 0 188300 4491224
-/+ buffers/cache: 2779152 9003576
Swap: 8388604 0 8388604

so, what’s the problem. I searched internet but did not got a solution.

thank you.

Источник

I have installed 3 different python script on my ubuntu 10.04 32 bit machine with python 2.6.5.

All of these use the urllib2 and I always get this error:

urllib2.URLError:
Why ?

Examples:

>>> import urllib2
>>> response = urllib2.urlopen("http://www.google.com")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1136, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>



>>> response = urllib2.urlopen("http://search.twitter.com/search.atom?q=hello&rpp=10&page=1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1136, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

UPDATE:

$ ping google.com
PING google.com (72.14.234.104) 56(84) bytes of data.
64 bytes from google.com (72.14.234.104): icmp_seq=1 ttl=54 time=25.3 ms
64 bytes from google.com (72.14.234.104): icmp_seq=2 ttl=54 time=24.6 ms
64 bytes from google.com (72.14.234.104): icmp_seq=3 ttl=54 time=25.1 ms
64 bytes from google.com (72.14.234.104): icmp_seq=4 ttl=54 time=25.0 ms
64 bytes from google.com (72.14.234.104): icmp_seq=5 ttl=54 time=23.9 ms
^C
--- google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 23.959/24.832/25.365/0.535 ms


$ w3m http://www.google.com
w3m: Can't load http://www.google.com.

$ telnet google.com 80
Trying 1.0.0.0...
telnet: Unable to connect to remote host: Connection timed out

UPDATE 2:

I am at home and I am using a router and an Access point :-. However I have just noticed that Firefox doesn’t work for me. But chrome, synaptic, etc does work.

UPDATE 3:

>>> useragent = 'Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Ubuntu/10.04 Chromium/6.0.472.62 Chrome/6.0.472.62 Safari/534.3)'
>>> request = urllib2.Request('http://www.google.com/')
>>> request.add_header('User-agent', useragent )
>>> urllib2.urlopen(request)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1136, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

Источник

urllib.error.URLError: #2115

COMPONENT NAME

Environment

ANSIBLE VERSION

BIGIP VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

urllib.error.URLError: #128