Urlopen error errno 11003 getaddrinfo failed

App Engine python27 urlfetch error: «[Errno 11003] getaddrinfo failed»

Part of Google Cloud Collective

I’m trying to switch my app engine app from python 2.5 to 2.7 and urlfetch.fetch() just doesn’t want to work (on the dev server at least; I haven’t deployed it yet). If I do something as simple as:

I get the following error:

Searching the internet has suggested it might be a firewall or proxy issue, but I turned off the firewall and I’m not behind a proxy.

This works fine in the python 2.5 dev server, but not the python 2.7 dev server. If I open a python 2.7 console and manually run socket.getaddrinfo(«www.google.com», 80) it works there as well.

Any ideas? Has anyone else encountered this?

1 Answer 1

Ok, I have been able to reproduce this issue and after a lot of going back and forth in git from a version that was working to a version that was not working I found that (at least in my case) the problem was in django-nonrel (because I updated all at once: django-nonrel and appengine).

So, please do the following: edit djangoappengine/settings_base.py and change

The real issue seems to be that in a previous version of django-nonrel, the import was actually written as: from google.appengine.api import api_proxy_stub_map (which was never found and the setup_env() was always called), so, when the import was fixed, the setup_env() was no longer called (which in my case made the error appear).

A 2nd note is that for some reason this was only happening when running inside of Eclipse/PyDev and it did work on the command line (although the environment variables seemed the same for me, so, I’m unsure why running from inside java actually made a difference).

Источник

urllib2.URLError:

even if I use another url, I get the same error.

I’m pretty sure there is no firewall running on my computer or router, and the internet (from a browser) works fine.

6 Answers 6

The problem, in my case, was that some install at some point defined an environment variable http_proxy on my machine when I had no proxy.

Removing the http_proxy environment variable fixed the problem.

The site’s DNS record is such that Python fails the DNS lookup in a peculiar way: it finds the entry, but zero associated IP addresses. (Verify with nslookup.) Hence, 11004, WSANO_DATA.

Prefix the site with ‘www.’ and try the request again. (Use nslookup to verify that its result is different, too.)

This fails essentially the same way with the Python Requests module:

requests.exceptions.ConnectionError: HTTPConnectionPool(host=’. ‘, port=80): Max retries exceeded with url: / (Caused by : [Errno 11004] getaddrinfo failed)

This may not help you if it’s a network-level issue but you can get some debugging info by setting debuglevel on httplib. Try this:

You probably need to use a proxy. Check your normal browser settings to find out which. Take a look at opening websites using urllib2 from behind corporate firewall — 11004 getaddrinfo failed for a similar problem with solution.,

Источник

Python web scraping : urllib.error.URLError: urlopen error [Errno 11001] getaddrinfo failed

This is the first time I am trying to use Python for Web scraping. I have to extract some information from a website. I work in an institution, so I am using a proxy for Internet access.

I have used this code. Which works fine with URLs like e.g. https://www.google.co.in, or https://www.pythonprogramming.net

But when I use this URL: http://www.genecards.org/cgi-bin/carddisp.pl?gene=APOA1 which I need for scraping data, it shows

Here is my code.

Please guide me on what the issue here which I am not able to understand. Also while searching for the above error, I read something about absolute URLs. Is that related to it?

2 Answers 2

The problem is that your proxy server, and your own host, seem to use two different DNS resolvers, or two resolvers updated at different instants in time.

So when you pass www.genecards.org , the proxy does not know that address, and the attempt to get address information (getAddrInfo) fails. Hence the error.

The problem is quite a bit more awkward than that, though. GeneCards.org is an alias for an Incapsula DNS host:

And that machine is itself a proxy, hiding the real GeneCards site behind (so you might use http://192.230.83.165/ as an address, and it would never work).

This kind of merry-go-round is used by those sites that, among other things — how shall I put it — take a dim view of being scraped:

So yes, you could try several things to make scraping work. Chances are that they will only work for a short time, before being shut down harder and harder. So in the best scenario, you would be forced to continuously update your scraping code. Which can, and will, break down whenever it’s most inconvenient to you.

This is no accident: it is intentional on GeneCards’ part, and clearly covered in their terms of service:

7.2 LifeMap may restrict, suspend or terminate the account of any Registered Users who abuses or misuses the GeneCards Suite Products. Misuse of the GeneCards Suite Products includes scraping, spidering and/or crawling GeneCards Suite Products; creating multiple or false profiles.

I suggest you take a different approach — try enquiring for a consultation license. Scraping a web site that does not care (or is unable, or hasn’t yet come around) to providing its information in a easier format is one thing — stealing that information is quite different.

Also, note that you’re connecting to a Squid proxy that in all probability is logging the username you’re using. Any scraping made through that proxy would immediately be traced back to that user, in the event that LifeMap files a complaint for unauthorized scraping.

Источник

не удается подключиться к *.onion сайтам с помощью python ( )

Я пытаюсь получить доступ к *.onion сайтам, используя python. Впрочем, успеха еще не было. Я прочитал много вопросов и ответов в stackoverflow, попробовал много разных способов решения этой проблемы: я пытался использовать Python 2.7 и Python 3.5 , пытался использовать urllib , urllib2 , requests (тогда я выяснил, что запросы не работают с носками), pysocks и т.д., но ничего не работает. Прямо сейчас я нахожусь в точке, где я получаю только следующую ошибку:

Нет, у меня нет брандмауэра, да, у меня хорошее подключение к Интернету, да, сайт существует. Я думаю, проблема в том, что это *.onion link.

Это то, что я делаю прямо сейчас:

и это то, что я получаю:

Я очень новичок во всем этом, поэтому я мог бы пропустить некоторые действительно простые части. Но я буду благодарен за любую помощь.

ps: при попытке доступа не к сайту *.onion, я получаю следующее:

Я на Linux, но код, который вы предоставили, не работал для меня. По внешнему виду это разрешение DNS не происходит над Tor (на основе ошибки 11001 WSAHOST_NOT_FOUND ). Я немного подозрительно, что он фактически использует Tor из-за ошибки 10061 (отказ в соединении).

В любом случае, я смог заставить его работать с этим:

PySocks говорит в своих документах:

Обратите внимание, что monkeypatching может не работать для всех стандартных модулей или для всех сторонних модулей и обычно не рекомендуется. Monkeypatching обычно является анти-шаблоном в Python.

socket.socket = socks.socksocket обезьян является использование socket.socket = socks.socksocket .

Если возможно, используйте Запросы с обработчиками протоколов socks5h:// для ваших прокси:

Источник

urllib.error.URLError: ？

Итак, мой код всего 4 строки. Я пытаюсь подключиться к веб-сайту, то, что я пытаюсь сделать после этого, не имеет значения, потому что ошибка возникла без других кодов.

И ошибка (кратко суммированная):

Вот что я пробовал.

1. Я искал возвращенную ошибку «Ошибка urlopen [Errno 11002]» в Google, и особенно в stackoverflow, ничего полезного не было возвращено (На самом деле вопросов об этой ошибке 11002 было не так много).
2. Затем я пытаюсь заменить аргумент веб-сайта (т. Е. «http: //python-data.dr -chuck.net/known_by_Fikret.html «) внутри функции urlopen с другим веб-сайтом» http : //www.pythonlearn.com/code/urllinks.py » . И это работало просто отлично. Ошибка не появляется.
3. Так что я думаю, что эта ошибка как-то связана с этим конкретным сайтом как таковым. Веб-сайт несколько динамичен, и я имею в виду, что его содержание будет меняться и превращаться в совершенно другую вещь. Но у меня нет больше знаний, чем просто описать то, что я видел.

И более длинная и полная версия ошибки:

2 ответа

Это означает, что либо ваша DNS-система не работает правильно, либо вы должны использовать прокси в своей сети, и она не определена правильно.

Если вам нужно использовать прокси-сервер, установите переменную среды HTTP_PROXY (и, необязательно, HTTPS_PROXY ) в правильной конфигурации для вашей сети. Формат: http://proxy.example.com:80 ; если вашему прокси-серверу нужны имя пользователя и пароль, вы должны передать его, например, так: http://username:password@proxy.example.com:80 .

При возникновении проблем с DNS попробуйте поискать домен из командной строки. Откройте командную строку и введите nslookup python-data.dr-chuck.net и посмотрите, вернет ли он вам IP-адрес.

Это проблема с DNS; по-видимому, ваша программа на Python не может разрешить имя хоста, которое вы указали. Может быть, DNS плохо настроен на хосте, на котором он работает?

Источник

The following is updated code which makes it easier to merge into
pandas/io/commons.py and call _urlopen with new params
I am not sure how to capture verify_ssl=False user input from read_csv to
allow user to bypass SSL cert verification.

import sys
is_py3 = sys.version_info[0] >= 3 # replace with 'compat.PY3'

## BEGIN SECTION modifications to pandas/io/common.py
import ssl
import base64


if is_py3:
    from urllib.parse import urlparse as parse_url
    from urllib.request import urlopen, build_opener, install_opener, 
            HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler, HTTPSHandler
    _urlopen = urlopen  
else:
    from urlparse import urlparse as parse_url
    from urllib2 import urlopen as _urlopen
    from urllib2 import Request
    from contextlib import contextmanager, closing  # noqa
    from functools import wraps  # noqa

    # @wraps(_urlopen)
    @contextmanager
    def urlopen(*args, **kwargs):
        with closing(_urlopen(*args, **kwargs)) as f:
            yield f


def split_uname_from_url(url_with_uname):
    o = parse_url( url_with_uname)
    usrch = '{}:{}@{}'.format( o.username, o.password, o.hostname)
    url_no_usrpwd = url_with_uname.replace( usrch , o.hostname)
    return o.username, o.password, url_no_usrpwd

def get_urlopen_args( url_with_uname, verify_ssl=True):
    uname, pwd, url_no_usrpwd = split_uname_from_url(url_with_uname)
    print('Calling [{}] -- uname:[{}] -- pwd[{}]'.format(url_no_usrpwd, uname, pwd))
    if is_py3:
        fn= get_urlopen_args_py3
    else:
        fn = get_urlopen_args_py2
    req, kwargs = fn( uname, pwd, url_no_usrpwd, verify_ssl=verify_ssl)
    return req, kwargs

def get_urlopen_args_py2( uname, pwd, url_no_usrpwd, verify_ssl=True):
    req = Request( url_no_usrpwd )
    base64string = base64.encodestring('{}:{}'.format(uname, pwd)).replace('n', '')
    req.add_header("Authorization", "Basic {}".format( base64string) )   
    # I hope pandas can support self signed certs too 
    kwargs = {}
    if not verify_ssl:
        kwargs['context'] = ssl._create_unverified_context()
    return req, kwargs

def get_urlopen_args_py3( uname, pwd, url_no_usrpwd, verify_ssl=True):
    passman = HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, url_no_usrpwd, uname, pwd)
    authhandler = HTTPBasicAuthHandler(passman)
    if verify_ssl:
        opener = build_opener(authhandler)
    else:
        context = ssl.create_default_context()
        context.check_hostname = False
        context.verify_mode = ssl.CERT_NONE
        opener = build_opener(authhandler, HTTPSHandler(context=context))
    install_opener(opener)
    return url_no_usrpwd, {}

## END SECTION modifications to pandas/io/common.py

def call_urlopen( url_with_uname, verify_ssl=False):
    # in get_filepath_or_buffer prior to calling _urlopen get params
    # not sure where to obtain verify_ssl from pd.read_csv 
    req, kwargs = get_urlopen_args(url_with_uname, verify_ssl) 
    resp = _urlopen(req , **kwargs)
    return resp_to_csv(resp)

    
def resp_to_csv(resp):
    csv = resp.read()
    if is_py3:
        csv = csv.decode('utf-8')
    return csv

    
def csv_to_df(csv_txt, **kwargs):
    '''
    @param csv_txt: text of csv rows.
    @param kwargs: to pass to pd.read_csv
    @return df
    '''
    import pandas as pd
    try: 
        from StringIO import StringIO #python2.7
    except:  
        from io import StringIO #python3.x. 
    buf = None
    df = None
    try:
        buf = StringIO(csv_txt)
        df = pd.read_csv(buf, **kwargs)
    finally:
        if buf:
            try:
                buf.close()
            except:
                pass
    return df

url_with_uname = 'https://pandasusr:pandaspwd@pandastest.mooo.com:5000/aaa.csv'
csv_txt = call_urlopen( url_with_uname, verify_ssl=False) 
df = csv_to_df( csv_txt.strip() ) 
print(df.to_string(index=False))

I am not quiet familiar with process involved around contributing to pandas — so please feel free to take over.

I downloaded the Client Downloader for Fusion 360. When I run the executuable, the client extracts itself, then starts running. I quickly get an «Unexpected Error» with a path to the log file. I looked at the log file and there seems to be an error with the scoket getting addr info…..

I tried disabling my virus protection, it didn’t help. Let me know if you have any ideas.

Also, if this is not in the right spot please move it.

2014-12-20 16:17:30,910 - root - DEBUG :: Logging initialized, with level 10, output file: C:UserstestAppDataLocalAutodeskautodesk.webdeploy.streamer.log
2014-12-20 16:17:30,910 - adsk.dls.streamer.manifests - DEBUG :: Loading build manifest from C:UserstestAppDataLocalTemp7ZipSfx.000resourcesbuildinfo.json
2014-12-20 16:17:30,910 - adsk.dls.streamer.manifests - DEBUG :: Type: <class '_io.TextIOWrapper'>, Members: ['_CHUNK_SIZE', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_checkClosed', '_checkReadable', '_checkSeekable', '_checkWritable', 'buffer', 'close', 'closed', 'detach', 'encoding', 'errors', 'fileno', 'flush', 'isatty', 'line_buffering', 'mode', 'name', 'newlines', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines']
2014-12-20 16:17:30,911 - adsk.dls.streamer.configuration - DEBUG :: Resource directory set to: C:UserstestAppDataLocalTemp7ZipSfx.000resources
2014-12-20 16:17:30,911 - __main__ - DEBUG :: Loading default language: en
2014-12-20 16:17:30,912 - __main__ - DEBUG :: Defining command line arguments
2014-12-20 16:17:30,913 - __main__ - DEBUG :: Overriding standard streams to capture output
2014-12-20 16:17:30,913 - __main__ - DEBUG :: Parsing command line arguments
2014-12-20 16:17:30,914 - __main__ - INFO :: Command-Line Argument Configuration:
Namespace(app_id='73e72ada57b7480280f7a6f4a289729f', connection=None, fulluninstall=False, globalinstall=False, headless=False, infofile=None, language='en', logconfig=None, logfile=None, loglevel=20, pipes=None, procfac=<function deploy_process at 0x0000000003275EA0>, purge=False, quietmode=False, stream='production')
-------------------------------------------------------------------------------
2014-12-20 16:17:30,914 - __main__ - DEBUG :: Resetting the standard streams
2014-12-20 16:17:30,914 - root - DEBUG :: Preparing to do final log configuration
2014-12-20 16:17:30,914 - adsk.dls.streamer.configuration - INFO :: Setting deploy stream to production
2014-12-20 16:17:30,914 - adsk.dls.streamer.configuration - INFO :: Setting target app id to 73e72ada57b7480280f7a6f4a289729f
2014-12-20 16:17:30,915 - adsk.dls.streamer.platform - INFO :: Instantiating windows platform object
2014-12-20 16:17:32,913 - adsk.dls.streamer.registry - INFO :: Failed to query database schema, initializing.
2014-12-20 16:17:33,070 - adsk.dls.streamer.registry - INFO :: Existing database is at revision 4
2014-12-20 16:17:33,071 - adsk.dls.streamer.process - INFO :: Pulling application manifest: 73e72ada57b7480280f7a6f4a289729f
2014-12-20 16:17:33,071 - adsk.dls.streamer.manifests - INFO :: Opening manifest at: https://dl.appstreaming.autodesk.com/production/67316f5e79bc48318aa5f7b6bb58243d/73e72ada57b7480280f...
2014-12-20 16:17:34,110 - adsk.dls.streamer.netlib - WARNING :: Problem perfroming network operation... Trying again:
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_eliburllibrequest.py", line 1248, in do_open
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_elibhttpclient.py", line 1065, in request
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_elibhttpclient.py", line 1103, in _send_request
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_elibhttpclient.py", line 1061, in endheaders
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_elibhttpclient.py", line 906, in _send_output
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_elibhttpclient.py", line 844, in send
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_elibhttpclient.py", line 1198, in connect
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_elibsocket.py", line 417, in create_connection
socket.gaierror: [Errno 11003] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "W:SRCNeutronInstallerStreamersrcadskdlsstreamernetlib.py", line 377, in __call__
  File "W:SRCNeutronInstallerStreamersrcadskdlsstreamermanifests.py", line 253, in _open
  File "W:SRCNeutronInstallerStreamersrcadskdlsstreamernetlib.py", line 347, in __call__
  File "W:SRCNeutronInstallerStreamersrcadskdlsstreamernetlib.py", line 91, in __init__
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_eliburllibrequest.py", line 469, in open
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_eliburllibrequest.py", line 487, in _open
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_eliburllibrequest.py", line 447, in _call_chain
  File "W:SRCNeutronInstallerStreamersrcadskdlsstreamernetlib.py", line 220, in https_open
  File "F:P4Neutron3PPYTHON3.3.5WIN64_full_eliburllibrequest.py", line 1251, in do_open
urllib.error.URLError: <urlopen error [Errno 11003] getaddrinfo failed>

I am trying to send one email using smtp server, but I am getting this below error.

gaierror                                  Traceback (most recent call last)
<ipython-input-17-f14b0f617ec8> in <module>
     14 
     15 try:
---> 16    smtpObj = smtplib.SMTP('nadim70.na@gmail.com',25)
     17    smtpObj.sendmail(sender, receivers, message)
     18    print ("Successfully sent email")
~anaconda3libsmtplib.py in __init__(self, host, port, local_hostname, timeout, source_address)
    249 
    250         if host:
--> 251             (code, msg) = self.connect(host, port)
    252             if code != 220:
    253                 self.close()
~anaconda3libsmtplib.py in connect(self, host, port, source_address)
    334         if self.debuglevel > 0:
    335             self._print_debug('connect:', (host, port))
--> 336         self.sock = self._get_socket(host, port, self.timeout)
    337         self.file = None
    338         (code, msg) = self.getreply()
~anaconda3libsmtplib.py in _get_socket(self, host, port, timeout)
    305             self._print_debug('connect: to', (host, port), self.source_address)
    306         return socket.create_connection((host, port), timeout,
--> 307                                         self.source_address)
    308 
    309     def connect(self, host='localhost', port=0, source_address=None):
~anaconda3libsocket.py in create_connection(address, timeout, source_address)
    705     host, port = address
    706     err = None
--> 707     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
    708         af, socktype, proto, canonname, sa = res
    709         sock = None
~anaconda3libsocket.py in getaddrinfo(host, port, family, type, proto, flags)
    750     # and socket type values to enum constants.
    751     addrlist = []
--> 752     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    753         af, socktype, proto, canonname, sa = res
    754         addrlist.append((_intenum_converter(af, AddressFamily),
gaierror: [Errno 11003] getaddrinfo failed

urllib.request.urlopen("http://username:password@......etc...", None)

urllib.error.URLError: urlopen error [Errno 11003] getaddrinfo failed

3 ответа

# create a password manager
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

# Add the username and password.
# If we knew the realm, we could use it instead of None.
top_level_url = "http://example.com/foo/"
password_mgr.add_password(None, top_level_url, username, password)

handler = urllib2.HTTPBasicAuthHandler(password_mgr)

# create "opener" (OpenerDirector instance)
opener = urllib2.build_opener(handler)

# use the opener to fetch a URL
opener.open(a_url)

# Install the opener.
# Now all calls to urllib2.urlopen use our opener.
urllib2.install_opener(opener)

import httplib2

h = httplib2.Http("/path/to/cache-directory")
h.add_credentials(username, password)
response, content = h.request(url)
assert response.status == 200