Urllib error httperror http error 401 unauthorized

My requirement is to download an abc.zip file from some website http://clientdownload.xyz.com/Documents/abc.zip

For this activity I have written a python script as follows:

    url_to_check = 'http://clientdownload.xyz.com/Documents/abc.zip'
    username = "user"
    password = "pwd"
    p = urllib2.HTTPPasswordMgrWithDefaultRealm()
    p.add_password(None, url_to_check, username, password)
    handler = urllib2.HTTPBasicAuthHandler(p)
    opener = urllib2.build_opener(handler)
    urllib2.install_opener(opener)
    zip_file = urllib2.urlopen(url_to_check).read()       
    file_name = 'somefile.zip'
    meta = zip_file.info()
    file_size = int(meta.getheaders("Content-Length")[0])
    print "Downloading: %s Bytes: %s" % (file_name, file_size)

    with open(file_name, 'wb') as dwn_file:
        dwn_file.write(zip_file.read())

Whereas I am getting the following errors when I run the script:

File «updateCheck.py», line 68, in check_update
zip_file = urllib2.urlopen(url_to_check).read() File «/usr/lib/python2.7/urllib2.py», line 126, in urlopen
return _opener.open(url, data, timeout) File «/usr/lib/python2.7/urllib2.py», line 406, in open
response = meth(req, response) File «/usr/lib/python2.7/urllib2.py», line 519, in http_response
‘http’, request, response, code, msg, hdrs) File «/usr/lib/python2.7/urllib2.py», line 444, in error
return self._call_chain(*args) File «/usr/lib/python2.7/urllib2.py», line 378, in _call_chain
result = func(*args) File «/usr/lib/python2.7/urllib2.py», line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 401: Unauthorized

I have given the user name and password properly but it throws unauthorized error.

When I tried to download it using wget link with -http-user and --ask-password options, I am able to download the file.

Also using the same script I am able to download files from other servers properly.

I ran this script to get more info:

import urllib2, re, time, sys

theurl='http://clientdownload.xxx.com/Documents/Forms/AllItems.aspx'

req = urllib2.Request(theurl)

try:
    handle = urllib2.urlopen(req)

except IOError, e:

    if hasattr(e, 'code'):

        if e.code != 401:
            print 'We got another error'
            print e.code
        else:
            print e.headers
            print e.headers['www-authenticate']

I got the following information:

Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
SPRequestGuid: 939bad00-40b7-49b9-bbbc-99d0267a1004
X-SharePointHealthScore: 0
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.6029
Date: Wed, 12 Feb 2014 13:14:19 GMT
Connection: close
Content-Length: 16

NTLM

My requirement is to download an abc.zip file from some website http://clientdownload.xyz.com/Documents/abc.zip

For this activity I have written a python script as follows:

    url_to_check = 'http://clientdownload.xyz.com/Documents/abc.zip'
    username = "user"
    password = "pwd"
    p = urllib2.HTTPPasswordMgrWithDefaultRealm()
    p.add_password(None, url_to_check, username, password)
    handler = urllib2.HTTPBasicAuthHandler(p)
    opener = urllib2.build_opener(handler)
    urllib2.install_opener(opener)
    zip_file = urllib2.urlopen(url_to_check).read()       
    file_name = 'somefile.zip'
    meta = zip_file.info()
    file_size = int(meta.getheaders("Content-Length")[0])
    print "Downloading: %s Bytes: %s" % (file_name, file_size)

    with open(file_name, 'wb') as dwn_file:
        dwn_file.write(zip_file.read())

Whereas I am getting the following errors when I run the script:

File «updateCheck.py», line 68, in check_update
zip_file = urllib2.urlopen(url_to_check).read() File «/usr/lib/python2.7/urllib2.py», line 126, in urlopen
return _opener.open(url, data, timeout) File «/usr/lib/python2.7/urllib2.py», line 406, in open
response = meth(req, response) File «/usr/lib/python2.7/urllib2.py», line 519, in http_response
‘http’, request, response, code, msg, hdrs) File «/usr/lib/python2.7/urllib2.py», line 444, in error
return self._call_chain(*args) File «/usr/lib/python2.7/urllib2.py», line 378, in _call_chain
result = func(*args) File «/usr/lib/python2.7/urllib2.py», line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 401: Unauthorized

I have given the user name and password properly but it throws unauthorized error.

When I tried to download it using wget link with -http-user and --ask-password options, I am able to download the file.

Also using the same script I am able to download files from other servers properly.

I ran this script to get more info:

import urllib2, re, time, sys

theurl='http://clientdownload.xxx.com/Documents/Forms/AllItems.aspx'

req = urllib2.Request(theurl)

try:
    handle = urllib2.urlopen(req)

except IOError, e:

    if hasattr(e, 'code'):

        if e.code != 401:
            print 'We got another error'
            print e.code
        else:
            print e.headers
            print e.headers['www-authenticate']

I got the following information:

Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
SPRequestGuid: 939bad00-40b7-49b9-bbbc-99d0267a1004
X-SharePointHealthScore: 0
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.6029
Date: Wed, 12 Feb 2014 13:14:19 GMT
Connection: close
Content-Length: 16

NTLM

getting this error as basic first app to connect to weather

py code

from flask import Flask, render_template, request

# import json to load JSON data to a python dictionary
import json

# urllib.request to make a request to api
import urllib.request

app = Flask(__name__)


@app.route('/', methods=['POST', 'GET'])
def weather():
    if request.method == 'POST':
        city = request.form['city']
    else:
        # for default name mathura
        city = 'mathura'

    # your API key will come here
    api = '74cbba5bab81bf66874f686f4d6232bd'

    # source contain json data from api
    source = urllib.request.urlopen(
        http: // api.openweathermap.org / data / 2.5 / forecast?id = 524901 & APPID = {74cbba5bab81bf66874f686f4d6232bd}


    # converting JSON data to a dictionary
    list_of_data = json.loads(source)

    # data for variable list_of_data
    data = {
        "country_code": str(list_of_data['sys']['country']),
        "coordinate": str(list_of_data['coord']['lon']) + ' '
                      + str(list_of_data['coord']['lat']),
        "temp": str(list_of_data['main']['temp']) + 'k',
        "pressure": str(list_of_data['main']['pressure']),
        "humidity": str(list_of_data['main']['humidity']),
    }
    print(data)
    return render_template('index.html', data=data)


if __name__ == '__main__':
    app.run(debug=True)

html;

<!DOCTYPE html>
<html lang="en" dir="ltr">

<head>
  <meta charset="utf-8">
  <title>weather</title>
  <!-- Latest compiled and minified CSS -->
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.0/css/bootstrap.min.css">

  <!-- jQuery library -->
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>

  <!-- Latest compiled JavaScript -->
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.0/js/bootstrap.min.js"></script>

</head>

<body>
  <nav class="row" style="background: #337ab7; color: white;">
    <h1 class="col-md-3 text-center">weather</h1>
  </nav>
  <br />
  <br />
  <center class="row">
    <form method="post" class="col-md-6 col-md-offset-3">
      <div class="input-group">
        <input type="text" class="form-control" name="city" placeholder="Search">
        <div class="input-group-btn">
          <button class="btn btn-primary" type="submit">
            <i class="glyphicon glyphicon-search"></i>
          </button>
        </div>
        <form>
  </center>
  <div class="row">
    {% if data.country_code and data.coordinate and data.temp and data.pressure and data.humidity %}
    <div class="col-md-6 col-md-offset-3">
      <h3>country code : {{data.country_code}}</h1>
        <h5>coordinate : {{data.coordinate}}</h5>
        <h5>temp : {{data.temp}}</h5>
        <h5>pressure : {{data.pressure}} </h5>
        <h5>humidity : {{data.humidity}}</h5>
    </div>
    {% endif %}
  </div>
</body>

</html>

not sure gonna search it now but taking so lnog to find working tut

КАК получить интернет-ресурсы с помощью пакета urllib

Author

Michael Foord

import urllib.request
with urllib.request.urlopen('http://python.org/') as response:
   html = response.read()

import shutil
import tempfile
import urllib.request

with urllib.request.urlopen('http://python.org/') as response:
    with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
        shutil.copyfileobj(response, tmp_file)

with open(tmp_file.name) as html:
    pass

import urllib.request

req = urllib.request.Request('http://www.voidspace.org.uk')
with urllib.request.urlopen(req) as response:
   the_page = response.read()

req = urllib.request.Request('ftp://example.com/')

import urllib.parse
import urllib.request

url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
          'location' : 'Northampton',
          'language' : 'Python' }

data = urllib.parse.urlencode(values)
data = data.encode('ascii') # data should be bytes
req = urllib.request.Request(url, data)
with urllib.request.urlopen(req) as response:
   the_page = response.read()

>>> import urllib.request
>>> import urllib.parse
>>> data = {}
>>> data['name'] = 'Somebody Here'
>>> data['location'] = 'Northampton'
>>> data['language'] = 'Python'
>>> url_values = urllib.parse.urlencode(data)
>>> print(url_values)  
name=Somebody+Here&language=Python&location=Northampton
>>> url = 'http://www.example.com/example.cgi'
>>> full_url = url + '?' + url_values
>>> data = urllib.request.urlopen(full_url)

>>> req = urllib.request.Request('http://www.pretend_server.org')
>>> try: urllib.request.urlopen(req)
... except urllib.error.URLError as e:
...     print(e.reason)      
...
(4, 'getaddrinfo failed')

# Table mapping response codes to messages; entries have the
# form {code: (shortmessage, longmessage)}.
responses = {
    100: ('Continue', 'Request received, please continue'),
    101: ('Switching Protocols',
          'Switching to new protocol; obey Upgrade header'),

    200: ('OK', 'Request fulfilled, document follows'),
    201: ('Created', 'Document created, URL follows'),
    202: ('Accepted',
          'Request accepted, processing continues off-line'),
    203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
    204: ('No Content', 'Request fulfilled, nothing follows'),
    205: ('Reset Content', 'Clear input form for further input.'),
    206: ('Partial Content', 'Partial content follows.'),

    300: ('Multiple Choices',
          'Object has several resources -- see URI list'),
    301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
    302: ('Found', 'Object moved temporarily -- see URI list'),
    303: ('See Other', 'Object moved -- see Method and URL list'),
    304: ('Not Modified',
          'Document has not changed since given time'),
    305: ('Use Proxy',
          'You must use proxy specified in Location to access this '
          'resource.'),
    307: ('Temporary Redirect',
          'Object moved temporarily -- see URI list'),

    400: ('Bad Request',
          'Bad request syntax or unsupported method'),
    401: ('Unauthorized',
          'No permission -- see authorization schemes'),
    402: ('Payment Required',
          'No payment -- see charging schemes'),
    403: ('Forbidden',
          'Request forbidden -- authorization will not help'),
    404: ('Not Found', 'Nothing matches the given URI'),
    405: ('Method Not Allowed',
          'Specified method is invalid for this server.'),
    406: ('Not Acceptable', 'URI not available in preferred format.'),
    407: ('Proxy Authentication Required', 'You must authenticate with '
          'this proxy before proceeding.'),
    408: ('Request Timeout', 'Request timed out; try again later.'),
    409: ('Conflict', 'Request conflict.'),
    410: ('Gone',
          'URI no longer exists and has been permanently removed.'),
    411: ('Length Required', 'Client must specify Content-Length.'),
    412: ('Precondition Failed', 'Precondition in headers is false.'),
    413: ('Request Entity Too Large', 'Entity is too large.'),
    414: ('Request-URI Too Long', 'URI is too long.'),
    415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
    416: ('Requested Range Not Satisfiable',
          'Cannot satisfy request range.'),
    417: ('Expectation Failed',
          'Expect condition could not be satisfied.'),

    500: ('Internal Server Error', 'Server got itself in trouble'),
    501: ('Not Implemented',
          'Server does not support this operation'),
    502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
    503: ('Service Unavailable',
          'The server cannot process the request due to a high load'),
    504: ('Gateway Timeout',
          'The gateway server did not receive a timely response'),
    505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
    }

>>> req = urllib.request.Request('http://www.python.org/fish.html')
>>> try:
...     urllib.request.urlopen(req)
... except urllib.error.HTTPError as e:
...     print(e.code)
...     print(e.read())  
...
404
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">nnn<html
  ...
  <title>Page Not Found</title>n
  ...

Python

3.11

• Unicode HOWTO

  1.12 В этом HOWTO рассматривается поддержка Python спецификации Unicode,представляющей текстовые данные,и объясняются различные проблемы,которые обычно возникают у людей.

• Преобразование в байты

  Противоположным методом bytes.decode()является str.encode(),который возвращает строку представления Unicode,закодированную запрошенной кодировкой.

• Подведение итогов

  Поэтому,если вы хотите быть готовым к HTTPError URLError,есть два основных подхода.

• Установка питоновых модулей (версия Legacy)

  Greg Ward Примечание Весь пакет distutils был устаревшим и будет удален в Python 3.12.

My public app is developed with Python API and uses Flask.

It successfully installs and I get and keep an offline access_token in my database with the scope [«write_products», «read_products», «read_script_tags», «write_script_tags»].

Now I want to use my new-found authenticated session to actually do something useful! The first order of business is inserting a script_tag on the customer’s shop.

So I do the following:

    def upsert_script_tag(self, src):
        script_reference={"event": "onload", "src": f"{src}"}
        print(f"UPSERTING SCRIPT TAG WITH '{script_reference}'")
        st=shopify.ScriptTag(script_reference)
        ret = st.save()
        if st.errors:
            print(f"ERROR: {st.errors.full_messages()}")
        print(f" + Return was :{ret}")

At which point I get the following in my log when it gets called:

 UPSERTING SCRIPT TAG WITH '{'event': 'onload', 'src': 'https://bsp.this-some-whaky-site.com/shopify/script_tag'}'
shopify_app     | Traceback (most recent call last):
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 286, in _open
shopify_app     |     http_response = self._handle_error(self._urlopen(request))
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 316, in _urlopen
shopify_app     |     return urllib.request.urlopen(request, timeout=self.timeout)
shopify_app     |   File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
shopify_app     |     return opener.open(url, data, timeout)
shopify_app     |   File "/usr/local/lib/python3.7/urllib/request.py", line 531, in open
shopify_app     |     response = meth(req, response)
shopify_app     |   File "/usr/local/lib/python3.7/urllib/request.py", line 641, in http_response
shopify_app     |     'http', request, response, code, msg, hdrs)
shopify_app     |   File "/usr/local/lib/python3.7/urllib/request.py", line 569, in error
shopify_app     |     return self._call_chain(*args)
shopify_app     |   File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
shopify_app     |     result = func(*args)
shopify_app     |   File "/usr/local/lib/python3.7/urllib/request.py", line 649, in http_error_default
shopify_app     |     raise HTTPError(req.full_url, code, msg, hdrs, fp)
shopify_app     | urllib.error.HTTPError: HTTP Error 401: Unauthorized
shopify_app     | 
shopify_app     | During handling of the above exception, another exception occurred:
shopify_app     | 
shopify_app     | Traceback (most recent call last):
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
shopify_app     |     return self.wsgi_app(environ, start_response)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
shopify_app     |     response = self.handle_exception(e)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
shopify_app     |     reraise(exc_type, exc_value, tb)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
shopify_app     |     raise value
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
shopify_app     |     response = self.full_dispatch_request()
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
shopify_app     |     rv = self.handle_user_exception(e)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
shopify_app     |     reraise(exc_type, exc_value, tb)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
shopify_app     |     raise value
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
shopify_app     |     rv = self.dispatch_request()
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
shopify_app     |     return self.view_functions[rule.endpoint](**req.view_args)
shopify_app     |   File "./shopify_app/shopify_bp/decorators.py", line 51, in decorated_function
shopify_app     |     return f(*args, **kwargs)
shopify_app     |   File "./shopify_app/shopify_bp/views.py", line 196, in settings_post
shopify_app     |     sm.upsert_script_tag(script_tag_url)
shopify_app     |   File "./fk/api/shopify/Shopify.py", line 159, in upsert_script_tag
shopify_app     |     script = st.save()
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/activeresource.py", line 824, in save
shopify_app     |     data=self.encode())
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 364, in post
shopify_app     |     return self._open('POST', path, headers=headers, data=data)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/shopify/base.py", line 23, in _open
shopify_app     |     self.response = super(ShopifyConnection, self)._open(*args, **kwargs)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 288, in _open
shopify_app     |     http_response = self._handle_error(err)
shopify_app     |   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 404, in _handle_error
shopify_app     |     raise UnauthorizedAccess(err)
shopify_app     | pyactiveresource.connection.UnauthorizedAccess: Response(code=401, body="b'{"errors":"[API] Invalid API key or access token (unrecognized login or wrong password)"}'", headers={'Date': 'Sun, 15 Sep 2019 22:17:42 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'close', 'Set-Cookie': '__cfduid=da23de8c***************0e609367701568585861; expires=Mon, 14-Sep-20 22:17:41 GMT; path=/; domain=.myshopify.com; HttpOnly', 'X-Sorting-Hat-PodId': '34', 'X-Sorting-Hat-ShopId': '5263130659', 'Referrer-Policy': 'origin-when-cross-origin', 'X-Frame-Options': 'DENY', 'X-ShopId': '5263130659', 'X-ShardId': '34', 'WWW-Authenticate': 'Basic Realm="Shopify API Authentication"', 'Strict-Transport-Security': 'max-age=7889238', 'X-Request-Id': 'cfb8e7c***************c-740f4136eced', 'X-Shopify-Stage': 'production', 'Content-Security-Policy': "default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://dme0ih8comzn4.cloudfront.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fscript_tags&source%5Bsection%5D=admin_api&source%5Buuid%5D=cfb8e7c6-0c90-4507-a9bc-740f4136eced", 'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Permitted-Cross-Domain-Policies': 'none', 'X-XSS-Protection': '1; mode=block; report=/xss-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fscript_tags&source%5Bsection%5D=admin_api&source%5Buuid%5D=cfb8e7c***************c-740f4136eced', 'X-Dc': 'gcp-us-central1,gcp-us-central1', 'set-cookie': '__cfduid=da23de8c***************0e609367701568585861; expires=Mon, 14-Sep-20 22:17:41 GMT; path=/; domain=.myshopify.com; HttpOnly', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'Server': 'cloudflare', 'CF-RAY': '51**********dfdf-FRA'}, msg="Unauthorized")

So how can I debug this? Is there some way I can get more information about why it was unauthorized?

https://www.merchbot.net