Error these packages do not match the hashes from the requirements file

Description When I try to install numpy==1.19.1 in a new conda environment : conda create -n test python==3.7.9 conda activate test pip install numpy==1.19.1 the following error pops up. Any help i...

@Jamsheeda-K

Description

When I try to install numpy==1.19.1 in a new conda environment :

  conda create -n test python==3.7.9
  conda activate test
  pip install numpy==1.19.1

the following error pops up. Any help is greatly appreciated.

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    numpy==1.19.1 from https://files.pythonhosted.org/packages/df/cb/7af5de0a4068ce314e13074a4f86afada7fa86a095489bbc80f5dabd4156/numpy-1.19.1-cp37-cp37m-win_amd64.whl#sha256=8343bf67c72e09cfabfab55ad4a43ce3f6bf6e6ced7acf70f45ded9ebb425055:
        Expected sha256 8343bf67c72e09cfabfab55ad4a43ce3f6bf6e6ced7acf70f45ded9ebb425055
             Got        288ddced999623301072f7083b3035e58cf9c9902ec2bd4e2c6499ff58f6452f

Expected behavior

successful installlation of numpy

pip version

21.1.3

Python version

3.7.9

OS

Windows 10

How to Reproduce

conda create -n test python==3.7.9
conda activate test
pip install numpy==1.19.1

Output

No response

Code of Conduct

  • I agree to follow the PSF Code of Conduct.

@pfmoore

The wheel from PyPI when I download it has the correct hash, which suggests that something is causing your download to result in the wrong wheel. That’s a network issue at your end, I’m afraid. It may indicate that your downloads are compromised or that some software like a proxy is corrupting them somehow.

❯ wget https://files.pythonhosted.org/packages/df/cb/7af5de0a4068ce314e13074a4f86afada7fa86a095489bbc80f5dabd4156/numpy-1.19.1-cp37-cp37m-win_amd64.whl
--2021-07-11 10:47:31--  https://files.pythonhosted.org/packages/df/cb/7af5de0a4068ce314e13074a4f86afada7fa86a095489bbc80f5dabd4156/numpy-1.19.1-cp37-cp37m-win_amd64.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 2a04:4e42:f::319, 151.101.61.63
Connecting to files.pythonhosted.org (files.pythonhosted.org)|2a04:4e42:f::319|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12929295 (12M) [application/octet-stream]
Saving to: 'numpy-1.19.1-cp37-cp37m-win_amd64.whl'

numpy-1.19.1-cp37-cp37m-win_a 100%[=================================================>]  12.33M  9.43MB/s    in 1.3s

2021-07-11 10:47:32 (9.43 MB/s) - 'numpy-1.19.1-cp37-cp37m-win_amd64.whl' saved [12929295/12929295]
❯ sha256sum.exe .numpy-1.19.1-cp37-cp37m-win_amd64.whl
8343bf67c72e09cfabfab55ad4a43ce3f6bf6e6ced7acf70f45ded9ebb425055 *.\numpy-1.19.1-cp37-cp37m-win_amd64.whl

@pfmoore

Update: I edited my previous comment, as it was incorrect.

@no-response

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don’t have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

@github-actions
github-actions
bot

locked as resolved and limited conversation to collaborators

Sep 24, 2021

Recently I landed a series of contributions to the Python package pip-tools:

  • Support URLs as packages #807
  • Fix –generate-hashes with bare VCS URLs #812
  • Fix pip-compile output for unsafe requirements #813
  • Warn when –generate-hashes output is uninstallable #814

pip-tools is a “set of command line tools to help you keep your pip-based [Python] packages fresh, even when you’ve pinned them.” My changes help the pip-compile --generate-hashes command work for more people.

This isn’t a lot of code in the grand scheme of things, but it’s the largest set of contributions I’ve made to a mainstream open source project, so this blog post is a celebration of me! 🎁💥🎉 yay. But it’s also a chance to talk about package manager security and open source contributions and stuff like that.

I’ll start high-level with “what are package managers” and work my way into the weeds, so feel free to jump in wherever you want.

What are package managers?

Package managers help us install software libraries and keep them up to date. If I want to load a URL and print the contents, I can add a dependency on a package like requests

$ echo 'requests' > requirements.txt
$ pip install -r requirements.txt
Collecting requests (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/8f/ea/140f18072bbcd81885a9490abb171792fd2961fd7f366be58396f4c6d634/requests-2.0.1-py2.py3-none-any.whl (439kB)
     |████████████████████████████████| 440kB 4.1MB/s
Installing collected packages: requests
Successfully installed requests-2.0.1

… and let requests do the heavy lifting:

>>> import requests
>>> requests.get('http://example.com').text
'<!doctype html>n<html>n<head>n    <title>Example Domain</title> ...'

But there’s a problem – if I install exactly the same package later, I might get a different result:

$ echo 'requests' > requirements.txt
$ pip install -r requirements.txt
Collecting requests (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
     |████████████████████████████████| 61kB 3.3MB/s
Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl (150kB)
     |████████████████████████████████| 153kB 10.6MB/s
Collecting idna<2.9,>=2.5 (from requests->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: certifi, urllib3, idna, chardet, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.2
<requirements.txt, pip install -r, import requests>

I got a different version of requests than last time, and I got some bonus dependencies (certifi, urllib3, idna, and chardet). Now my code might not do the same thing even though I did the same thing, which is not how anyone wants computers to work. (I’ve cheated a little bit here by showing the first example as though pip install had been run back in 2013.)

So the next step is to pin the versions of my dependencies and their dependencies, using a package like pip-tools:

$ echo 'requests' > requirements.in
$ pip-compile
$ cat requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
certifi==2019.3.9         # via requests
chardet==3.0.4            # via requests
idna==2.8                 # via requests
requests==2.22.0
urllib3==1.25.2           # via requests

(There are other options I could use instead, like pipenv or poetry. For now I still prefer pip-tools, for roughly the reasons laid out by Hynek Schlawack.)

Now when I run pip install -r requirements.txt I will always get the same version of requests, and the same versions of its dependencies, and my program will always do the same thing.

… just kidding.

The problem with pinning Python packages

Unfortunately pip-compile doesn’t quite lock down our dependencies the way we would hope! In Python land you don’t necessarily get the same version of a package by asking for the same version number. That’s because of binary wheels.

Up until 2015, it was possible to change a package’s contents on PyPI without changing the version number, simply by deleting the package and reuploading it. That no longer works, but there is still a loophole: you can delete and reupload binary wheels.

Wheels are a new-ish binary format for distributing Python packages, including any precompiled programs written in C (or other languages) used by the package. They speed up installs and avoid the need for users to have the right compiler environment set up for each package. C-based packages typically offer a bunch of wheel files for different target environments – here’s bcrypt’s wheel files for example.

So what happens if a package was originally released as source, and then the maintainer wants to add binary wheels for the same release years later? PyPI will allow it, and pip will happily install the new binary files. This is a deliberate design decision: PyPI has “made the deliberate choice to allow wheel files to be added to old releases, though, and advise folks to use –no-binary and build their own wheel files from source if that is a concern.”

That creates room for weird situations, like this case where wheel files were uploaded for the hiredis 0.2.0 package on August 16, 2018, three years after the source release on April 3, 2015. The package had been handed over without announcement from Jan-Erik Rediger to a new volunteer maintainer, ifduyue, who uploaded the binary wheels. ifduyue’s personal information on Github consists of: a new moon emoji; an upside down face emoji; the location “China”; and an image of Lanny from the show Lizzie McGuire with spirals for eyes. In a bug thread opened after ifduyue uploaded the new version of hiredis 0.2.0, Jan-Erik commented that users should “please double-check that the content is valid and matches the repository.”

ifduyue's user account on github.com

The problem is that I can’t do that, and most programmers can’t do that. We can’t just rebuild the wheel ourselves and expect it to match, because builds are not reproducible unless one goes to great lengths like Debian does. So verifying the integrity of an unknown binary wheel requires rebuilding the wheel, comparing a diff, and checking that all discrepancies are benign – a time-consuming and error-prone process even for those with the skills to do it.

So the story of hiredis looks a lot like a new open source developer volunteering to help out on a project and picking off some low-hanging fruit in the bug tracker, but it also looks a lot like an attacker using the perfect technique to distribute malware widely in the Python ecosystem without detection. I don’t know which one it is! As a situation it’s bad for us as users, and it’s not fair to ifduyue if in fact they’re a friendly newbie contributing to a project.

(Is the hacking paranoia warranted? I think so! As Dominic Tarr wrote after inadvertently handing over control of an npm package to a bitcoin-stealing operation, “I’ve shared publish rights with other people before. … open source is driven by sharing! It’s great! it worked really well before bitcoin got popular.”)

This is a big problem with a lot of dimensions. It would be great if PyPI packages were all fully reproducible and checked to verify correctness. It would be great if PyPI didn’t let you change package contents after the fact. It would be great if everyone ran their own private package index and only added packages to it that they had personally built from source that they personally checked, the way big companies do it. But in the meantime, we can bite off a little piece of the problem by adding hashes to our requirements file. Let’s see how that works.

Adding hashes to our requirements file

Instead of just pinning packages like we did before, let’s try adding hashes to them:

$ echo 'requests==2.0.1' > requirements.in
$ pip-compile --generate-hashes
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
requests==2.0.1 
    --hash=sha256:8cfddb97667c2a9edaf28b506d2479f1b8dc0631cbdcd0ea8c8864def59c698b 
    --hash=sha256:f4ebc402e0ea5a87a3d42e300b76c292612d8467024f45f9858a8768f9fb6f6e

Now when pip-compile pins our package versions, it also fetches the currently-known hashes for each requirement and adds them to requirements.txt (an example of the crypto technique of “TOFU” or “Trust On First Use”). If someone later comes along and adds new packages, or if the https connection to PyPI is later insecure for whatever reason, pip will refuse to install and will warn us about the problem:

$ pip install -r requirements.txt
...
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    requests==2.0.1 from https://files.pythonhosted.org/packages/8f/ea/140f18072bbcd81885a9490abb171792fd2961fd7f366be58396f4c6d634/requests-2.0.1-py2.py3-none-any.whl#sha256=f4ebc402e0ea5a87a3d42e300b76c292612d8467024f45f9858a8768f9fb6f6e (from -r requirements.txt (line 7)):
        Expected sha256 8cfddb97667c2a9edaf28b506d2479f1b8dc0631cbdcd0ea8c8864def59c6981
        Expected     or f4ebc402e0ea5a87a3d42e300b76c292612d8467024f45f9858a8768f9fb6f61
             Got        f4ebc402e0ea5a87a3d42e300b76c292612d8467024f45f9858a8768f9fb6f6e

But there are problems lurking here! If we have packages that are installed from Github, then pip-compile can’t hash them and pip won’t install them:

$ echo '-e git+https://github.com/requests/requests@master#egg=requests' > requirements.in
$ pip-compile --generate-hashes
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
-e git+https://github.com/requests/requests@master#egg=requests
certifi==2019.3.9 
    --hash=sha256:59b7658e26ca9c7339e00f8f4636cdfe59d34fa37b9b04f6f9e9926b3cece1a5 
    --hash=sha256:b26104d6835d1f5e49452a26eb2ff87fe7090b89dfcaee5ea2212697e1e1d7ae
chardet==3.0.4 
    --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae 
    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691
idna==2.8 
    --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 
    --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c
urllib3==1.25.2 
    --hash=sha256:a53063d8b9210a7bdec15e7b272776b9d42b2fd6816401a0d43006ad2f9902db 
    --hash=sha256:d363e3607d8de0c220d31950a8f38b18d5ba7c0830facd71a1c6b1036b7ce06c
$ pip install -r requirements.txt
Obtaining requests from git+https://github.com/requests/requests@master#egg=requests (from -r requirements.txt (line 7))
ERROR: The editable requirement requests from git+https://github.com/requests/requests@master#egg=requests (from -r requirements.txt (line 7)) cannot be installed when requiring hashes, because there is no single file to hash.

That’s a serious limitation, because -e requirements are the only way pip-tools knows to specify installations from version control, which are useful while you wait for new fixes in dependencies to be released. (We mostly use them at LIL for dependencies that we’ve patched ourselves, after we send fixes upstream but before they are released.)

And if we have packages that rely on dependencies pip-tools considers unsafe to pin, like setuptools, pip will refuse to install those too:

$ echo 'Markdown' > requirements.in
$ pip-compile --generate-hashes
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
markdown==3.1 
    --hash=sha256:fc4a6f69a656b8d858d7503bda633f4dd63c2d70cf80abdc6eafa64c4ae8c250 
    --hash=sha256:fe463ff51e679377e3624984c829022e2cfb3be5518726b06f608a07a3aad680
$ pip install -r requirements.txt
Collecting markdown==3.1 (from -r requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/f5/e4/d8c18f2555add57ff21bf25af36d827145896a07607486cc79a2aea641af/Markdown-3.1-py2.py3-none-any.whl
Collecting setuptools>=36 (from markdown==3.1->-r requirements.txt (line 7))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools>=36 from https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl#sha256=c7769ce668c7a333d84e17fe8b524b1c45e7ee9f7908ad0a73e1eda7e6a5aebf (from markdown==3.1->-r requirements.txt (line 7))

This can be worked around by adding --allow-unsafe, but (a) that sounds unsafe (though it isn’t), and (b) it won’t pop up until you try to set up a new environment with a low version of setuptools, potentially days later on someone else’s machine.

Those two problems meant that, when I set out to convert our Caselaw Access Project code to use --generate-hashes, I did it wrong a few times in a row, leading to multiple hours spent debugging problems I created for me and other team members (sorry, Anastasia!). I ended up needing a fancy wrapper script around pip-compile to rewrite our requirements in a form it could understand. I wanted it to be a smoother experience for the next people who try to secure their Python projects.

So I filed a series of pull requests:

Support URLs as packages

Support URLs as packages #807 and Fix –generate-hashes with bare VCS URLs #812 laid the groundwork for fixing --generate-hashes, by teaching pip-tools to do something that had been requested for years: installing packages from archive URLs. Where before, pip-compile could only handle Github requirements like this:

-e git+https://github.com/requests/requests@master#egg=requests

It can now handle requirements like this:

https://github.com/requests/requests/archive/master.zip

And zipped requirements can be hashed, so the resulting requirements.txt comes out looking like this, and is accepted by pip install:

https://github.com/requests/requests/archive/master.zip 
   	     --hash=sha256:3c3d84d35630808bf7750b0368b2c7988f89d9f5c2f2633c47f075b3d5015638

This was a long process, and began with resurrecting a pull request from 2017 that had first been worked on by nim65s. I started by just rebasing the existing work, fixing some tests, and submitting it in the hopes the problem had already been solved. Thanks to great feedback from auvipy, atugushev, and blueyed, I ended up making 14 more commits (and eventually a follow-up pull request) to clean up edge cases and get everything working.

Landing this resulted in closing two other pip-tools pull requests from 2016 and 2017, and feature requests from 2014 and 2018.

Warn when --generate-hashes output is uninstallable

The next step was Fix pip-compile output for unsafe requirements #813 and Warn when –generate-hashes output is uninstallable #814. These two PRs allowed pip-compile --generate-hashes to detect and warn when a file would be uninstallable for hashing reasons. Fortunately pip-compile has all of the information it needs at compile time to know that the file will be uninstallable and to make useful recommendations for what to do about it:

$ pip-compile --generate-hashes
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
# WARNING: pip install will require the following package to be hashed.
# Consider using a hashable URL like https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip
-e git+https://github.com/jazzband/pip-tools@7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3#egg=pip-tools
click==7.0 
    --hash=sha256:2335065e6395b9e67ca716de5f7526736bfa6ceead690adf616d925bdc622b13 
    --hash=sha256:5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7
first==2.0.2 
    --hash=sha256:8d8e46e115ea8ac652c76123c0865e3ff18372aef6f03c22809ceefcea9dec86 
    --hash=sha256:ff285b08c55f8c97ce4ea7012743af2495c9f1291785f163722bd36f6af6d3bf
markdown==3.1 
    --hash=sha256:fc4a6f69a656b8d858d7503bda633f4dd63c2d70cf80abdc6eafa64c4ae8c250 
    --hash=sha256:fe463ff51e679377e3624984c829022e2cfb3be5518726b06f608a07a3aad680
six==1.12.0 
    --hash=sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c 
    --hash=sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools==41.0.1        # via markdown

Hopefully, between these two efforts, the next project to try using –generate-hashes will find it a shorter and more straightforward process than I did!

Things left undone

Along the way I discovered a few issues that could be fixed in various projects to help the situation. Here are some pointers:

First, the warning to use --allow-unsafe seems unnecessary – I believe that --allow-unsafe should be the default behavior for pip-compile. I spent some time digging into the reasons that pip-tools considers some packages “unsafe,” and as best I can tell it is because it was thought that pinning those packages could potentially break pip itself, and thus break the user’s ability to recover from a mistake. This seems to no longer be true, if it ever was. Instead, failing to use –allow-unsafe is unsafe, as it means different environments will end up with different versions of key packages despite installing from identical requirements.txt files. I started some discussion about that on the pip-tools repo and the pip repo.

Second, the warning not to use version control links with --generate-hashes is necessary only because of pip’s decision to refuse to install those links alongside hashed requirements. That seems like a bad security tradeoff for several reasons. I filed a bug with pip to open up discussion on the topic.

Third, PyPI and binary wheels. I’m not sure if there’s been further discussion on the decision to allow retrospective binary uploads since 2017, but the example of hiredis makes it seem like that has some major downsides and might be worth reconsidering. I haven’t yet filed anything for this.

Personal reflections (and, thanks Jazzband!)

I didn’t write a ton of code for this in the end, but it was a big step for me personally in working with a mainstream open source project, and I had a lot of fun – learning tools like black and git multi-author commits that we don’t use on our own projects at LIL, collaborating with highly responsive and helpful reviewers (thanks, all!), learning the internals of pip-tools, and hopefully putting something out there that will make people more secure.

pip-tools is part of the Jazzband project, which is an interesting attempt to make the Python package ecosystem a little more sustainable by lowering the bar to maintaining popular packages. I had a great experience with the maintainers working on pip-tools in particular, and I’m grateful for the work that’s gone into making Jazzband happen in general.

Issue

I’m trying to install some packages in a docker container, and there is a problem when installing from a requirements.txt file. This line:

RUN python3.8 -m pip install -r requirements.txt

fails with the error:

...
Collecting torch
  Downloading torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl (735.5 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch from https://files.pythonhosted.org/packages/89/c1/72e9050d3e31e4df983f6e06799a1a4c896427c1e5645a6d810940944b60/torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl#sha256=fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787 (from -r requirements.txt (line 3)):
        Expected sha256 fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787
             Got        d5466637c17c3ae0c81c00d93a0b7c8d8428cfd216f54953a11d0788ea7b74fb

The requirements.txt file is the following:

numpy
opencv-python
torch

However, when installing these packages one at a time everything works fine:

RUN python3.8 -m pip install numpy
RUN python3.8 -m pip install opencv-python
RUN python3.8 -m pip install torch

Any ideas how to solve this?

*** EDIT ***

Dockerfile up to that point:

FROM public.ecr.aws/lambda/python:3.8
COPY requirements.txt ./

Solution

You could try a couple of things. Depending on your base image, you could run pip install in this way:

RUN pip install -r requirements.txt

Another option would be to change your requirements.txt such that it is version controlled. Then you can be sure you have compatible versions and is a good practice in general. Eg.:

torch==1.8.0

Try to run Docker again with without caches:

docker build -no-cache

Or you could check this answer:

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. when updating Django

Answered By – Chiel

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Skip to content



Open


Issue created Jul 03, 2020 by King Chung Huang@kinghuang

Install Python packages uploaded to PyPi Repository with same version results in hash error

Summary

If I upload a package to a project’s PyPi Repository, and then subsequently upload another package with the same version number, a hash error occurs when I try to install the package.

Steps to reproduce

  • Create a Python module and upload it to a project’s PyPi Repository as described in GitLab PyPi Repository.
  • Make a change to the module and upload it to the PyPi Repository with the same version number.
  • Attempt to install the module from the PyPi Repository.

Example Project

N/A. Project is on a private instance.

What is the current bug behavior?

When I try to install the module, pip reports a hash error.

/# pip install dagster-dask-ext --index-url https://__token__:XXXXXXXXX@gitlab.transzap.com/api/v4/projects/898/packages/pypi/simple
Looking in indexes: https://__token__:****@gitlab.transzap.com/api/v4/projects/898/packages/pypi/simple
Collecting dagster-dask-ext
  Downloading https://gitlab.transzap.com/api/v4/projects/898/packages/pypi/files/3b2de0193a7d54183dee15ac4523fd6b3db4d390b568e676f8fffcee95f6354c/dagster_dask_ext-0.0.0.dev0-py3-none-any.whl (4.7 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    dagster-dask-ext from https://gitlab.transzap.com/api/v4/projects/898/packages/pypi/files/3b2de0193a7d54183dee15ac4523fd6b3db4d390b568e676f8fffcee95f6354c/dagster_dask_ext-0.0.0.dev0-py3-none-any.whl#sha256=3b2de0193a7d54183dee15ac4523fd6b3db4d390b568e676f8fffcee95f6354c:
        Expected sha256 3b2de0193a7d54183dee15ac4523fd6b3db4d390b568e676f8fffcee95f6354c
             Got        61e5fc07d37c75c0909abb14440049d5ffcf093d9b7b372e3fd6d393ab8aa451

What is the expected correct behavior?

I should be able to install the module using pip.

The documentation for Uploading Packages says that GitLab will serve the most recent file if the same version is uploaded multiple times.

If you upload the same package with the same version multiple times, each consecutive upload
is saved as a separate file. When installing a package, GitLab will serve the most recent file.

Relevant logs and/or screenshots

I originally published the package on July 1. I then published some changes today (July 3) using the same version number (0.0.0.dev0). Both sets of files are shown in the PyPi Repository.

image

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.27.0
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	13.1.1-ee
Revision:	a604fffd4ba
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://gitlab.transzap.com
HTTP Clone URL:	https://gitlab.transzap.com/some-group/some-project.git
SSH Clone URL:	ssh://git@gitlab.transzap.com:2222/some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	13.3.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/2 ... yes 5/5 ... yes 179/6 ... yes 179/9 ... yes 16/10 ... yes 5/16 ... yes 5/17 ... yes 20/20 ... yes 20/21 ... yes 21/22 ... yes 21/23 ... yes 24/25 ... yes 24/26 ... yes 24/28 ... yes 20/29 ... yes 16/30 ... yes 27/31 ... yes 27/32 ... yes 28/33 ... yes 28/35 ... yes 21/36 ... yes 3/38 ... yes 28/39 ... yes 58/40 ... yes 16/42 ... yes 46/45 ... yes 38/48 ... yes 38/50 ... yes 20/51 ... yes 85/52 ... yes 85/53 ... yes 85/54 ... yes 85/55 ... yes 85/56 ... yes 39/57 ... yes 38/58 ... yes 179/59 ... yes 179/61 ... yes 68/66 ... yes 58/72 ... yes 68/103 ... yes 57/104 ... yes 68/111 ... yes 14/114 ... yes 38/115 ... yes 38/120 ... yes 85/122 ... yes 85/123 ... yes 28/124 ... yes 28/125 ... yes 85/131 ... yes 142/132 ... yes 85/136 ... yes 38/139 ... yes 24/141 ... yes 85/142 ... yes 179/144 ... yes 54/147 ... yes 85/148 ... yes 24/155 ... yes 35/157 ... yes 5/160 ... yes 5/161 ... yes 179/162 ... yes 65/164 ... yes 23/166 ... yes 24/170 ... yes 60/172 ... yes 35/176 ... yes 24/177 ... yes 24/179 ... yes 24/180 ... yes 65/185 ... yes 57/187 ... yes 23/189 ... yes 38/190 ... yes 71/196 ... yes 85/198 ... yes 58/204 ... yes 83/205 ... yes 23/206 ... yes 23/208 ... yes 23/209 ... yes 65/213 ... yes 35/216 ... yes 65/217 ... yes 139/218 ... yes 69/219 ... yes 65/220 ... yes 65/223 ... yes 24/224 ... yes 58/225 ... yes 85/226 ... yes 23/227 ... yes 65/228 ... yes 85/230 ... yes 23/232 ... yes 85/240 ... yes 65/247 ... yes 75/267 ... yes 75/268 ... yes 76/270 ... yes 76/272 ... yes 76/273 ... yes 179/276 ... yes 179/277 ... yes 24/278 ... yes 5/281 ... yes 5/282 ... yes 5/283 ... yes 77/291 ... yes 24/292 ... yes 81/294 ... yes 170/295 ... yes 24/305 ... yes 61/310 ... yes 68/316 ... yes 58/317 ... yes 139/318 ... yes 23/319 ... yes 16/320 ... yes 58/322 ... yes 83/329 ... yes 61/334 ... yes 170/335 ... yes 23/336 ... yes 83/337 ... yes 225/340 ... yes 23/341 ... yes 226/345 ... yes 171/346 ... yes 61/351 ... yes 23/356 ... yes 23/357 ... yes 38/359 ... yes 53/369 ... yes 53/370 ... yes 53/371 ... yes 170/376 ... yes 226/378 ... yes 78/379 ... yes 38/382 ... yes 94/383 ... yes 89/384 ... yes 142/385 ... yes 102/386 ... yes 6/387 ... yes 104/389 ... yes 104/390 ... yes 104/391 ... yes 104/392 ... yes 127/398 ... yes 105/399 ... yes 23/409 ... yes 94/410 ... yes 58/411 ... yes 104/413 ... yes 131/414 ... yes 106/415 ... yes 107/416 ... yes 107/417 ... yes 6/418 ... yes 107/419 ... yes 107/420 ... yes 105/423 ... yes 24/424 ... yes 94/426 ... yes 127/429 ... yes 105/430 ... yes 94/432 ... yes 94/433 ... yes 35/434 ... yes 142/435 ... yes 35/436 ... yes 35/437 ... yes 23/441 ... yes 131/450 ... yes 225/451 ... yes 116/453 ... yes 23/455 ... yes 116/456 ... yes 116/457 ... yes 77/459 ... yes 161/460 ... yes 94/463 ... yes 77/464 ... yes 120/466 ... yes 120/467 ... yes 116/469 ... yes 38/470 ... yes 118/471 ... yes 111/472 ... yes 111/473 ... yes 94/474 ... yes 77/475 ... yes 116/478 ... yes 116/480 ... yes 116/481 ... yes 107/482 ... yes 121/484 ... yes 52/485 ... yes 120/486 ... yes 111/487 ... yes 225/489 ... yes 120/490 ... yes 107/491 ... yes 61/492 ... yes 120/493 ... yes 120/497 ... yes 131/501 ... yes 118/503 ... yes 118/504 ... yes 118/508 ... yes 131/509 ... yes 139/511 ... yes 139/512 ... yes 6/517 ... yes 138/518 ... yes 141/519 ... yes 141/520 ... yes 138/521 ... yes 134/523 ... yes 124/526 ... yes 140/528 ... yes 140/529 ... yes 141/532 ... yes 141/533 ... yes 140/535 ... yes 141/536 ... yes 164/538 ... yes 38/541 ... yes 145/544 ... yes 81/545 ... yes 81/546 ... yes 142/547 ... yes 38/548 ... yes 3/551 ... yes 226/552 ... yes 105/553 ... yes 124/554 ... yes 107/558 ... yes 226/559 ... yes 61/560 ... yes 140/561 ... yes 141/569 ... yes 141/570 ... yes 140/571 ... yes 131/572 ... yes 141/573 ... yes 140/574 ... yes 151/581 ... yes 131/583 ... yes 131/584 ... yes 131/585 ... yes 131/586 ... yes 131/587 ... yes 131/588 ... yes 154/590 ... yes 154/592 ... yes 154/593 ... yes 154/594 ... yes 140/604 ... yes 135/608 ... yes 157/611 ... yes 131/612 ... yes 206/613 ... yes 141/614 ... yes 140/615 ... yes 139/617 ... yes 139/618 ... yes 139/619 ... yes 139/621 ... yes 131/623 ... yes 154/625 ... yes 154/626 ... yes 154/628 ... yes 131/629 ... yes 131/630 ... yes 142/631 ... yes 131/633 ... yes 161/634 ... yes 161/637 ... yes 131/638 ... yes 163/641 ... yes 141/642 ... yes 131/643 ... yes 164/645 ... yes 131/646 ... yes 164/647 ... yes 131/650 ... yes 161/653 ... yes 161/654 ... yes 161/655 ... yes 161/656 ... yes 131/657 ... yes 154/658 ... yes 163/660 ... yes 164/661 ... yes 131/662 ... yes 131/663 ... yes 154/664 ... yes 161/670 ... yes 161/671 ... yes 164/672 ... yes 131/673 ... yes 161/674 ... yes 131/675 ... yes 161/676 ... yes 161/677 ... yes 131/678 ... yes 131/679 ... yes 167/680 ... yes 131/681 ... yes 160/682 ... yes 131/683 ... yes 131/684 ... yes 131/685 ... yes 172/686 ... yes 172/687 ... yes 173/688 ... yes 167/689 ... yes 206/690 ... yes 173/691 ... yes 131/692 ... yes 167/698 ... yes 206/699 ... yes 173/702 ... yes 186/703 ... yes 176/704 ... yes 182/706 ... yes 173/707 ... yes 182/708 ... yes 167/709 ... yes 182/710 ... yes 173/711 ... yes 182/712 ... yes 61/713 ... yes 6/714 ... yes 142/715 ... yes 131/717 ... yes 216/719 ... yes 198/720 ... yes 139/721 ... yes 161/722 ... yes 131/723 ... yes 187/727 ... yes 16/728 ... yes 116/729 ... yes 131/730 ... yes 167/731 ... yes 131/732 ... yes 192/733 ... yes 191/734 ... yes 104/735 ... yes 131/736 ... yes 227/737 ... yes 95/740 ... yes 190/741 ... yes 95/742 ... yes 196/743 ... yes 196/744 ... yes 198/745 ... yes 199/746 ... yes 199/747 ... yes 176/748 ... yes 191/749 ... yes 190/750 ... yes 131/751 ... yes 241/752 ... yes 190/753 ... yes 189/754 ... yes 191/755 ... yes 196/757 ... yes 37/758 ... yes 104/760 ... yes 131/765 ... yes 210/768 ... yes 210/769 ... yes 131/770 ... yes 131/771 ... yes 104/772 ... yes 131/773 ... yes 222/774 ... yes 203/775 ... yes 176/776 ... yes 191/777 ... yes 191/778 ... yes 191/779 ... yes 188/780 ... yes 161/781 ... yes 212/783 ... yes 212/784 ... yes 188/785 ... yes 131/786 ... yes 105/787 ... yes 131/788 ... yes 212/789 ... yes 192/790 ... yes 241/791 ... yes 241/792 ... yes 213/793 ... yes 214/794 ... yes 135/795 ... yes 198/796 ... yes 224/797 ... yes 131/799 ... yes 43/801 ... yes 139/802 ... yes 139/803 ... yes 131/804 ... yes 161/805 ... yes 161/806 ... yes 220/807 ... yes 220/808 ... yes 216/809 ... yes 213/810 ... yes 214/811 ... yes 188/812 ... yes 131/813 ... yes 131/814 ... yes 139/815 ... yes 142/816 ... yes 164/817 ... yes 131/818 ... yes 229/819 ... yes 3/820 ... yes 135/821 ... yes 154/822 ... yes 135/824 ... yes 231/825 ... yes 213/826 ... yes 214/827 ... yes 222/828 ... yes 179/829 ... yes 222/830 ... yes 191/831 ... yes 235/833 ... yes 235/834 ... yes 191/835 ... yes 235/837 ... yes 203/841 ... yes 203/842 ... yes 131/843 ... yes 131/844 ... yes 244/845 ... yes 179/846 ... yes 245/847 ... yes 245/848 ... yes 155/850 ... yes 248/851 ... yes 250/852 ... yes 179/855 ... yes 164/856 ... yes 227/857 ... yes 252/858 ... yes 12/859 ... yes 135/860 ... yes 248/861 ... yes 235/862 ... yes 245/864 ... yes 235/866 ... yes 252/868 ... yes 252/869 ... yes 248/870 ... yes 248/871 ... yes 177/874 ... yes 235/875 ... yes 131/876 ... yes 164/877 ... yes 254/878 ... yes 161/879 ... yes 254/880 ... yes 252/881 ... yes 255/882 ... yes 254/883 ... yes 176/884 ... yes 257/886 ... yes 257/887 ... yes 257/888 ... yes 224/889 ... yes 258/890 ... yes 257/892 ... yes 257/893 ... yes 131/894 ... yes 191/895 ... yes 244/896 ... yes 259/897 ... yes 259/898 ... yes 131/899 ... yes 215/900 ... yes 213/901 ... yes 131/902 ... yes 241/905 ... yes 217/906 ... yes 217/907 ... yes 134/908 ... yes 131/909 ... yes 131/910 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.27.0) Git user has default SSH configuration? ... yes Active users: ... 69 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

Unknown.

Edited Jul 03, 2020 by King Chung Huang

Понравилась статья? Поделить с друзьями:
  • Error there was an error with your transaction please try again
  • Error there was an error while processing your request
  • Error there was a problem with the editor
  • Error there is no unique constraint matching given keys for referenced table users
  • Error there is already a certificate for openvpn