Error 3 while decompressing data incorrect header check

«zlib.error: Error -3 while decompressing data: incorrect header check» when trying to run the exe which is encrypted with pyinstaller. #2247

I’m packaging the exe using pyinstaller, and I’m also providing a key to encrypt it. But on trying to run the exe, I get the following error:

I don’t get this error when I package it without a key.

The text was updated successfully, but these errors were encountered:

I just encountered same error.
it turns out I enable exclude the Crypto module.
hope it helps.

Run with —clean option after you change block_cipher value.

Run with —clean option after you change block_cipher value.

@derkan : what do you mean by this?

I have this exact same issue (on Mac and Windows). I invoke the pyinstaller script with —key=ABCDEF0123456789 .

I have the —clean option in the same command. I’ve tried putting —key= before and after the —clean but it makes no difference.

I also do an rm -rf build dist before invoking pyinstaller.

This might be part of the solution perhaps?

Maybe using header detection might be useful?

adding 32 to windowBits will trigger header detection
zlib.decompress(zlib_data, zlib.MAX_WBITS|32)

Or the possibly the decryption is wrong and it’s trying to decompress junk?

You are using an outdated version of PyInstaller. Only the newest version is supported. But encrypted bytecode is not working, see #3531

I’m using pyinstaller v3.4, which is the latest release. The download page says:

Release 3.4: stable, supports Python 2.7, 3.4–3.7

If the —key= option is no longer supported then the documentation needs to be updated to reflect that. If it is supported and working on Python 2, but not on Python 3, then the documentation should reflect that.

That is the quickest way to clarify the issue and will make these reports and comments less frequent 😉

Источник

How to decode the gzip compressed data returned in a HTTP Response in python?

I have created a client/server architecture in python, I take HTTP request from the client which is served by requesting another HTTP server through my code.

When I get the response from the third server I am not able to decode the gzip compressed data, I first split the response data using rn as separation character which got me the data as the last item in the list then I tried decompressing it with

but it is giving me an error of incorrect headers. How should I go with this problem ?

I want to read the data that is been transferred between the client and the 2nd server.

2 Answers 2

Specify the wbits when using zlib.decompress(string, wbits, bufsize) see end of «troubleshooting» for example.

Troubleshooting

Lets start out with a a curl command that downloads a byte-range response with an unknown «content-encoding» (note: we know before hand it is some sort of compressed thing, mabye deflate maybe gzip ):

With the following response headers:

So to the point.

Lets display the hex output of the first 10 bytes: curl -r 266472196-266472208 $URL | xxd

We can see some basics of what we are working with with the hex values.

Roughly meaning its probably a gzip ( 1f8b ) using deflate ( 0800 ) without a modification time ( 0000 0000 ), or any extra flags set ( 00 ), using a fat32 system( 00 ).

So onto the python:

notice anything similar?:

And on to the decompression let’s just try at random based on the (documentation):

«zlib.error: Error -2 while preparing to decompress data: inconsistent stream state»:

«Error -3 while decompressing data: incorrect header check»:

Источник

Error -3 while decompressing data: incorrect header check — urllib2

I am checking for url status with this code:

the url i am checking is:

the Exception Location is:

http status is 200 which is ok for my case, but i am getting this error

I actually need only http status, why is it reading the whole content?

2 Answers 2

You may have any number of reasons why you choose httplib2 , but it’s far too easy to get the status code of an HTTP request using the python module requests . Install with the following command:

See an extremely simple example below.

Unless you have a considerable constraint that needs httplib2 explicitly, this solves your problem.

This may be a bug (or just uncommon design decision) in httplib2 . I don’t get this problem with urllib2 or httplib in the 2.x stdlib, or urllib.request or http.client in the 3.x stdlib, or the third-party libraries requests , urllib3 , or pycurl .

So, is there a reason you need to use this particular library?

I actually need only http status, why is it reading the whole content?

Well, most HTTP libraries are going to read and parse the whole content, or at least the headers, before returning control. That way they can respond to simple requests about the headers or chunked encoding or MIME envelope or whatever without any delay.

Also, many of them automate things like 100 continue, 302 redirect, various kinds of auth, etc., and there’s no way they could do that if they didn’t read ahead. In particular, according to the description for httplib2 , handling these things automatically is one of the main reasons you should use it in the first place.

Also, the first TCP read is nearly always going to include the headers anyway, so why not read them?

This means that if the headers are invalid, you may get an exception immediately. They may still provide a way to get the status code (or the raw headers, or other information) anyway.

As a side note, if you only want the HTTP status, you should probably send a HEAD request rather than a GET. Unless you’re writing and testing a server, you can almost always rely on the fact that, as the RFC says, the status and headers should be identical to what you’d get with GET . In fact, that would almost certainly solve things in this case—if there is no body to decompress, the fact that httplib2 has gotten confused into thinking the body is gzipped when it isn’t won’t matter anyway.

Источник

Jpeg Image : zlib.error : Error -3 while decompressing data : incorrect header check

I have a jpg image that is in memory and therefore is in bytes. I then compress it with the following:

I compress it to store it into a MongoDB database. This image is one of the elements in the record that I save. I save the with the following.

I got an error with bytes and Json, so I simply convert the bytes image to a string with the following.

Now, I want to recover the image from the json file. I thought that doing the same steps in the opposite order would work but no.

So here is what I do to store:

I thought this would work:

I receive the zlib.error : Error -3 during the following:

EDIT:

Okay so I was toying around, and I think the issue might be the .encode() . I start with b» » . After str(b» «) , I get «b’ ‘» . If I do the .encode() , I get b»b’ ‘» . How do I deal with this?

1 Answer 1

str() is useful to display something — it creates text readable for human.

It may show that you had bytes ( «b»» ) or it may show string like xe0 for values which can’t be converted to chars. But it doesn’t have to create text useful for keeping in database.

Many databases have field to keep bytes and then you could keep image as bytes (without converting to pickle which may only add more bytes, and without compressing because images already use some compression)

If you have to keep (or send by internet) file as string then better convert it to base64 . And this method used by some APIs to send image in JSON .

Источник

zlib.error: Error -3 while decompressing: incorrect header check

I have a gzip file and I am trying to read it via Python as below:

it throws this error:

How can I overcome it?

8 Answers 8

You have this error:

Which is most likely because you are trying to check headers that are not there, e.g. your data follows RFC 1951 ( deflate compressed format) rather than RFC 1950 ( zlib compressed format) or RFC 1952 ( gzip compressed format).

choosing windowBits

But zlib can decompress all those formats:

• to (de-)compress deflate format, use wbits = -zlib.MAX_WBITS
• to (de-)compress zlib format, use wbits = zlib.MAX_WBITS
• to (de-)compress gzip format, use wbits = zlib.MAX_WBITS | 16

examples

obvious test for zlib :

test for deflate :

the data is also compatible with gzip module:

adding 32 to windowBits will trigger header detection

using gzip instead

or you can ignore zlib and use gzip module directly; but please remember that under the hood, gzip uses zlib .

Update: dnozay’s answer explains the problem and should be the accepted answer.

Try the gzip module, code below is straight from the python docs.

I just solved the «incorrect header check» problem when uncompressing gzipped data.

You need to set -WindowBits => WANT_GZIP in your call to inflateInit2 (use the 2 version)

Yes, this can be very frustrating. A typically shallow reading of the documentation presents Zlib as an API to Gzip compression, but by default (not using the gz* methods) it does not create or uncompress the Gzip format. You have to send this non-very-prominently documented flag.

This does not answer the original question, but it may help someone else that ends up here.

The zlib.error: Error -3 while decompressing: incorrect header check also occurs in the example below:

The example is a minimal reproduction of something I encountered in some legacy Django code, where Base64 encoded bytes (from an HTTP POST) were being stored in a Django CharField (instead of a BinaryField ).

When reading a CharField value from the database, str() is called on the value, without an explicit encoding , as can be seen in the Django source.

If neither encoding nor errors is given, str(object) returns object.str(), which is the “informal” or nicely printable string representation of object. For string objects, this is the string itself. If object does not have a str() method, then str() falls back to returning repr(object).

So, in the example, we are inadvertently base64-decoding

The zlib decompression in the example would succeed if an explicit encoding were used, e.g. str(b64_encoded_bytes, ‘utf-8’) .

NOTE specific to Django:

What’s especially tricky: this issue only arises when retrieving a value from the database. See for example the test below, which passes (in Django 3.0.3):

Источник

У вас есть эта ошибка:

zlib.error: Error -3 while decompressing: incorrect header check

Скорее всего, потому, что вы пытаетесь проверить заголовки, которых нет, например. ваши данные следуют RFC 1951 (deflate сжатый формат), а не RFC 1950 (zlib сжатый формат) или RFC 1952 (gzip сжатый формат).

выбор windowBits

Но zlib может распаковать все эти форматы:

• to (de-) compress deflate format, используйте wbits = -zlib.MAX_WBITS
• to (de-) сжать формат zlib, используйте wbits = zlib.MAX_WBITS
• to (de-) compress gzip format, используйте wbits = zlib.MAX_WBITS | 16

См. документацию в http://www.zlib.net/manual.html#Advanced (раздел inflateInit2)

примеры

тестовые данные:

>>> deflate_compress = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS)
>>> zlib_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS)
>>> gzip_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16)
>>> 
>>> text = '''test'''
>>> deflate_data = deflate_compress.compress(text) + deflate_compress.flush()
>>> zlib_data = zlib_compress.compress(text) + zlib_compress.flush()
>>> gzip_data = gzip_compress.compress(text) + gzip_compress.flush()
>>> 

очевидный тест для zlib:

>>> zlib.decompress(zlib_data)
'test'

test для deflate:

>>> zlib.decompress(deflate_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(deflate_data, -zlib.MAX_WBITS)
'test'

test для gzip:

>>> zlib.decompress(gzip_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|16)
'test'

данные также совместимы с модулем gzip:

>>> import gzip
>>> import StringIO
>>> fio = StringIO.StringIO(gzip_data)
>>> f = gzip.GzipFile(fileobj=fio)
>>> f.read()
'test'
>>> f.close()

автоматическое определение заголовка (zlib или gzip)

добавление 32 в windowBits приведет к обнаружению заголовка

>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|32)
'test'
>>> zlib.decompress(zlib_data, zlib.MAX_WBITS|32)
'test'

вместо gzip

или вы можете игнорировать zlib и напрямую использовать модуль gzip; но помните, что под капотом gzip использует zlib.

fh = gzip.open('abc.gz', 'rb')
cdata = fh.read()
fh.close()

У вас такая ошибка:

zlib.error: Error -3 while decompressing: incorrect header check

Это, скорее всего, потому, что вы пытаетесь проверить заголовки, которых нет, например, ваши данные следуют RFC 1951 (deflate сжатый формат), а не RFC 1950 (zlib сжатый формат) или RFC 1952 (gzip сжатый формат).

выбор windowBits

Но zlib может распаковать все эти форматы:

• чтобы (де-) сжать deflate формат, используйте wbits = -zlib.MAX_WBITS
• чтобы (де-) сжать zlib формат, используйте wbits = zlib.MAX_WBITS
• чтобы (де-) сжать gzip формат, используйте wbits = zlib.MAX_WBITS | 16

См. Документацию в http://www.zlib.net/manual.html#Advanced (раздел inflateInit2)

Примеры

данные испытаний:

>>> deflate_compress = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS)
>>> zlib_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS)
>>> gzip_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16)
>>> 
>>> text = '''test'''
>>> deflate_data = deflate_compress.compress(text) + deflate_compress.flush()
>>> zlib_data = zlib_compress.compress(text) + zlib_compress.flush()
>>> gzip_data = gzip_compress.compress(text) + gzip_compress.flush()
>>> 

очевидный тест на zlib:

>>> zlib.decompress(zlib_data)
'test'

испытание для deflate:

>>> zlib.decompress(deflate_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(deflate_data, -zlib.MAX_WBITS)
'test'

испытание для gzip:

>>> zlib.decompress(gzip_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|16)
'test'

данные также совместимы с gzip модуль:

>>> import gzip
>>> import StringIO
>>> fio = StringIO.StringIO(gzip_data)  # io.BytesIO for Python 3
>>> f = gzip.GzipFile(fileobj=fio)
>>> f.read()
'test'
>>> f.close()

автоматическое определение заголовка (zlib или gzip)

добавить 32 в windowBits вызовет обнаружение заголовка

>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|32)
'test'
>>> zlib.decompress(zlib_data, zlib.MAX_WBITS|32)
'test'

через gzip вместо

или вы можете игнорировать zlib И использовать gzip модуль напрямую; но пожалуйста, помните, что под капотом, gzip использования zlib.

fh = gzip.open('abc.gz', 'rb')
cdata = fh.read()
fh.close()

У вас есть эта ошибка:

Который, скорее всего, потому что вы пытаетесь проверить заголовки, которые не существуют, например, вашего сведения следующее в RFC 1951 (сдуваться сжатый формат), а не в RFC 1950 (zlib в сжатом формате) или в RFC 1952(gzip в` сжатом формате).

выбор windowBits

Но библиотеку может распаковывать все эти форматы:

• для (де-)формат сжатия сдуваться, wbits использовать = - с zlib.MAX_WBITS
• для (де-)формат сжатия библиотеку, wbits использовать = с zlib.MAX_WBITS
• для (де-)формат сжатия с помощью gzip, wbits использовать = с zlib.MAX_WBITS | 16

Ознакомиться с документацией в inflateInit2 http://www.zlib.net/manual.html#advanced (раздел«)

примеры

тестовые данные:

очевидный тест для библиотеку:

тест для сдуваться:

тест для помощи gzip:

эти данные также совместимый с gzip в модуле:

автоматическое определение заголовка (zlib или с помощью gzip)

добавление 32 до windowBits вызовет обнаружение заголовка

использование gzip-сжатия вместо

или вы можете игнорировать эту библиотеку и модуль использовать gzip-сжатия непосредственно; но следует помнить, что под капотом, в gzip использует библиотеку.