Zlib decompress error

You have this error:

zlib.error: Error -3 while decompressing: incorrect header check

Which is most likely because you are trying to check headers that are not there, e.g. your data follows RFC 1951 (deflate compressed format) rather than RFC 1950 (zlib compressed format) or RFC 1952 (gzip compressed format).

choosing windowBits

But zlib can decompress all those formats:

• to (de-)compress deflate format, use wbits = -zlib.MAX_WBITS
• to (de-)compress zlib format, use wbits = zlib.MAX_WBITS
• to (de-)compress gzip format, use wbits = zlib.MAX_WBITS | 16

See documentation in http://www.zlib.net/manual.html#Advanced (section inflateInit2)

examples

test data:

>>> deflate_compress = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS)
>>> zlib_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS)
>>> gzip_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16)
>>> 
>>> text = '''test'''
>>> deflate_data = deflate_compress.compress(text) + deflate_compress.flush()
>>> zlib_data = zlib_compress.compress(text) + zlib_compress.flush()
>>> gzip_data = gzip_compress.compress(text) + gzip_compress.flush()
>>> 

obvious test for zlib:

>>> zlib.decompress(zlib_data)
'test'

test for deflate:

>>> zlib.decompress(deflate_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(deflate_data, -zlib.MAX_WBITS)
'test'

test for gzip:

>>> zlib.decompress(gzip_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|16)
'test'

the data is also compatible with gzip module:

>>> import gzip
>>> import StringIO
>>> fio = StringIO.StringIO(gzip_data)  # io.BytesIO for Python 3
>>> f = gzip.GzipFile(fileobj=fio)
>>> f.read()
'test'
>>> f.close()

automatic header detection (zlib or gzip)

adding 32 to windowBits will trigger header detection

>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|32)
'test'
>>> zlib.decompress(zlib_data, zlib.MAX_WBITS|32)
'test'

using gzip instead

or you can ignore zlib and use gzip module directly; but please remember that under the hood, gzip uses zlib.

fh = gzip.open('abc.gz', 'rb')
cdata = fh.read()
fh.close()

For applications that require data compression, the functions in this module
allow compression and decompression, using the zlib library. The zlib library
has its own home page at http://www.zlib.net. There are known
incompatibilities between the Python module and versions of the zlib library
earlier than 1.1.3; 1.1.3 has a security vulnerability, so we recommend using
1.1.4 or later.

zlib’s functions have many options and often need to be used in a particular
order. This documentation doesn’t attempt to cover all of the permutations;
consult the zlib manual at http://www.zlib.net/manual.html for authoritative
information.

For reading and writing .gz files see the gzip module.

The available exception and functions in this module are:

exception zlib.error¶

Exception raised on compression and decompression errors.

zlib.adler32(data[, value])¶

Computes an Adler-32 checksum of data. (An Adler-32 checksum is almost as
reliable as a CRC32 but can be computed much more quickly.) The result
is an unsigned 32-bit integer. If value is present, it is used as
the starting value of the checksum; otherwise, a default value of 1
is used. Passing in value allows computing a running checksum over the
concatenation of several inputs. The algorithm is not cryptographically
strong, and should not be used for authentication or digital signatures. Since
the algorithm is designed for use as a checksum algorithm, it is not suitable
for use as a general hash algorithm.

Changed in version 3.0: Always returns an unsigned value.
To generate the same numeric value across all Python versions and
platforms, use adler32(data) & 0xffffffff.

zlib.compress(data, level=-1)¶

Compresses the bytes in data, returning a bytes object containing compressed data.
level is an integer from 0 to 9 or -1 controlling the level of compression;
1 is fastest and produces the least compression, 9 is slowest and
produces the most. 0 is no compression. The default value is -1
(Z_DEFAULT_COMPRESSION). Z_DEFAULT_COMPRESSION represents a default
compromise between speed and compression (currently equivalent to level 6).
Raises the error exception if any error occurs.

Changed in version 3.6: level can now be used as a keyword parameter.

zlib.compressobj(level=-1, method=DEFLATED, wbits=15, memLevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict])¶

Returns a compression object, to be used for compressing data streams that won’t
fit into memory at once.

level is the compression level – an integer from 0 to 9 or -1.
A value of 1 is fastest and produces the least compression, while a value of
9 is slowest and produces the most. 0 is no compression. The default
value is -1 (Z_DEFAULT_COMPRESSION). Z_DEFAULT_COMPRESSION represents a default
compromise between speed and compression (currently equivalent to level 6).

method is the compression algorithm. Currently, the only supported value is
DEFLATED.

The wbits argument controls the size of the history buffer (or the
“window size”) used when compressing data, and whether a header and
trailer is included in the output. It can take several ranges of values:

• +9 to +15: The base-two logarithm of the window size, which
  therefore ranges between 512 and 32768. Larger values produce
  better compression at the expense of greater memory usage. The
  resulting output will include a zlib-specific header and trailer.
• −9 to −15: Uses the absolute value of wbits as the
  window size logarithm, while producing a raw output stream with no
  header or trailing checksum.
• +25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the
  window size logarithm, while including a basic gzip header
  and trailing checksum in the output.

The memLevel argument controls the amount of memory used for the
internal compression state. Valid values range from 1 to 9.
Higher values use more memory, but are faster and produce smaller output.

strategy is used to tune the compression algorithm. Possible values are
Z_DEFAULT_STRATEGY, Z_FILTERED, and Z_HUFFMAN_ONLY.

zdict is a predefined compression dictionary. This is a sequence of bytes
(such as a bytes object) containing subsequences that are expected
to occur frequently in the data that is to be compressed. Those subsequences
that are expected to be most common should come at the end of the dictionary.

Changed in version 3.3: Added the zdict parameter and keyword argument support.

zlib.crc32(data[, value])¶

Computes a CRC (Cyclic Redundancy Check) checksum of data. The
result is an unsigned 32-bit integer. If value is present, it is used
as the starting value of the checksum; otherwise, a default value of 0
is used. Passing in value allows computing a running checksum over the
concatenation of several inputs. The algorithm is not cryptographically
strong, and should not be used for authentication or digital signatures. Since
the algorithm is designed for use as a checksum algorithm, it is not suitable
for use as a general hash algorithm.

Changed in version 3.0: Always returns an unsigned value.
To generate the same numeric value across all Python versions and
platforms, use crc32(data) & 0xffffffff.

zlib.decompress(data, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)¶

Decompresses the bytes in data, returning a bytes object containing the
uncompressed data. The wbits parameter depends on
the format of data, and is discussed further below.
If bufsize is given, it is used as the initial size of the output
buffer. Raises the error exception if any error occurs.

The wbits parameter controls the size of the history buffer
(or “window size”), and what header and trailer format is expected.
It is similar to the parameter for compressobj(), but accepts
more ranges of values:

• +8 to +15: The base-two logarithm of the window size. The input
  must include a zlib header and trailer.
• 0: Automatically determine the window size from the zlib header.
  Only supported since zlib 1.2.3.5.
• −8 to −15: Uses the absolute value of wbits as the window size
  logarithm. The input must be a raw stream with no header or trailer.
• +24 to +31 = 16 + (8 to 15): Uses the low 4 bits of the value as
  the window size logarithm. The input must include a gzip header and
  trailer.
• +40 to +47 = 32 + (8 to 15): Uses the low 4 bits of the value as
  the window size logarithm, and automatically accepts either
  the zlib or gzip format.

When decompressing a stream, the window size must not be smaller
than the size originally used to compress the stream; using a too-small
value may result in an error exception. The default wbits value
corresponds to the largest window size and requires a zlib header and
trailer to be included.

bufsize is the initial size of the buffer used to hold decompressed data. If
more space is required, the buffer size will be increased as needed, so you
don’t have to get this value exactly right; tuning it will only save a few calls
to malloc().

Changed in version 3.6: wbits and bufsize can be used as keyword arguments.

zlib.decompressobj(wbits=15[, zdict])¶

Returns a decompression object, to be used for decompressing data streams that
won’t fit into memory at once.

The wbits parameter controls the size of the history buffer (or the
“window size”), and what header and trailer format is expected. It has
the same meaning as described for decompress().

The zdict parameter specifies a predefined compression dictionary. If
provided, this must be the same dictionary as was used by the compressor that
produced the data that is to be decompressed.

Note

If zdict is a mutable object (such as a bytearray), you must not
modify its contents between the call to decompressobj() and the first
call to the decompressor’s decompress() method.

Changed in version 3.3: Added the zdict parameter.

Compression objects support the following methods:

Compress.compress(data)¶

Compress data, returning a bytes object containing compressed data for at least
part of the data in data. This data should be concatenated to the output
produced by any preceding calls to the compress() method. Some input may
be kept in internal buffers for later processing.

Compress.flush([mode])¶

All pending input is processed, and a bytes object containing the remaining compressed
output is returned. mode can be selected from the constants
Z_SYNC_FLUSH, Z_FULL_FLUSH, or Z_FINISH,
defaulting to Z_FINISH. Z_SYNC_FLUSH and
Z_FULL_FLUSH allow compressing further bytestrings of data, while
Z_FINISH finishes the compressed stream and prevents compressing any
more data. After calling flush() with mode set to Z_FINISH,
the compress() method cannot be called again; the only realistic action is
to delete the object.

Compress.copy()¶

Returns a copy of the compression object. This can be used to efficiently
compress a set of data that share a common initial prefix.

Decompression objects support the following methods and attributes:

Decompress.unused_data¶

A bytes object which contains any bytes past the end of the compressed data. That is,
this remains b"" until the last byte that contains compression data is
available. If the whole bytestring turned out to contain compressed data, this is
b"", an empty bytes object.

Decompress.unconsumed_tail¶

A bytes object that contains any data that was not consumed by the last
decompress() call because it exceeded the limit for the uncompressed data
buffer. This data has not yet been seen by the zlib machinery, so you must feed
it (possibly with further data concatenated to it) back to a subsequent
decompress() method call in order to get correct output.

Decompress.eof¶

A boolean indicating whether the end of the compressed data stream has been
reached.

This makes it possible to distinguish between a properly-formed compressed
stream, and an incomplete or truncated one.

New in version 3.3.

Decompress.decompress(data, max_length=0)¶

Decompress data, returning a bytes object containing the uncompressed data
corresponding to at least part of the data in string. This data should be
concatenated to the output produced by any preceding calls to the
decompress() method. Some of the input data may be preserved in internal
buffers for later processing.

If the optional parameter max_length is non-zero then the return value will be
no longer than max_length. This may mean that not all of the compressed input
can be processed; and unconsumed data will be stored in the attribute
unconsumed_tail. This bytestring must be passed to a subsequent call to
decompress() if decompression is to continue. If max_length is zero
then the whole input is decompressed, and unconsumed_tail is empty.

Changed in version 3.6: max_length can be used as a keyword argument.

Decompress.flush([length])¶

All pending input is processed, and a bytes object containing the remaining
uncompressed output is returned. After calling flush(), the
decompress() method cannot be called again; the only realistic action is
to delete the object.

The optional parameter length sets the initial size of the output buffer.

Decompress.copy()¶

Returns a copy of the decompression object. This can be used to save the state
of the decompressor midway through the data stream in order to speed up random
seeks into the stream at a future point.

Information about the version of the zlib library in use is available through
the following constants:

zlib.ZLIB_VERSION¶

The version string of the zlib library that was used for building the module.
This may be different from the zlib library actually used at runtime, which
is available as ZLIB_RUNTIME_VERSION.

zlib.ZLIB_RUNTIME_VERSION¶

The version string of the zlib library actually loaded by the interpreter.

New in version 3.3.

Распаковать сжатые данные из буфера при помощи zlib.

Синтаксис:

import zlib

zlib.decompress(data, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)

Параметры:

• data — сжатые байты,
• wbits=MAX_WBITS — управляет размером буфера, форматом заголовка и трейлера,
• bufsize=DEF_BUF_SIZE — начальный размер выходного буфера.

Возвращаемое значение:

• распакованные данные представленные в байтах.

Описание:

Функция decompress() модуля zlib распаковывает сжатые данные представленные как байты, возвращая объект байтов, содержащий несжатые данные.

Если задано значение аргумента bufsize, оно используется в качестве начального размера выходного буфера.

Вызывает исключение zlib.error, если возникает какая-либо ошибка.

Параметр wbits управляет размером буфера истории или «размером окна» и ожидаемым форматом заголовка и трейлера. Он похож на параметр для функции zlib.compressobj(), но принимает больше диапазонов значений:

• от +8 до +15: логарифм основания-два размера окна. Входные данные должны включать заголовок zlib и трейлер.
• 0: автоматически определять размер окна из заголовка zlib. Поддерживается только начиная с версии библиотеки zlib 1.2.3.5.
• от -8 до -15: использует абсолютное значение wbits в качестве логарифма размера окна. Входными данными должен быть необработанный поток без заголовка или трейлера.
• от +24 до +31 = 16 + (от 8 до 15): использует младшие 4 бита значения в качестве логарифма размера окна. Входные данные должны включать заголовок gzip и трейлер.
• от +40 до +47 = 32 + (от 8 до 15): использует младшие 4 бита значения в качестве логарифма размера окна и автоматически принимает формат zlib или gzip.

При распаковке потока размер окна не должен быть меньше размера, первоначально использовавшегося для сжатия потока. Использование слишком маленького значения может привести к исключению zlib.error. Значение wbits по умолчанию соответствует наибольшему размеру окна и требует включения заголовка и трейлера библиотеки zlib.

Аргумент bufsize — это начальный размер буфера, используемого для хранения распакованных данных. Если требуется больше места, размер буфера будет увеличиваться по мере необходимости, поэтому вам не нужно точно определять это значение. Его настройка сохранит только несколько вызовов malloc().

Примеры использования:

>>> import zlib
# создадим массив данных
>>> text = 'Привет docs-python.ru '
>>> data = []
>>> for _ in range(10):
...     data.append(text * 20)
...
# преобразование текста в байты
>>> byte_data = 'nn'.join(data).encode('utf-8')
# сжимаем данные
>>> compress = zlib.compress(byte_data, level=-1)
# распаковываем сжатые данные в буфер
>>> decompress = zlib.decompress(compress)
# преобразуем байты в текст
>>> text = decompress.decode('utf-8')
# выведем на печать первые 22 символа
>>> text[0:22]
# 'Привет docs-python.ru Привет'

Mar-29-2020, 12:44 PM
(This post was last modified: Mar-29-2020, 12:44 PM by DreamingInsanity.)

(this is sort of about python, but it might get slightly off topic. I couldn’t find anywhere else it would make sense to post this, and since I’m coding it in python, I though I might as well post it here)

Slight backstory:
For many years, I have had an issue with one of my games where I am not able to save any of my progress. I have done much research but nothing has seemed to have worked, so I though I’d take matters into my own hands and try and fix it myself.

Here’s a snippet of what the save file looks like inside:
ÃÂFqÄQ˛˘fl≥-ˆ&[email protected]Öë]ù!+—Ïá$ë 2©MløëΑ*Œ∞áõìÔœÌ Ä∏‡Ì™:y"2≠¥7}˘Ê¥∞Æ;â"ºGªh˛∑¸

Now, according to the hours of research I have done, the way this save file (and apparently all others) is encrypted is by:
gzip encode -> base64 encode -> XOR^11 (I believe)
Meaning to decode the file, you have to work backwards:
XOR^11 -> base64 decode -> gzip decode
Simple right? I wish.

Here’s the code I have created to do that:

a = open("/Users/everyone/Desktop/Save.dat", "rb").read()
newchars = []
for i in a:
    newchars.append(i^11)
saveData = bytearray(newchars).decode("utf-8", "ignore")

b64 = base64.b64decode(b"H4sIAAAAAAAAC9y"+str(saveData).replace("-","+").replace("_", "/").encode())

print(b64)

#print( b := bytes(map(lambda x: x ^ 11, b64)))
print(zlib.decompress(b64, -zlib.MAX_WBITS))

The issue is with the gzip decoding:

Error:
Traceback (most recent call last):
  File "/Users/everyone/Desktop/test.py", line 39, in <module>
    print(zlib.decompress(b64, -zlib.MAX_WBITS))
zlib.error: Error -3 while decompressing data: invalid block type
[Finished in 0.2s with exit code 1]

I’m not the first person to try and decrypt these game saves

The issue I’ve got is that.. the same error comes up using others’ game save decryptors. I also have use .decode(«utf-8», «ignore»), which ll the other ones do not, otherwise I get invalid unicode character errors. I also have to append the gzip-ed, base64 encoded header to open the file, not just to save it like the one on GitHub does.

If we take a look a snippet my Windows using friend’s game save (I’m using Mac OS X):
?xBJJJJJJJJH&r2X<[:~ANR&Mhh}i=S|}[email protected]{h8x>}M<LjyzMQS9LSA][email protected]_NTm>IJl~ZcZXXZAG&||
Well that’s very different — and a lot more correct, since this doesn’t error when put into the game save decryptor.
This lead me down the path that my game save was very corrupted, however, if that was the case, then the game shouldn’t be able to read it either. I tested that by putting my friends game save into my save folder to see if I would load his progress… and I didn’t which clearly means that my save file can’t be corrupt, and that the save files differ between platforms.

Remember the encryption method used? Well that’s also apparently the built in cocos2d-x game engine encryption method, which is used in the game, meaning the save files should not differ between platforms.

Anyway, away from that bit off-topic information. When I run my python script, before the error, I print the base64 encoded data:

Output:
b'x1fx8bx08x00x00x00x00x00x00x0bxdcx8cxcdx9fx84)_xfdx16x0fxb5xafxdbxf4-xccx7f4lxeaxbfxe2xd3xbbsxfa/xe9x82x97x8ex1bx14rxc2xd6xf6x9dxe6xe1x90Uxc1Jq!)xd2xc0xd4x1cxf8x9d3x14xc2ed`kRxf16nx83W_KlP7{xe5xd9Fx1dx95xc0xfatxb05x10xa81xc4:xd5Oxc7`!Xxfcx8e~x13xd2xa8Gxcexa3x8exefxa9xf8xcdx03>'

x1fx8bx08x00x00x00x00x00x00x0bxdcx8c
Something along those lines is the header for a gzip file.

According to that website, if the FLG byte is 0 (which it is for me, or at least with the header I appended), the data type is FTEXT which is «probably ASCII text», but the text inside of my file is definitely not ASCII.

So finally, my question is, why can I not unzip this data? (is it even gzip format?)

Feel free to clarify anything, it’s a bit confusing.

One last edit: I forgot to mention, when the files is decrypted, the output should be a plain text XML file.