Lzma library error corrupted input data

Corrupt input data when using lzma to decompress a file #92018

Bug report

The built-in lzma module failed to decompress a valid .lzma file with the following errors:

The .lzma file was compressed using the public-domain C library written by Igor Pavlov, see
https://github.com/fangq/zmat/tree/master/src/easylzma/pavlov

before compression, the binary buffer has a length of 1966104 bytes, after compression, the file, mat.lzma (can be downloaded from this link) has a length of 1536957 bytes.

when running file mat.lzma , it prints

I was able to decompress this file using either the C library mentioned above, or using the below NodeJS/JavaScript script (with either lzma-purejs or lzma npm modules)

the above script corrected decoded the buffer:

however, using the below python script, I got an error

Because Igor Pavlov’s C library implements the original lzma algorithm, so I believe the FORMAT_ALONE flag was used correctly.

I want to mention that the test file mat.lzma can be correctly decompressed using lzma -d on Ubuntu 20.04, but it gives an error on Ubuntu 18.04 and 22.04 (both uses xz utils based lzma), I believe this is due to the nature that the two lzma commands are different

Your environment

Python 3.6 on Ubuntu 18.04
Python 3.8 on Ubuntu 20.04
Python 3.10 on Ubuntu 22.04

The text was updated successfully, but these errors were encountered:

Does it work with FORMAT_RAW?

@serhiy-storchaka, no, FORMAT_RAW also failed

It seems you are hitting a problem with how xz (and its lzma lib used by python) handles the end-of-stream marker.
As can be seen in xz’s lzma_decoder.c, lzma_decode() has a problem when this marker is present while the uncompressed size is known.

Which doesn’t match Igor Pavlov’s specs from his DOC/lzma-specification.txt:

If «Uncompressed size» field contains ones in all 64 bits, it means that
uncompressed size is unknown and there is the «end marker» in stream,
that indicates the end of decoding point.
In opposite case, if the value from «Uncompressed size» field is not
equal to ((2^64) — 1), the LZMA stream decoding must be finished after
specified number of bytes (Uncompressed size) is decoded. And if there
is the «end marker», the LZMA decoder must read that marker also.

xz and python can decompress your file after replacing the uncompressed size at the head of your file:

Источник

Arch Linux

You are not logged in.

#1 2013-01-27 13:57:09

System update borked due to Lzma library error/need advice [FIXED]

It seems that my glibc update was bad and now my system is throwing error I can’t fix. I do not know how to recover from this.

Last edited by maggie (2013-01-27 14:45:40)

#2 2013-01-27 14:01:06

Re: System update borked due to Lzma library error/need advice [FIXED]

Can you downgrade glibc and filesystem from the ARM and repeat?

#3 2013-01-27 14:07:02

Re: System update borked due to Lzma library error/need advice [FIXED]

It still gives me conflicting file errors and the news items say to not to force it.

#4 2013-01-27 14:09:37

Re: System update borked due to Lzma library error/need advice [FIXED]

I believe that refers to the new update, since you are restoring a previous state, I don’t think it matters. you can try to force it or wait for someone with more experience with these things to post. I would force it, but I have pretty robust backups. Your call.

#5 2013-01-27 14:12:28

Re: System update borked due to Lzma library error/need advice [FIXED]

Downgrading with forcing worked and my update was successful. What would cause those errors? Why didn’t pacman fail the package if it was broken before it tried to install the package?

#6 2013-01-27 14:14:18

Re: System update borked due to Lzma library error/need advice [FIXED]

Glad it worked for you. I have no idea why the package failed the integrity check. Maybe Allan or falconindy will see this thread and comment?

Last edited by graysky (2013-01-27 14:26:06)

#7 2013-01-27 14:23:41

Re: System update borked due to Lzma library error/need advice [FIXED]

It passed the integrity check, so the download is fine. I’d be running memcheck.

#8 2013-01-27 14:46:10

Re: System update borked due to Lzma library error/need advice [FIXED]

Thank you graysky and allan. The memtest fails within 20 seconds. Time to RMA the memory.

Источник

pkg update fail to unpack downloaded archive. #1290

Since about a week or two, cannot update packages — pkg update fails on unpacking downloaded file. There is plenty of disk space (over 50Gb) and permissions are correct everywhere.

Here is and machine info

and pkg update output

The text was updated successfully, but these errors were encountered:

Which pkg version are you using? (pkg -vv output will be enough)
As I can see something is blocking exchange between pkg and remote site. Are you using proxy server?

No proxy or any blocking software used. On the same server I can «lynx to» http://pkg0.bme.FreeBSD.org/FreeBSD:10:amd64/latest without any problems, download any archive and all downloaded archives are correct.

Running pkg —debug update -f also doesn’t reveal much info, the same error about extract error and signature.

And here is output you asked for.

That’s strange. Can you please for comment one line in your /usr/local/etc/pkg/repos/FreeBSD.conf to make sure mirror is OK for sure (like this):

1. fetch http://pkg0.bme.FreeBSD.org/FreeBSD:10:amd64/latest/packagesite.txz
2. fetch http://pkg0.bme.FreeBSD.org/FreeBSD:10:amd64/latest/meta.txz
3. md5 packagesite.txz
4. md5 meta.txz

MD5 (packagesite.txz) = cae27b2be033ed4ef95a9321eae983b0
MD5 (meta.txz) = f06576a4e1314ad60ea207807fca0387

Yandex mirror gives the same. Upgrade to 10.1 didn’t help. I’m not familiar with all procedures pkg does upon updating packages but it looks like a broken pkg update mechanism.

BTW if there is a way to put reliably known correct FreeBSD.meta instead of zero-length one and try to run package update again? But where to get one?

hum can you show please show me uname -a and file /bin/sh
can you also
Why are you enforcing a mirror?
As that box have been upgraded from 9 to 10?
if yes can you remove your /usr/local/etc/pkg/repo/FreeBSD.conf and let the one already available in base to the job?

Yes, it used to be 9.x and then 10.0 when it came to production and now 10.1

I guess mirror enforcing left since 9.x as there were some issued but that didn’t appear to be a problem until the last week or so. Putting back default (at any location either /etc or /usr/local…) doesn’t really help — produces the same error.

Kernel is custom built one but options hasn’t changed since 10.0 when pkg used to work

I suspect something went wrong during you upgrade, can you try first pkg bootstrap -f
then pkg update?

Bootstrapping didn’t help, unfortunately. As regard to upgrade, whole thing used to work and time to time I do run pkg upgrade to get recent packages. Just about the last week that didn’t work and started to give me this error.

The message that disturbs me come from pkg/libpkg/pkg_repo.c# lines 141-144. To me it looks like a broken signature/hash checking routines on the particular installation but why that could have happened — pure mystery 😔

i did try to repeat this case, but no luck.
Can you trace run pkg under ktrace and provide dump via some URL?

I’m not a guru of reading dumps but this part, which is just after receiving chunks of data from remote host looks disturbing

What’s that capability mode is?

I really do not understand what tries to run stat and is not allowed to.

Sorry I misread the kdump, this failure you are showing are not related to your issue (even if they should be fixed) the issue happens before, All the «No signature found» beside I can see the signature being read in the sandbox, but somehow it is not passed to the application.

is it possible that you do grab the pkg distfiles, run ./configure and send me the config.log and pkg_config.h that has been created?

Also can you show ldd /usr/local/sbin/pkg and try pkg-static update -f command, just for check. Will result will be same or not.

So, ldd shows no problem

but… pkg-static worked! and updated FreeBSD.meta. Now when I ran pkg upgrade I got next «feedback»:

Does it mean Lzma is broken?

Right so that confirms something is wrong with your upgrade! the only difference between pkg-static and pkg is that pkg-static the fact that pkg-static does not use local libraries.

How to you perform your upgrade?

Btw the is strange:

Installed packages to be REINSTALLED:
pkg-1.5.5 (needed shared library changed)

and is not supposed to happen given pkg «need libraries»

Well, as usual: freebsd-update -r 10.1 upgrade .

BTW, @bapt do you still need ./configure output?

That will help yes

P.S. If there are PMs at GitHub?

hum I see nothing wrong in you configure.

I meant Private Messages 🙂

Hm, same ldd from my box (10.1-RELEASE-p15)

You have some libthr.
Can you show cat /etc/libmap.conf and ls -la /usr/local/etc/libmap.d (if it exist)

First file include second folder and both are empty.

well, bad news here is that your system damaged..
freebsd-update as i can see don’t work with custom kernels, so i don’t know how exactly update was done or maybe it was aborted in the middle. But libthr sense is something from past..

I can only suggest you to try fetch 10.1-RELEASE sources from SVN and rebuild system(kernel/userbase) once more time. It strange what you don’t having issues with other applications on this server.

at i think you can try use pig-static till that moment as workaround. If it will work for you.

freebsd-update will work with custom kernels. You should edit /etc/freebsd-update.conf and change:

(or drop src as well if you’re checking the system sources out of SVN.) Of course, given this you will need to build and install a new kernel when a patch level update affects its sources.

OP could also try freebsd-update’s IDS mode to see if you can spot any libraries that don’t match the known checksums from the freebsd-update distribution.

Источник

Arch Linux

You are not logged in.

#1 2010-04-13 11:09:14

[SOLVED] Can’t upgrade anything!

having returned from a week away, i now have a couple of hundred mbs of packages to upgrade.
unfortunately, it would appear that NONE of the packages in the new tar.xz format will work, providing me with an error saying

File kernel26-2.6.33.2-1-i686.pkg.tar.xz is corrupted. Do you want to delete it?

this appears for virtually every package. there was an error once involving libarchive, which i upgraded on its own (and that oddly worked) and the error went away. there was also previously an error with libssl, also, which i fixed by adding new symlinks. i now have a semi-upgraded and confused system. help!

EDIT: I am using the most up-to-date mirror that there is

Last edited by benji.ijneb (2010-04-18 17:13:36)

#2 2010-04-13 11:47:26

Re: [SOLVED] Can’t upgrade anything!

Happens once in a while to me — keep downloading and sooner or later you’ll get rid of it.

That’s my experience fwiw.

#3 2010-04-13 11:48:27

Re: [SOLVED] Can’t upgrade anything!

Did you «pacman -Syy» after changes mirrors? This is almost certainly a mirror problem.

All men have stood for freedom.
For freedom is the man that will turn the world upside down.
Gerrard Winstanley.

#4 2010-04-13 13:14:57

Re: [SOLVED] Can’t upgrade anything!

> there was an error once involving libarchive, which i upgraded on its own (and that oddly worked)
Maybe libarchive is still in .pkg.tar.gz format? New pacman & co. handle the .pkg.tar.xz files w/o a problem.

#5 2010-04-13 18:52:20

Re: [SOLVED] Can’t upgrade anything!

thanks loads. after waiting a bit, many of the packages are now installing.
but there are now two new errors, both of which are related to LZMA compression.
some packages (eg kernel26) give me this once they have downloaded, but before they install/upgrade:

error: error while reading package /var/cache/pacman/pkg/kernel26-2.6.33.2-1-i686.pkg.tar.xz: Lzma library error: Corrupted input dataTruncated input file (needed 5896704 bytes, only 0 available)
error: failed to commit transaction (libarchive error)

some, once the install has completed, give me this:

upgrading xorg-server [##################] 100% error: could not extract usr/bin/xvfb-run (Lzma library error: Corrupted input data) error: problem occurred while upgrading xorg-server (36/50)

thanks for your help so far!

#6 2010-04-14 22:52:29

Re: [SOLVED] Can’t upgrade anything!

bump. any help with this?

#7 2010-04-14 23:07:40

Re: [SOLVED] Can’t upgrade anything!

Show us your /etc/pacman.conf and the output of ‘pacman -Q pacman’.

#8 2010-04-14 23:31:12

Re: [SOLVED] Can’t upgrade anything!

]$ pacman -Q pacman
pacman 3.3.3-5

#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir = /
#DBPath = /var/lib/pacman/
#CacheDir = /var/cache/pacman/pkg/
#LogFile = /var/log/pacman.log
HoldPkg = pacman glibc
# If upgrades are available for these packages they will be asked for first
SyncFirst = pacman
#XferCommand = /usr/bin/wget —passive-ftp -c -O %o %u
#XferCommand = /usr/bin/curl %u > %o
#CleanMethod = KeepInstalled

# Pacman won’t upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg = nvidia nvidia-utils
#IgnoreGroup =

# Misc options (all disabled by default)
#NoPassiveFtp
#UseSyslog
#ShowSize
#UseDelta
#TotalDownload

#
# REPOSITORIES
# — can be defined here or included from another file
# — pacman will search repositories in the order defined here
# — local/custom mirrors can be added here or in separate files
# — repositories listed first will take precedence when packages
# have identical names, regardless of version number
# — URLs will have $repo replaced by the name of the current repo
#
# Repository entries are of the format:
# [repo-name]
# Server = ServerName
# Include = IncludePath
#
# The header [repo-name] is crucial — it must be present and
# uncommented to enable the repo.
#

Источник

Bug report

The built-in lzma module failed to decompress a valid .lzma file with the following errors:

_lzma.LZMAError: Corrupt input data

The .lzma file was compressed using the public-domain C library written by Igor Pavlov, see
https://github.com/fangq/zmat/tree/master/src/easylzma/pavlov

before compression, the binary buffer has a length of 1966104 bytes, after compression, the file, mat.lzma (can be downloaded from this link) has a length of 1536957 bytes.

when running file mat.lzma, it prints

mat.lzma: LZMA compressed data, non-streamed, size 1966104

I was able to decompress this file using either the C library mentioned above, or using the below NodeJS/JavaScript script (with either lzma-purejs or lzma npm modules)

const fs = require('fs')
const lzma = require('lzma-purejs')

async function main() {
  var data=lzma.decompressFile(fs.readFileSync('mat.lzma'));
  console.log(data.length)
}
main().then(() => console.log('Done'))

the above script corrected decoded the buffer:

$ node testlzma.js 
1966104
Done

however, using the below python script, I got an error

import lzma

filename='mat.lzma'
buf=lzma.open(filename,  format=lzma.FORMAT_ALONE).read();

print(len(buf))

error message:

$ python3 testlzma.py
Traceback (most recent call last):
  File "testlzma.py", line 4, in <module>
    buf=lzma.open(filename,  format=lzma.FORMAT_ALONE).read();
  File "/usr/lib/python3.6/lzma.py", line 200, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.6/_compression.py", line 103, in read
    data = self._decompressor.decompress(rawblock, size)
_lzma.LZMAError: Corrupt input data

Because Igor Pavlov’s C library implements the original lzma algorithm, so I believe the FORMAT_ALONE flag was used correctly.

I want to mention that the test file mat.lzma can be correctly decompressed using lzma -d on Ubuntu 20.04, but it gives an error on Ubuntu 18.04 and 22.04 (both uses xz utils based lzma), I believe this is due to the nature that the two lzma commands are different

fangq@ubuntu20_04:~$ lzma --version
LZMA command line tool 9.22
LZMA SDK 9.22

fangq@ubuntu20_04:~$ lzma -v -d mat.lzma
mat.lzma:	 21.83% -- replaced with mat

fangq@ubuntu18_04:~$ lzma --version
xz (XZ Utils) 5.2.2
liblzma 5.2.2

fangq@ubuntu18_04$ lzma -v -d mat.lzma
mat.lzma (1/1)
 99.9 %   1,500.9 KiB / 1,920.0 KiB = 0.782                                    
lzma: mat.lzma: Compressed data is corrupt
 99.9 %   1,500.9 KiB / 1,920.0 KiB = 0.782     

Your environment

Python 3.6 on Ubuntu 18.04
Python 3.8 on Ubuntu 20.04
Python 3.10 on Ubuntu 22.04