Zstd error 20 corrupted block detected

help in debugging ‘Decompression error: Corrupted block detected’ #2014

We do create millions of zstd compressed files. But recently, I started to get into occasional ‘Decompression error: Corrupted block detected’ like once in 10-12 days for 1 file. I tried to read stuff around like open issues and tried debugging where I checked system issues around that time but I don’t find anything.
File were not very small of single frame like I see in issues earlier. Example in one of case, It was of 40 MB uncompressed which became 22 MB compressed later.

Can someone points me how can I debug what could be causing it

The text was updated successfully, but these errors were encountered:

That’s a pretty difficult topic.

From our experience, detectable corruption errors typically come from 3 sources. In decreasing frequency order :

• Software issue : one user software, somewhere in the pipeline, has a bug, with the most typical one being truncating data. Buffer corruption is also in this list, with multi-threading increasing the odds of such thing happening. The most common bugs tend to happen more than once every 10 days. But some rare subtle ones may require many concurrent triggering conditions, and manifest only once in many millions cases.
• Hardware issue : so this is much rarer, but at the scale of gigantic datacenter, it is bound to happen from time to time. What’s very helpful in this case is to know which system generated the data. When the corruption errors consistently come from the same hardware unit, then it’s a pretty good sign.
• Library issue : we have not had to face such issue in recent time, but yes, it’s always a possibility. If the library has a bug, then it will generally show up in a wide range of systems and applications (since it’s integrated basically everywhere). This is a pretty good sign that something is wrong in the library itself.

Presuming you have no instrumentation and rely purely on the content of the frame to determine a potential cause, this is a very hard case scenario, with low prospects of identifying a root cause.

Something possible is to enable and store decompression logs, in order to determine where exactly the decompression process fails. To do that, one needs to compile zstd from source, adding DEBUGLEVEL=5 to the list of parameters. Higher levels will increase the verbosity of logs, but level 5 should be enough to determine which block exactly fails.

Knowing where the decompression process fails can be helpful, especially when it’s close to the beginning or the end of the frame. But sometimes, it’s simply not enough to determine a cause.

An important tool to debug corruption is an ability to track provenance and (sometimes) transfer of files. Without a dedicated tool for this goal, it’s still possible to add watermarks to compressed frames, by appending skippable frames, which are allowed to contain any custom content, so adding an «origin» field there can prove valuable at debug time.

The issue being : it must be already there to be useful. This won’t help for your current corrupted frames.

edit : the build macro is actually called DEBUGLEVEL .

Источник

[Decoding error (36) : Corrupted block detected] zstd file corruption #2852

Describe the bug
A clear and concise description of what the bug is.
We used the zstd compression format, but in the process of using it, we found that the ZSTD compressed file was occasionally damaged. We currently believe that it has been damaged during the process of writing to the disk.

So，we set debuglevel to 5, and find all corruption file have same problems, zstd max block size is 131072, but corruption file block size all are 390821.

I want to know how to avoid this problems, thx.

To Reproduce
Steps to reproduce the behavior:

1. Downloads data ‘. ‘
2. Run ‘. ‘ with flags ‘. ‘
3. Scroll up on the log to ‘. ‘
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots and charts
If applicable, add screenshots and charts to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. Mac]
• Version [e.g. 22]
• Compiler [e.g gcc]
• Flags [e.g O2]
• Other relevant hardware specs [e.g. Dual-core]
• Build system [e.g. Makefile]

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

Источник

ZSTD_decompress sometimes return corruption_detected instead of dstSize_tooSmall since 718f00ff6fe42db7e6ba09a7f7992b3e85283f77 #2222

Describe the bug
Since 718f00f, ZSTD_decompress for some buffer & size returns ZSTD_error_corruption_detected instead of ZSTD_error_dstSize_tooSmall depending on the size

To Reproduce
I have a reproducer but I can’t share it as it contains PII 🙁
I’ll happy to try any patch

git bisect run sh -c ‘git clean -xfd && make lib && gcc ../bugzstd/bugzstd.c -o bugzstd -Ilib/ lib/libzstd.a && ./bugzstd’
bisecting between v1.4.4 and v1.4.5 gave me 718f00f

This impacts librdkafka in confluentinc/librdkafka#2943
When ZSTD_getFrameContentSize returns ZSTD_CONTENTSIZE_UNKNOWN , librdkafka triple the dst buffer size until the decompression succeeds
Also librdkafka uses streaming compression but non streaming decompression
https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_zstd.c

Expected behavior
Both size return «Destination buffer is too small»

Actual Behavior
After 718f00f, the second call returns «Corrupted block detected»

Desktop (please complete the following information):

• OS: Fedora 32 & CentOS 7
• Compiler: gcc
• Flags: defaults
• Build system: Makefile

The text was updated successfully, but these errors were encountered:

Источник

ZSTD 1.4.5: Unable to begin ZSTD decompression (out buffer is 11022 bytes): Corrupted block detected #2943

The problem is likely in ZSTD, but I’m reporting it here to help/warn people
You can close it right away if you think there is nothing to do for you

Description

Using libzstd-1.4.5-3.el7.x86_64.rpm with librdkafka 1.0.1 or 1.4.4, I’m getting the following error on the consumer:

Data was produce by librdkafka 1.0.1 / libzstd 1.4.5

How to reproduce

To be done (might not have time before next week)

Producer:
kafkacat -P -E -X compression.codec=zstd -X queue.buffering.max.ms=100 -X broker.version.fallback=2.2.0 -X security.protocol=sasl_plaintext -X sasl.mechanisms=PLAIN -X sasl.username=kafkacat -X sasl.password=’ABCD’ -b 10.30.71.37 -t vme1

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.4.4 (at first was 1.0.1 )
• Apache Kafka version: 2.2.1
• librdkafka client configuration: api.version.fallback.ms=0, broker.version.fallback=0.11.0
• Operating system: CentOS 7
• Provide logs (with debug=.. as necessary) from librdkafka

• Provide broker log excerpts
• Critical issue

The text was updated successfully, but these errors were encountered:

Источник

Incompatibility between —long=31 and -22 —ultra? #1672

I tried to compress a file with zstd -22 —ultra —long=31 and then decompress it with zstd -d —long=31 and got the following error:

Is this a known issue? Just checking before I try to create a repro and/or debug this since it takes about 2 hours to compress the file (it’s

7GB), so it may take a while to reproduce the issue. Unfortunately, the file I’m compressing isn’t something I can pass around, so I can’t just send it to you.

I’ve tried this with the same file on two different platforms (different hardware and different Linux kernel versions) and have run into the same things both times. In both cases, the code I had was built from c0eb6c9, by running with make with no extra options specified.

The text was updated successfully, but these errors were encountered:

Hi @danluu ,
this seems related to #1653 , which was recently fixed.
Try commit 9038579 (or latest dev ) if possible.
Indeed, testing high compression levels on very large data set is quite lengthy .

Thanks for the quick response! I still can get an error with 096714d, although it happens at a slightly different place when decompressing:

From the linked issue, it sounds like it’s possible that the it doesn’t matter what data is used and only the size & settings matter? Let me see if I can get a repro with an all 0s file of the same size or a /dev/urandom file of the same size. If that doesn’t work, I can try generating some synthetic data that’s less unnatural.

I wasn’t able to reproduce with a toy file from /dev/* , but the file from #1653 (http://kirill.med.u-tokai.ac.jp/data/temp/Picea%20abies%20[genbank%20GCA_5F900067695.1%202016-11-09].fna.gz) gives me a similar looking error.

Because the file is huge, I compressed with 32 threads, zstd -22 —ultra —long=31 -T32 . On decompression ( zstd -d —long=31 ), I get 6178 MB. tmp.fna.zst : Decoding error (36) : Corrupted block detected when using 096714d.

Thanks for the feedback @danluu ,
and sorry for the long time it takes testing.

This issue was supposed to be fixed by #1659 .
It was tested with the same fna file, so it’s surprising that it’s still there.

Maybe the fix doesn’t work well enough.
There is also a small remaining possibility that the issue was actually correctly fixed in #1659, corresponding to commit 9038579, but sometimes later another commit generated a problem featuring same symptom (I’m suspecting 857e608, but that’s just a wild guess).

To be investigated .

No problem! Another curious thing is that I also tried single-threaded compression for a few «large» (> 10GB) files and they all hung at 6144MB compressed. One of them was the filed linked above and two of them were files pull of prime numbers that compressed and decompressed correctly when compressed with -T32 .

For all of the above, I built the update with make and didn’t do make clean , I did a make clean and started all of these again on the off chance it’s due to an issue with the Makefile .

That’s a very interesting data point, although there is a potential of confusion when labelling «single-threaded». It could be either :

• pass the —single-thread command
• use -T1 (which is the same as default)

—single-thread is the «real» single-threaded version. It’s equivalent to ZSTD_compressStream() . This version is supposed to be immune to bug in #1653, and if you nonetheless find that it still fails, then there is something else going on.

-T1 is actually «multi-threaded» by design, and just happens to use only one worker thread. The rest of the machinery works exactly the same as if it was N >= 2 threads, meaning there is a separate I/O thread, and work is split into pre-defined job sizes which are then passed onto workers, which preload data, etc. As a side-note that’s why zstd generates exactly the same compressed content whatever the nb of threads (for a given a version and compression level).

So -T1 will suffer exactly the same issue as any -T# , while —single-thread should not.

Thanks for the clarification! My previous comment refers to default threading, no T specified.

Yes, so default is equivalent to -T1 , which means, it uses the multi-thread engine, with only 1 active worker.
That’s the setup where we identified the issue with —long=31 .

Also, the nature of the bug (presuming it’s the same one) is that it overflows internal indexes.
And in order to skip all defenses related to index overflow, there are multiple conditions :

• Use a window size of 2 GB ( —long=31 )
• Use Multi-threading mode with overlapLog==9 (automatically the case with highest compression levels), thus triggering a history preload of 2 GB
• Re-use an existing compression context ( CCtx ), where the starting index is already high enough to trigger the overflow

Re-using an existing context is something which is done automatically internally. It will necessarily happen if nb_jobs > nb_workers .

But as a consequence, when specifying a very large number of workers, it can reach a point where nb_workers >= nb_jobs , hence each CCtx is only used once. And therefore, the bug should no longer be triggered.

This refers to your test using -T32 «that compressed and decompressed correctly». I don’t remember what’s the size of a job with —long=31 setting, but I would hazard that it’s probably 2 GB. Thus if the file is -T32 will create more workers than necessary, and each worker will use a brand new CCtx , thus circumventing the problematic re-usage pattern.

Finally, I should have mentioned it earlier :

zstd can also be compiled with all its assert() turned on, so that it would ideally abort during compression, hence much closer to the effective fault point.

This is done by using build macro DEBUGLEVEL

make V=1 clean zstd MOREFLAGS=-DDEBUGLEVEL=1

is one way to produce such a binary.

Thanks, I’ll try that in the future!

I was able to reproduce both the failure on decompression and the hang on compression after 6144MB with default T / T=1 with a clean build.

On the hangs, gdb shows them hanging at

The other spawned process looks less interesting?

Thanks @danluu !
That’s very helpful, it gives a clear hint of where to look for !

Indeed, #1659 cleared the issue for the main compression path, but ldm has a separate index data structure, and may well be prone to the same problem.

I’ve had this happen for multiple different files when using -T0 —long=31 —ultra -22 and I’d say it’s a very severe error because it silently corrupts your data making it impossible to recover — thankfully I still didn’t trust zstd enough yet and tried decompressing before deleting the original files first.

Also, it’s really not a niche use case because it happened to me for at least three different files and the above flag combination is simply the strongest compression level you can easily find from help. (As in, in my opinion you should release this as a hotfix as soon as possible and mark it as the severe problem it is so people can verify if they unknowingly corrupted their data)

This is an important bug to fix, and I’ll look into it this week. If you replace —long=31 with —zstd=wlog=31 you will probably work around the bug.

I’d argue it is a niche use case since it only occurs with large data on the highest settings, and most users won’t spend hours compressing their data. However, just because it is a niche use case doesn’t mean it isn’t worthy of a quick fix! We take any data corruption bugs extremely seriously.

I’ve been able to reproduce both the hang and the decoding error on the file pa.fna with sha1sum 4969c9cfc8880fdacfe9ff7d39a12ec2a69da047 .

I suspect that the decoding error and hang are closely related.

I was not able to reproduce it at level 16, so it might have something to do with btultra2.

Источник

That’s a pretty difficult topic.

From our experience, detectable corruption errors typically come from 3 sources. In decreasing frequency order :

• Software issue : one user software, somewhere in the pipeline, has a bug, with the most typical one being truncating data. Buffer corruption is also in this list, with multi-threading increasing the odds of such thing happening. The most common bugs tend to happen more than once every 10 days. But some rare subtle ones may require many concurrent triggering conditions, and manifest only once in many millions cases.
• Hardware issue : so this is much rarer, but at the scale of gigantic datacenter, it is bound to happen from time to time. What’s very helpful in this case is to know which system generated the data. When the corruption errors consistently come from the same hardware unit, then it’s a pretty good sign.
• Library issue : we have not had to face such issue in recent time, but yes, it’s always a possibility. If the library has a bug, then it will generally show up in a wide range of systems and applications (since it’s integrated basically everywhere). This is a pretty good sign that something is wrong in the library itself.

Presuming you have no instrumentation and rely purely on the content of the frame to determine a potential cause, this is a very hard case scenario, with low prospects of identifying a root cause.

Something possible is to enable and store decompression logs, in order to determine where exactly the decompression process fails. To do that, one needs to compile zstd from source, adding DEBUGLEVEL=5 to the list of parameters. Higher levels will increase the verbosity of logs, but level 5 should be enough to determine which block exactly fails.

Knowing where the decompression process fails can be helpful, especially when it’s close to the beginning or the end of the frame. But sometimes, it’s simply not enough to determine a cause.

An important tool to debug corruption is an ability to track provenance and (sometimes) transfer of files. Without a dedicated tool for this goal, it’s still possible to add watermarks to compressed frames, by appending skippable frames, which are allowed to contain any custom content, so adding an «origin» field there can prove valuable at debug time.

The issue being : it must be already there to be useful. This won’t help for your current corrupted frames.

edit : the build macro is actually called DEBUGLEVEL.

help in debugging ‘Decompression error: Corrupted block detected’ #2014

Comments

smanikim commented Feb 25, 2020 •

We do create millions of zstd compressed files. But recently, I started to get into occasional ‘Decompression error: Corrupted block detected’ like once in 10-12 days for 1 file. I tried to read stuff around like open issues and tried debugging where I checked system issues around that time but I don’t find anything.
File were not very small of single frame like I see in issues earlier. Example in one of case, It was of 40 MB uncompressed which became 22 MB compressed later.

Can someone points me how can I debug what could be causing it

The text was updated successfully, but these errors were encountered:

Cyan4973 commented Feb 25, 2020 •

That’s a pretty difficult topic.

From our experience, detectable corruption errors typically come from 3 sources. In decreasing frequency order :

• Software issue : one user software, somewhere in the pipeline, has a bug, with the most typical one being truncating data. Buffer corruption is also in this list, with multi-threading increasing the odds of such thing happening. The most common bugs tend to happen more than once every 10 days. But some rare subtle ones may require many concurrent triggering conditions, and manifest only once in many millions cases.
• Hardware issue : so this is much rarer, but at the scale of gigantic datacenter, it is bound to happen from time to time. What’s very helpful in this case is to know which system generated the data. When the corruption errors consistently come from the same hardware unit, then it’s a pretty good sign.
• Library issue : we have not had to face such issue in recent time, but yes, it’s always a possibility. If the library has a bug, then it will generally show up in a wide range of systems and applications (since it’s integrated basically everywhere). This is a pretty good sign that something is wrong in the library itself.

Presuming you have no instrumentation and rely purely on the content of the frame to determine a potential cause, this is a very hard case scenario, with low prospects of identifying a root cause.

Something possible is to enable and store decompression logs, in order to determine where exactly the decompression process fails. To do that, one needs to compile zstd from source, adding DEBUGLEVEL=5 to the list of parameters. Higher levels will increase the verbosity of logs, but level 5 should be enough to determine which block exactly fails.

Knowing where the decompression process fails can be helpful, especially when it’s close to the beginning or the end of the frame. But sometimes, it’s simply not enough to determine a cause.

An important tool to debug corruption is an ability to track provenance and (sometimes) transfer of files. Without a dedicated tool for this goal, it’s still possible to add watermarks to compressed frames, by appending skippable frames, which are allowed to contain any custom content, so adding an «origin» field there can prove valuable at debug time.

The issue being : it must be already there to be useful. This won’t help for your current corrupted frames.

edit : the build macro is actually called DEBUGLEVEL .

smanikim commented Feb 26, 2020 •

I looked from hardware and software issue perspective. Given the nature of issue and its occurence, I can not confirm as well as deny possibility. Just to add: We did not change anything in terms of hardware/zstd version etc. This issue was not happening 6-7 weeks ago but started occurring lately.

My build ran with below params but it did not change anything in decompression logs.

cc -I../lib -I../lib/common -I../lib/compress -I../lib/dictBuilder -DXXH_NAMESPACE=ZSTD_ -I../lib/legacy -DZSTD_MULTITHREAD -DDEBUGLOG=5 -DZSTD_LEGACY_SUPPORT=5 -O3 -DBACKTRACE_ENABLE=0 -pthread ../lib/common/debug.c ../lib/common/entropy_common.c ../lib/common/error_private.c ../lib/common/fse_decompress.c ../lib/common/pool.c ../lib/common/threading.c ../lib/common/xxhash.c ../lib/common/zstd_common.c ../lib/compress/fse_compress.c ../lib/compress/hist.c ../lib/compress/huf_compress.c ../lib/compress/zstd_compress.c ../lib/compress/zstd_compress_literals.c ../lib/compress/zstd_compress_sequences.c ../lib/compress/zstd_double_fast.c ../lib/compress/zstd_fast.c ../lib/compress/zstd_lazy.c ../lib/compress/zstd_ldm.c ../lib/compress/zstd_opt.c ../lib/compress/zstdmt_compress.c ../lib/decompress/huf_decompress.c ../lib/decompress/zstd_ddict.c ../lib/decompress/zstd_decompress.c ../lib/decompress/zstd_decompress_block.c ../lib/dictBuilder/cover.c ../lib/dictBuilder/divsufsort.c ../lib/dictBuilder/fastcover.c ../lib/dictBuilder/zdict.c ../lib/legacy/zstd_v05.c ../lib/legacy/zstd_v06.c ../lib/legacy/zstd_v07.c zstdcli.o util.o timefn.o fileio.o benchfn.o benchzstd.o datagen.o dibio.o -o zstd -pthread

logs are something like

$zstd -vvv -d in.tar.zst
*** zstd command line interface 64-bits v1.4.4, by Yann Collet ***
in.tar.zst : 5 MB. in.tar.zst : Decoding error (36) : Corrupted block detected

$ zstd -vvv -l in.tar.zst
*** zstd command line interface 64-bits v1.4.4, by Yann Collet ***
in.tar.zst
Zstandard Frames: 1
Window Size: 2.00 MB (2097152 B)
Compressed Size: 22.57 MB (23668400 B)
Check: XXH64

it actually enabled build debug logs. Do you think I compiled it right way

Источник

spurious corrupted block error with ZSTD_decompressContinue #918

Comments

romange commented Nov 17, 2017 •

Hi,
I am using double buffer technique to decompress the provided file bc_compress.zst that was compressed to support decompression parameters below. However on the attached file the program fails with «Corrupted block detected» error.
zstd —test bc_compress.zst succeeds. The program that demonstrates this behavior is here
Or below. I am using v1.3.2 version.

The text was updated successfully, but these errors were encountered:

Cyan4973 commented Nov 17, 2017

The frame uses a window size of 256 KB, which is exactly the size of the 2 allocated blocks.

Such window size means that, from time to time (but effectively very rarely, due to complex constraints on the encoder side), the encoder may generate a match which is barely short of 256 KB, like (256K-2) bytes for example.

Your code seems to imply that res==BLOCK_SIZE . Are you sure that’s always the case ?

romange commented Nov 17, 2017 •

The code implies that the uncompressed size of each block is at most BLOCK_SIZE. res can be smaller than BLOCK_SIZE.

The compression code used bound of ZSTD_compressBound(BLOCK_SIZE) to limit the raw data for each block and set ZSTD_parameters.cParams.windowLog = 18 .

Cyan4973 commented Nov 17, 2017 •

In which case, I believe it’s important to see the compression code, as many situations can potentially go wrong, but may be naturally avoided by your implementation.

Note that a safe buffer size for decoding is provided through function ZSTD_decodingBufferSize_min () , though it presumes a single continuous roll-buffer, instead of a double buffer methodology like your example.

romange commented Nov 17, 2017 •

I can not understand as well which path is faulty — decoder or encoder. The snippet from the compression code is given below.

1. Lets assume that my compression code is faulty. Then how come zstd —test passes?
2. Another bit of information: I tried to reproduce the compression run in a clean environment, i.e. to record both raw data (uncompressed bc_compress.zst) and the raw block sizes of the original run and then to «replay» the compression. Then I compared the compressed blocks from the clean environment to the according blocks from the attached file. I get differences in block 50 (0 based). It’s the same block that is reported as corrupted in the attached file.

Could it be that due to compression context reuse, the encoder somehow reaches a weird state?

romange commented Nov 17, 2017

the snippet compresses a single buffer. pos_ can be smaller than BLOCK_SIZE but not much. It’s always greater than BLOCK_SIZE — 128 .

romange commented Nov 17, 2017 •

A short update: if I recreate ZSTD_CCtx object for each frame — the problem goes away. I suspect ZSTD_compressBegin does not clear some state field which causes the compressor to reach an invalid state.

Cyan4973 commented Nov 17, 2017 •

I believe the issue is related to a large back reference, getting close to the limit ( ).

The reason it works fine with the reference decoder is that its internal buffer is large enough ( >= ZSTD_decodingBufferSize_min() ) to cope with any situation, as long as the windowSize limit is respected.

In your case, the decoder uses less memory, so it becomes reliant on assumptions on how the encoder works, which is more fragile.

Typically, I would expect the decoder to work when all blocks are always full.
Unfortunately, as mentioned, blocks are not necessarily full.
Depending on exact implementation details, there are several possible scenarios that can go wrong, in different ways.

Unfortunately, these scenarios can be pretty complex to describe, and require a few combined conditions, which make them rare to produce, though not impossible.
Consider this example :
Segment1 ( slot0 ): 128 KB
Segment2 ( slot1 , following slot0 , hence following Segment1 ) : 128 KB — 128
Segment3 ( slot0 again, hence not following Segment2 ) : 128 KB — 128

While compressing Segment3 , if Segment3 is using same memory slot as Segment1 , the compressor invalidates all positions from Segment1 , up to Segment3Size == 128 KB — 128.
Note that the last 128 bytes of Segment1 are still considered valid, because they are not invalidated, and belong to the same contiguous area as Segment2 , because it was full.

So now, let’s assume that at position (128 KB — 136) within Segment3 , which means Segment3End — 8 , the compressor generates a back-reference with a distance (256 KB — 136). This back-reference now points into Segment1 , at position (128 KB — 128). From the encoder perspective, this position is still valid.
For the decoder, since Segment1 is at slot 0, Segment2 at slot 1, and Segment3 at slot 0 again, Segment3 is overwriting Segment1 . This back-reference leads just 8 bytes in front of the current cursor.
Problem is, the decoder works in «stripes» : at each sequence it writes more than it needs, with the assumption that garbage data will be overwritten at next sequence. This generally works fine, but if next back-reference points at a position too close from previous match, some of its first bytes might be modified during previous sequence. In above scenario, this is possible if the previous match length is slot0 and slot1 , or by ensuring that all block sizes are necessarily

Источник

[Decoding error (36) : Corrupted block detected] zstd file corruption #2852

Comments

hk-lrzy commented Nov 11, 2021

Describe the bug
A clear and concise description of what the bug is.
We used the zstd compression format, but in the process of using it, we found that the ZSTD compressed file was occasionally damaged. We currently believe that it has been damaged during the process of writing to the disk.

So，we set debuglevel to 5, and find all corruption file have same problems, zstd max block size is 131072, but corruption file block size all are 390821.

I want to know how to avoid this problems, thx.

To Reproduce
Steps to reproduce the behavior:

1. Downloads data ‘. ‘
2. Run ‘. ‘ with flags ‘. ‘
3. Scroll up on the log to ‘. ‘
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots and charts
If applicable, add screenshots and charts to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. Mac]
• Version [e.g. 22]
• Compiler [e.g gcc]
• Flags [e.g O2]
• Other relevant hardware specs [e.g. Dual-core]
• Build system [e.g. Makefile]

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

Источник

ZSTD_decompress sometimes return corruption_detected instead of dstSize_tooSmall since 718f00ff6fe42db7e6ba09a7f7992b3e85283f77 #2222

Comments

champtar commented Jun 30, 2020

Describe the bug
Since 718f00f, ZSTD_decompress for some buffer & size returns ZSTD_error_corruption_detected instead of ZSTD_error_dstSize_tooSmall depending on the size

To Reproduce
I have a reproducer but I can’t share it as it contains PII 🙁
I’ll happy to try any patch

git bisect run sh -c ‘git clean -xfd && make lib && gcc ../bugzstd/bugzstd.c -o bugzstd -Ilib/ lib/libzstd.a && ./bugzstd’
bisecting between v1.4.4 and v1.4.5 gave me 718f00f

This impacts librdkafka in confluentinc/librdkafka#2943
When ZSTD_getFrameContentSize returns ZSTD_CONTENTSIZE_UNKNOWN , librdkafka triple the dst buffer size until the decompression succeeds
Also librdkafka uses streaming compression but non streaming decompression
https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_zstd.c

Expected behavior
Both size return «Destination buffer is too small»

Actual Behavior
After 718f00f, the second call returns «Corrupted block detected»

Desktop (please complete the following information):

• OS: Fedora 32 & CentOS 7
• Compiler: gcc
• Flags: defaults
• Build system: Makefile

The text was updated successfully, but these errors were encountered:

Источник

Zstd: Ошибки кодирования при использовании словаря с помощью zstdmt

Уважаемая команда zstd,
Кажется, есть проблема с декодированием zstd, как описано в этой ошибке:
# 883816
Я вижу ту же проблему в своей системе.

Все 10 Комментарий

Для расследования нам понадобится дело о воспроизведении.
Я пробовал эту тестовую установку, используя несколько тестовых файлов, но пока ни один из них не дал сбой.
Скорее всего, для отображения требуются определенные файлы.

Я бы хотел, но файлы, которые я пробовал, — это мой личный почтовый ящик, так что вы можете представить, что я не хочу этим делиться. Я не мог воспроизвести его на maildirs из публичных списков рассылки, таких как debian-devel. Однако я заметил, что всегда повреждаются самые большие файлы. Из

1500 файлов 1000 из них имеют размер от 1 до 10 килобайт, затем 400 из них имеют размер от 10 до 100 килобайт, а остальные имеют размер до 25 мегабайт. В списке рассылки debian-devel нет такого распределения, самый большой файл — всего 90 килобайт.

Я снова тестировал эту установку с расширенным набором тестов, содержащим большие файлы (> 10 МБ),
но пока ни один из них не провалился.
Боюсь, мне понадобится дело о репродукциях, чтобы провести расследование.

_Correction_: Я тестировал ветку dev .
Переключившись на v1.3.2 , как и в тесте debian, я могу заметить ошибку, особенно при использовании сжатия zstdmt со словарем для больших файлов.
Кажется, ошибка уже исправлена ​​в ветке dev , так как я не могу воспроизвести ее, используя те же файлы и словарь. Довольно сложно определить, какое именно изменение помогает, но я обычно смотрю в этом направлении .

Я бы порекомендовал протестировать ветку dev на тестовом наборе, где вы наблюдали проблему.

А пока я помещу предупреждение на v1.3.2 .

Мне удалось создать тестовый пример, которым я могу поделиться, который воспроизводит проблему в 100% случаев на моей машине. Поскольку он слишком велик, чтобы его можно было связать с проблемой GitHub, я временно сделал его доступным здесь:

Обратите внимание, у меня эта проблема возникает на ЦП с 6 ядрами, 12 потоками.

Спасибо за репродукцию @gsliepen.
Я могу подтвердить диагноз: v1.3.2 не работает с несколькими файлами (около дюжины, все> 4 МБ),
в то время как dev версия ветки отлично работает для всех файлов.
Таким образом, ложное исправление кажется уже интегрированным и будет частью следующего выпуска.

А пока, если вам срочно нужно быстрое исправление, вы можете выбрать один из следующих вариантов:

• при использовании cli: сжимайте файлы по одному, поскольку проблема возникает только тогда, когда несколько файлов сжимаются в одной команде.
• переключиться на ветку dev , так как там ошибка больше не возникает
• отключить многопоточность, так как это полезно только для больших файлов (> несколько МБ)
• отключить словарь, так как он эффективен только для небольших файлов ( 11 дек. 2017

git bisect с этим скриптом говорит, что он исправлен фиксацией fc8d293460d6767558843ac8231c249dcc704382 из PR # 891.

Спасибо, что посмотрели на него и за тестовый пример. Насколько я понимаю, исправление от commit fc8d293 не решает проблему. Я по-прежнему получаю сообщения «Обнаружен поврежденный блок» с тестовым примером, предоставленным @gsliepen.

Спасибо за тестирование @mestia! Мы полагаем, что выяснили проблему, но еще не опубликовали ее. Фиксация fc8d293 просто скрывает проблему при сжатии одного файла, но остается при сжатии нескольких файлов.

При сжатии в несколько потоков и со словарем мы можем перепутать размер окна и использовать для второго фрагмента размер окна больше, чем для первого. Поскольку размер окна первого фрагмента — это то, что записано в кадре, второй фрагмент может иметь слишком большие смещения.

Ветка dev скрыла эту проблему, но основная причина еще не решена. @ cyan4973 изучает это.

Если у вас есть поврежденные данные, которые вам необходимо восстановить, вы можете интерпретировать заголовок кадра, используя формат zstd, изменить размер окна на больший, тогда вы сможете успешно распаковать данные. Проверить заголовок можно с помощью zstd -lv file.zst . Сжатый фрейм по умолчанию содержит контрольную сумму, поэтому вы можете быть уверены, что она верна (вы можете проверить с помощью zstd -l file.zst ).

Мы рассматриваем возможность создания новой версии zstd позже на этой неделе, как только более глубокая основная причина будет полностью исследована и устранена. Эта ошибка требует, чтобы несколько условий были собраны для отображения, и действительно, текущая ветвь dev маскирует проблему, делая невозможным сгенерировать _все_ эти условия вместе из cli. Но это не означает, что основная причина устранена должным образом.
Еще больше будет после расследования.

В четверг, 14 декабря 2017 г., в 18:02:02 +0000 Янн Колле написал:

Закрыт № 944 через e28305fcca441564a57758d84da8ef73afe7d85d.

Я могу подтвердить, что это устранило проблему. Благодаря!

—
Встретил vriendelijke groet / с наилучшими пожеланиями,
Гус Слипен [email protected]

Источник

Страницы: [1]   Вниз

0 Пользователей и 1 Гость просматривают эту тему.

Страницы: [1]   Вверх

2020-07-25 03:54:28 INFO: Starting Backup of VM 102 (qemu)
2020-07-25 03:54:28 INFO: status = stopped
2020-07-25 03:54:28 INFO: backup mode: stop
2020-07-25 03:54:28 INFO: ionice priority: 7
2020-07-25 03:54:28 INFO: VM Name: foxtrot
2020-07-25 03:54:28 INFO: include disk 'scsi0' 'local-data:vm-102-disk-0' 25G
2020-07-25 03:54:28 INFO: include disk 'scsi1' 'local-data:vm-102-disk-1' 250G
2020-07-25 03:54:28 INFO: creating vzdump archive '/mnt/pve/local-backup/dump/vzdump-qemu-102-2020_07_25-03_54_28.vma.zst'
2020-07-25 03:54:28 INFO: starting kvm to execute backup task
2020-07-25 03:54:29 INFO: started backup task '1b057519-9c73-44cb-b2b9-82ba6f7dde53'
2020-07-25 03:54:32 INFO: status: 0% (112.9 MiB of 275.0 GiB), duration 3, read: 37.6 MiB/s, write: 22.7 MiB/s
2020-07-25 03:54:46 INFO: status: 1% (2.9 GiB of 275.0 GiB), duration 17, read: 201.4 MiB/s, write: 101.6 MiB/s
2020-07-25 03:55:07 INFO: status: 2% (5.9 GiB of 275.0 GiB), duration 38, read: 146.2 MiB/s, write: 101.3 MiB/s
2020-07-25 03:56:30 INFO: status: 3% (8.4 GiB of 275.0 GiB), duration 121, read: 30.8 MiB/s, write: 26.3 MiB/s
2020-07-25 03:57:58 INFO: status: 4% (11.0 GiB of 275.0 GiB), duration 209, read: 31.0 MiB/s, write: 29.7 MiB/s
2020-07-25 03:59:01 INFO: status: 5% (13.8 GiB of 275.0 GiB), duration 272, read: 45.9 MiB/s, write: 43.3 MiB/s
2020-07-25 03:59:06 INFO: status: 6% (17.2 GiB of 275.0 GiB), duration 277, read: 690.6 MiB/s, write: 76.6 MiB/s
.....
2020-07-25 04:24:50 INFO: status: 94% (258.6 GiB of 275.0 GiB), duration 1821, read: 148.1 MiB/s, write: 141.6 MiB/s
2020-07-25 04:25:17 INFO: status: 95% (261.3 GiB of 275.0 GiB), duration 1848, read: 104.5 MiB/s, write: 97.4 MiB/s
2020-07-25 04:25:49 INFO: status: 96% (264.1 GiB of 275.0 GiB), duration 1880, read: 88.4 MiB/s, write: 86.8 MiB/s
2020-07-25 04:26:35 INFO: status: 97% (266.8 GiB of 275.0 GiB), duration 1926, read: 60.6 MiB/s, write: 57.5 MiB/s
2020-07-25 04:26:59 INFO: status: 98% (269.6 GiB of 275.0 GiB), duration 1950, read: 120.8 MiB/s, write: 116.4 MiB/s
2020-07-25 04:27:24 INFO: status: 99% (272.3 GiB of 275.0 GiB), duration 1975, read: 109.3 MiB/s, write: 106.5 MiB/s
2020-07-25 04:27:52 INFO: status: 100% (275.0 GiB of 275.0 GiB), duration 2003, read: 99.0 MiB/s, write: 94.5 MiB/s
2020-07-25 04:27:52 INFO: transferred 275.00 GiB in 2003 seconds (140.6 MiB/s)
2020-07-25 04:27:52 INFO: Backup is sparse: 38% (105.40 GiB) zero data
2020-07-25 04:27:52 INFO: stopping kvm after backup task
2020-07-25 04:27:59 INFO: archive file size: 77.17GB
2020-07-25 04:27:59 INFO: Finished Backup of VM 102 (00:33:31)