Error processing tar file exit status 1 unexpected eof

Description

This was first reported on Mac by @flagbug: docker/for-mac#2388 (comment) but I believe it is a generic problem with the engine, not something specific to Mac.

To reproduce:

/tmp/test # cat Dockerfile
FROM ubuntu:18.04

RUN head -c 8G </dev/urandom >file

/tmp/test # docker build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:18.04
 ---> 735f80812f90
Step 2/2 : RUN head -c 8G </dev/urandom >file
 ---> Running in 4debf740eb97
Error processing tar file(exit status 1): unexpected EOF

Describe the results you received:

The error «Error processing tar file(exit status 1): unexpected EOF»

Describe the results you expected:

The build should complete successfully.

Additional information you deem important (e.g. issue happens only occasionally):

The logs have:

2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.598573889Z" level=debug msg="Tar with options on /var/lib/docker/overlay2/f89859e8af0b77222f209807333da05295ee03d1a9e2255895cf10ec57cce417/diff" storage-driver=overlay2
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.602482132Z" level=debug msg="Applying tar in /var/lib/docker/overlay2/278e40f6fc45ea9828cb090bf339b4f85561fa03dd2446626c3d9b6a272d8f5b/diff" storage-driver=overlay2
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.750205351Z" level=debug msg="Cleaning up layer 278e40f6fc45ea9828cb090bf339b4f85561fa03dd2446626c3d9b6a272d8f5b: Error processing tar file(exit status 1): unexpected EOF"
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.751542982Z" level=error msg="Can't add file /var/lib/docker/overlay2/f89859e8af0b77222f209807333da05295ee03d1a9e2255895cf10ec57cce417/diff/file to tar: io: read/write on closed pipe"
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.751599637Z" level=error msg="Can't close tar writer: io: read/write on closed pipe"

If I switch the command to head -c 8589934591 </dev/urandom >file i.e. 8GiB — 1 byte then it succeeds correctly.

I suspect this is same underlying issue as #37305 where a microsoft/mssql-server-linux database backup is restored and then docker committed. I suspect the root cause is that an individual file exceeds 8GiB, which is a limit in the tar header. According to wikipedia:

Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files. To overcome this limitation, star in 2001 introduced a base-256 coding that is indicated by setting the high-order bit of the leftmost byte of a numeric field. GNU-tar and BSD-tar followed this idea. Additionally, versions of tar from before the first POSIX standard from 1988 pad the values with spaces instead of zeroes.

Output of docker version:

/tmp/test # docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:04:40 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:13:46 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker info:

/tmp/test # docker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.93-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: linuxkit-025000000001
ID: UFJG:WKJW:OBVN:JP4C:NXIX:YJ7Z:JHQS:AT4L:RO2C:MN7I:YMUW:EQN7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 34
 Goroutines: 68
 System Time: 2018-08-02T14:49:58.963009758Z
 EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Docker error processing tar file occurs due to insufficient space or due to permission issues.

Here at Bobcares, we have seen several causes for this error while troubleshooting Docker issues as part of our Docker Hosting Support.

Today we’ll take a look at the cause for this error and how to fix it.

Why does Docker error processing tar file occur

Now let’s see why does this Docker error occur.

Generally, this error occurs while running any command in the Docker. This occurs if there is no space left on the device.
Also, in some cases, we noticed that this error also arose when there were some permission issues with certain files.

However, the reason for this error differs accordingly.

For instance, the error appears as below.

How we fix Docker error processing tar file

Let’s move to the solution part of this error. Here are the cases where we resolved this error for our customers.

1. Remove unused images

Recently, one of our customers approached us with the same error. He was trying to clear the space from his device and ran into the below error.

ERROR: Error processing tar file(exit status 1): unexpected EOF.

Now, let’s see how our Support Engineers fix this error for our customers.

First, we used a built-in command to remove unused images (Version 1.13+)

docker image prune

Then we ran the below command to stop the Docker.

systemctl stop docker

After that, we backed up the docker folder /var/lib/docker and removed it by running the below command

NOTE: This will remove images, containers, volumes, … make sure you back it up first.

rm -rf /var/lib/docker

Once this is done, we started the Docker service by running the below command.

systemctl start docker

This, fixed the error.

2. Permission issue

In a few cases, we saw that the files owned by root in the directory having permission issues. Providing the files proper permissions fixed the issue. For that, we used the below command.

chown -R <username>:<group> /var/lib/docker/tmp

[Need any further assistance in fixing Docker errors? – We’re available 24*7]

Conclusion

In short, this Docker error occurs due to insufficient space or due to permission issues. Today, we saw how our Support Engineers resolve this error to our customers.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;

Docker Error Processing tar file – Quick solution

Docker error processing tar file occurs due to insufficient space or due to permission issues.

Here at Bobcares, we have seen several causes for this error while troubleshooting Docker issues as part of our Docker Hosting Support.

Today we’ll take a look at the cause for this error and how to fix it.

Why does Docker error processing tar file occur

Now let’s see why does this Docker error occur.

Generally, this error occurs while running any command in the Docker. This occurs if there is no space left on the device.
Also, in some cases, we noticed that this error also arose when there were some permission issues with certain files.

However, the reason for this error differs accordingly.

For instance, the error appears as below.

How we fix Docker error processing tar file

Let’s move to the solution part of this error. Here are the cases where we resolved this error for our customers.

1. Remove unused images

Recently, one of our customers approached us with the same error. He was trying to clear the space from his device and ran into the below error.

ERROR: Error processing tar file(exit status 1): unexpected EOF.

Now, let’s see how our Support Engineers fix this error for our customers.

First, we used a built-in command to remove unused images (Version 1.13+)

Then we ran the below command to stop the Docker.

After that, we backed up the docker folder /var/lib/docker and removed it by running the below command

NOTE: This will remove images, containers, volumes, … make sure you back it up first.

Once this is done, we started the Docker service by running the below command.

This, fixed the error.

2. Permission issue

In a few cases, we saw that the files owned by root in the directory having permission issues. Providing the files proper permissions fixed the issue. For that, we used the below command.

[Need any further assistance in fixing Docker errors? – We’re available 24*7]

Conclusion

In short, this Docker error occurs due to insufficient space or due to permission issues. Today, we saw how our Support Engineers resolve this error to our customers.

Are you using Docker based apps?

There are proven ways to get even more out of your Docker containers! Let us help you.

Spend your time in growing business and we will take care of Docker Infrastructure for you.

Источник

docker pull fail with failed to register layer: Error processing tar file(exit status 1): invalid argument #97

Behavior

I’m building vm image for openstack, I wanna pull some docker images during build image process.
some how it can’t start docker daemon during package installation.

so i just start daemon manually

here is my script:

looks like pull image correctly, but have some problem while register layer

Steps to reproduce the behavior

Output of docker version :

Output of docker info :

Additional environment details (AWS, VirtualBox, physical, etc.)
vm disk image building process (use https://docs.openstack.org/diskimage-builder/latest/index.html)

The text was updated successfully, but these errors were encountered:

same as this issue
i’m in uncompatiable Backing filesystem : tmpfs

just close this.

after given image-disk-create —no-tmpfs .
able to use backing filesystem with ext4 but still have same problem here

build cmd:
disk-image-create -a amd64 —no-tmpfs —install-type package —image-size 5G -o ubuntu-$.qcow2 ubuntu bootloader devuser local-config openssh-server cleanup-kernel-initrd cloud-init-datasources

I have the same

$ sudo docker pull redis
Using default tag: latest
latest: Pulling from library/redis
be8881be8156: Extracting [==================================================>] 22.49MB/22.49MB
d6f5ea773ca3: Download complete
735cc65c0db4: Download complete
ff89c30e4d8c: Download complete
59bf782a86b3: Download complete
ce8aaa9fe90a: Download complete
failed to register layer: Error processing tar file(exit status 1): invalid argument

$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.06.0-ce
Storage Driver: overlay2
Backing Filesystem:
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Kernel Version: 4.4.0-17134-Microsoft
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.89GiB
Name: DESKTOP-V3N3P9H
ID: 3H42:T2GW:QWLI:AMZW:YKTH:3QCI:H5IL:SWYC:G337:PO2F:SIZQ:4CYI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support

how can I deal with it.

another same post here: moby/moby#34817 ,
maybe u can found solution there. (i haven’t found solution yet)
are you in some chroot / vm env. ?

I have same issue. However, I am in chroot environment. Wondering if it’s possible to make docker work in such environment.

./docker info

DEBU[2019-05-27T14:15:14.212268660Z] Calling GET /_ping
DEBU[2019-05-27T14:15:14.213233780Z] Calling GET /v1.39/info
WARN[2019-05-27T14:15:14.214784160Z] Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Kernel Version: 4.1.51-ws-symbol
OSType: linux
Architecture: aarch64
CPUs: 2
Total Memory: 870.7MiB
Name: (none)
ID: AWJE:2XHF:F6IX:AFBY:RKSO:ZAD5:QORR:OWBG:RXWM:UHBC:24JT:PT55
Docker Root Dir: /mnt/usb/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Источник

fail with failed to register layer: Error processing tar file(exit status 1): invalid argument #34817

Behavior

I’m building vm image for openstack, I wanna pull some docker images during build image process.
some how it can’t start docker daemon during package installation.

so i just start daemon manually

here is my script:

looks like pull image correctly, but have some problem while register layer

Steps to reproduce the behavior

Output of docker version :

Output of docker info :

Additional environment details (AWS, VirtualBox, physical, etc.)
vm disk image building process (use https://docs.openstack.org/diskimage-builder/latest/index.html)

The text was updated successfully, but these errors were encountered:

also try with docker import , but still doesn’t work.

use strace -F to see detail error

and my mount info during vm image building, get by cat /proc/self/mountinfo

after investigation , maybe because my host is using linux kernel 4.4.0 , but vm build for ubuntu(14.04) is still 3.13.

so causing problems during mounting AUFS.

try to build ubuntu xenial (default with kernel 4.4.0) is worked !

it happens again, despite using the same kernel version.
so I guess the problem isn’t all related to kernel version but related to mounted path. ?

seems has an error during extract tar file ..

maybe same issue here: #7585 (comment)

Are you seeing the same problem with other storage drivers? e.g., if you use overlay2 , do you get the same issue?

not try with overlay2

will try it when i have time

and what further info can i provide ?

in additional, try the solution mention in freenode channel

do mount -o bind chroot $chroot_dir; before chroot , but still doesn’t work .

strace file related operation.

but don’t know what root cause is.

maybe mount with MS_PRIVATE fail in nested chroot?

disk-image-builder process is here:

1.EINVAL source had an invalid superblock.

2.EINVAL A remount operation (MS_REMOUNT) was attempted, but source was
not already mounted on target.

3.EINVAL A move operation (MS_MOVE) was attempted, but source was not a
mount point, or was ‘/’.

4.EINVAL mountflags includes more than one of MS_SHARED, MS_PRIVATE,
MS_SLAVE, or MS_UNBINDABLE.

5.EINVAL mountflags includes MS_SHARED, MS_PRIVATE, MS_SLAVE, or
MS_UNBINDABLE and also includes a flag other than MS_REC or
MS_SILENT.

6.EINVAL An attempt was made to bind mount an unbindable mount.

Источник

Expected behavior

Success import image.

Actual behavior

«Error response from daemon: Error processing tar file(exit status 1): archive/tar: invalid tar header»

Information

• If we want to export container using > then we got wrong file/error on import.
• If we want to import image using | then we got error
• Diagnostic ID: 65607FD9-95F8-4697-86E9-82CAC340FD2A/2017-04-20_21-49-26
• Windows 10 Pro
• Docker 17.05.0-ce-rc1-wind8 (11189) edge 73d01bb
• Temporary solution for export is to use: docker export —output=»export.tar» container_id
• Temporary solution for import is to use: docker import export.tar

Steps to reproduce the behavior

1. docker export container_id > export.tar
2. cat export.tar | docker import — exampleimagelocal:new

The text was updated successfully, but these errors were encountered:

This could be related to a PowerShell limitation: #758

I made some tests on my machine with the different ouput types (STDOUT or file output switch -o) and the sizes differs quite a bit and when I opened the files in a diff-tool it complains about different carriage return types.

Moving on I also opened the two files in a hex editor, and turns out the version that was created using STDOUT had a hex 0x00 character for every second character. My guess is that it might be the classical problem where carriage return differs between linux and windows, and also a problem where windows uses UTF-16 by default and not UTF-8. Effectively corrupting any form of character strings or binary data.

My system is a windows 10 pro system and I also used powershell for generating the files. Might be good to mention that when I saved the file using the file output switch -o, i managed to import the file again, but when using STDOUT, the import yielded the exact same error.

Steps to reproduce:
1 open powershell
2. docker pull alpine
3. docker save alpine > alpine.tar
4. docker rmi alpine
5. docker load -i alpine.tar (Error processing tar file(exit status 1): archive/tar: invalid tar header)

The same procedure will work if doing it from a normal command window (cmd) so most likely a pure powershell problem. However using the file output switch (-o) when saving and not STDOUT will work from powershell so it’s probably best to avoid using any form of output streams when working in powershell.

Also files created with powershell are considerably larger. For example:

• Valid Cassandra image file created with CMD: 323,032KB
• Faulty Cassandra image created with Powershell: 653,040KB

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

Источник

Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root767627221: device or resource busy. #40008

Description

Steps to reproduce the issue:

Describe the results you received:

# docker run hello-world
Unable to find image ‘hello-world:latest’ locally
latest: Pulling from library/hello-world
1b930d010525: Extracting 977B/977B
docker: failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root767627221: device or resource busy.

Output of docker version :

Output of docker info :

The text was updated successfully, but these errors were encountered:

hm, I don’t recall having seen this error before. Did you keep the daemon and system logs from around the time this happened? Do you have software (e.g. anti-virus, or monitoring software) running on the host that could be keeping these files opened?

I haven’t seen that either. The error comes from here:

errCleanup = fmt . Errorf ( «Error cleaning up after pivot: %v» , errCleanup )

and this code was last changed in 2017. I have no idea why this could ever happen.

@Xelawk As suggested earlier, it would be beneficial to see the system logs around that time.

Faced the same problem during docker pull command.

Next attempts failing immediately:

The only info I can found in journalctl is:

Restarting docker service doesn’t help. Only system reboot helps.

And in general system starts to behave weirdly: sudo is asking for password after ≈10 seconds delay. So may be it is not docker issue and something suddenly broke in my system, but what? Disk space seems to be ok.

OS: Manjaro Linux. Docker installed via package manager.

@Envek that looks like a custom build of docker (perhaps it’s a distro package?) I see its built with Golang 1.14, which has breaking changes wrt handling of syscalls, which could potentially play a role. Are you seeing the same problem with packages from download.docker.com ?

Yes, it is a distro package docker 1:19.03.12-2 from Arch Linux. And it worked well for a year and broke just recently. Maybe after last update.

There is 2 years old bug in Arch packages FS#58690 but I can’t find any clues in it.

Any operation with docker fails and then system starts to behave weirdly.

They definitely shouldn’t be building with Go 1.14, but I’ve seen more issues with the Arch packages (there was one earlier this week where they used incorrect compile options for containerd, causing things to break in subtle ways).

Might be good to see if it reproduces on the official docker packages, or otherwise report to the Arch linux package issue tracker

Created new ticket in Arch bug tracker: FS#67639

Linux Kernel upgrade solved it for me: upgraded from 4.19.138 to 5.4.57

Yes upgrading kernel worked for me too. Just a quick reminder how to upgrade a kernel: https://wiki.manjaro.org/index.php/Manjaro_Kernels

Let me go ahead and close this issue, but feel free to continue the conversation

I can confirm this, had the same error message and have fixed it by switching from kernel 4.19 to 5.4.

I don’t really know what the cause is, but I can contribute additional findings: From what I could see, it seems that after a docker pull some symlinks on the host system were broken, even though their targets should have still been available.

/var/run should be a symlink to ../run (which would be /run ), but i got completely different results when listing the contents of /var/run and /run . This seems to be the reason why the docker CLI will fail completely after a pull attempt, it expects /var/run/docker.sock , but that is only available as /run/docker.sock due to the failing symlink.

/etc/mtab should be a symlink to ../proc/self/mounts , but is inaccessible (broken) after the fact. This turns out to be an issue if you want to install any packages with pacman after the fact.

All that is fixed by a reboot, too, but I haven’t found a way to perform a docker pull on kernel 4.19.

I have the exact same problem, although I trigger it in a different way. I use packer to create containers. I ran into this error when copying a file to the container. But after the «docker: Upload failed» text, everything is the same. The odd message about pivot. The truly bizarre behavior of the symlinks failing. Etc. etc.

I experimented a bit with downgrading and rolling back to a snapshot and upgrading specific things.

This odd problem behavior isn’t solely due to the change in the docker level.

I tested the permutations of:

docker: 19.03.11-1 and 19.03.12-2
containerd: 1.3.4-2 and 1.4.0-2
linux: 4.19.126-1 and 4.19.143-1

The definitive factor was the change in the linux kernel. On 4.19.126, I had no issue no matter the versions of the other things. On 4.19.143, this problem case arose no matter the versions of the other things. I first experienced the problem on 4.19.141. So somewhere between 126 and 141, something changed in the kernel that created this problem.

This, to me, dovetails with others’ experience that upgrading the kernel to 5 resolved the issue.

I faced exactly the same issue on Manjaro with 4.19.143-1-MANJARO x86_64 after docker updated to 19.03.12-ce
Upgrading kernel to 5.4.62-1-MANJARO x86_64 solved problem.

I got the same issue 2 hours ago after i worked already more than 4 hours today with docker and weeks before. What killed it in the end was doing docker-compose down —rmi=all . After that docker won’t work anymore. Additionally my system was getting into a weird state where everything with sudo involved is taking about 10 seconds to execute. I’ll try the kernel suggestion 😛

/Update: Sadly i cannot update as my system couldn’t find the boot partition after using the kernel version 5.4. So i tried to switch back to finding the issue. As jketterl mentioned, it has to do something with broken symlinks. I’ll try to find out which symlinks are broken (similiar to the broken /etc/mtab symlink). What currently bugs me most is why did it happen in the first place? It seems like this a serious issue when system relevant symlinks are getting invalid. (I fixed the /etc/mtab by myself so i can use pacman again, but the origin of the issue is still unknown)

Additionally my system was getting into a weird state where everything with sudo involved is taking about 10 seconds to execute.

Yes, I also noticed this.

The same problem happened to me, also.

And /etc/mtab is broken.

Linux jimmy-arch 4.19.152-1-MANJARO #1 SMP Sat Oct 17 10:51:27 UTC 2020 x86_64 GNU/Linux

Error still remains. Cannot update to Kernel 5.4 as i get the following error:

Packages (1) linux54-5.4.74-1

Total Download Size: 85.02 MiB
Total Installed Size: 140.16 MiB

:: Proceed with installation? [Y/n] Y
error: could not open file: /etc/mtab: No such file or directory
error: could not determine filesystem mount points
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.

Edit:
I think I found a solution:

I just had this same problem, same OS, Manjaro, with kernel 4.19.152-1-MANJARO. Unfortunately, I cannot upgrade to 5.4, or anything higher, because my laptop randomly freezes with any of those kernels.

Did anybody find out the root cause, or should I try to downgrade my kernel to 4.19.126-1 (or, well, the latest 4.19 working kernel)?

Bug was introduced into Manjaro linux419 (https://gitlab.manjaro.org/packages/core/linux419)
at commit 98da16f.

Specifically, the problem is proc_mounts.patch which was added then.

Last good release was 4.19.133 (if it was released — cannot be sure) or 4.19.132.

You can build the latest kernel without proc_mounts.patch. Three modifications are required
in PKGBUILD: the call to patch; the source; the checksum.

I have tested this on 4.19.152. It does fix the problem.

Applying proc_mounts.patch to Manjaro 5.4 recreates the exact same problem.

Applying proc_mounts.patch to ArchLinux 5.4 does not create the problem.

I think the aufs patches are tripping over themselves. But I’m not yet sure.

Furthermore, I believe the problem would affect containers in general and not just docker.

I have thoughts about the true root cause. But they’re not much further beyond speculation.
So I decided to post this much at the moment to help anyone stuck.

Источник