Io error saving source cache

I did a system update with sudo apt-get update and rebooted. After the reboot an error came up showing that i'm out of disk space. was barly able to access my desktop. I checked the space util and

I did a system update with sudo apt-get update and rebooted. After the reboot an error came up showing that i’m out of disk space. was barly able to access my desktop.
I checked the space util and noticed that there are 2 very large files (over 20GB) in the directory: /var/lib/apt/list/partial/

  • za.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-amd64_Packages.bz2 (21.2 GB)
  • security.ubuntu.com_ubuntu_dists_vivid-security_universe_binary-amd64_Packages.bz2 (21.2 GB)

I managed to delete them and rebooted the system and all was fine. Then I did an update again and the same issue occured again.

Is this a bug?

System info :
Ubuntu 15.04 — Daily updated

Carl H's user avatar

Carl H

6,0115 gold badges26 silver badges40 bronze badges

asked May 28, 2015 at 7:26

Shawn Botha's user avatar

If you look, for example for your first file here, you’ll notice that the size of these files is barely 100KB, so you have something pesky going on—probably a network failure or something like that. Have you tried to check the content of these monster files? Is the filesystem clean?

Try to remove all the cached packages:

sudo apt-get clean

Then repair the database:

sudo apt-get install -f

Finally repeat the update and upgrade:

sudo apt-get update 
sudo apt-get upgrade

Synetech's user avatar

answered May 28, 2015 at 9:01

Rmano's user avatar

RmanoRmano

31.2k15 gold badges116 silver badges187 bronze badges

I’ve experienced this bug that made my computer runs very slow. At first I thought it’s because the new ppa I’ve just added. But then I discover that it’s a bug that has been confirmed in the launchpad:
https://bugs.launchpad.net/ubuntu/vivid/+source/apt/+bug/1445239

It turns out there’s a problem in the ServerState::HeaderLine() where it parses

Content-Range header:

Content-Range: bytes 587-600/601

And in HttpServerState::RunData() there’s a code In.Limit(Size - StartPos); that returns a -1 hence infinite loop for the MaxGet.

Go check the thread for full info.

Launchpad user Anton Blanchard and the others have made a patch for the bug, but it’s still in the dev distro of Willy Werewolf and waiting for the developers of Vivid to fix this problem.

For now I suggest to be careful when running the apt-get update and be standby to cancel it if the lists/partial folder gets to grown into the gigabytes size. Deleting the folder will temporarily fixed the problem for me.
Or you could schedule a cron jobs for it.

answered Jun 7, 2015 at 3:35

gnomeria's user avatar

sudo apt-get update updates the package manager with the latest versions of software — this is what the two .bz2 files are. It does not actually install new software.

You need to follow sudo apt-get update with sudo apt-get upgrade, which will install the latest versions of everything.

You can combine them into one command with

sudo apt-get update && sudo apt-get upgrade

answered May 28, 2015 at 8:19

Carl H's user avatar

Carl HCarl H

6,0115 gold badges26 silver badges40 bronze badges

3

Источник

Output of df -h:


Filesystem                       Size  Used Avail Use% Mounted on
udev                             979M  4.0K  979M   1% /dev
tmpfs                            197M  1.1M  196M   1% /run
/dev/disk/by-partlabel/userdata  9.3G  3.1G  6.3G  34% /userdata
/dev/loop0                       2.0G  2.0G     0 100% /
/dev/loop1                       243M  231M  6.5M  98% /android/system
none                             4.0K     0  4.0K   0% /android
tmpfs                            982M  4.0K  982M   1% /etc/fstab
tmpfs                            982M  2.0M  980M   1% /var/lib/lxc/android/rootfs
/dev/disk/by-partlabel/system    3.8G  8.2M  3.8G   1% /android/cache
/dev/disk/by-partlabel/protect1  5.9M   44K  5.6M   1% /android/protect_f
/dev/disk/by-partlabel/protect2  5.9M   44K  5.6M   1% /android/protect_s
none                             4.0K     0  4.0K   0% /sys/fs/cgroup
tmpfs                            982M   16K  982M   1% /tmp
none                             5.0M     0  5.0M   0% /run/lock
none                             982M  188K  981M   1% /run/shm
none                             100M     0  100M   0% /run/user
cgmfs                            100K     0  100K   0% /run/cgmanager/fs
tmpfs                            982M     0  982M   0% /media
tmpfs                            982M     0  982M   0% /var/lib/openvpn/chroot/tmp
tmpfs                            982M     0  982M   0% /var/lib/sudo
tmpfs                            197M   40K  197M   1% /run/user/32011
tmpfs                            197M     0  197M   0% /run/user/0
/dev/mmcblk1                      60G  256K   60G   1% /media/phablet/7FE6-1555

I noticed that /android/system was using lots of space, even tough this device never had android installed on it.

I decided to reinstall Ubuntu Touch, just for good measure. This is the output of df -h after re-installation:

Filesystem                       Size  Used Avail Use% Mounted on
udev                             979M  4.0K  979M   1% /dev
tmpfs                            197M  1.1M  196M   1% /run
/dev/disk/by-partlabel/userdata  9.3G  1.9G  7.5G  20% /userdata
/dev/loop0                       2.0G  1.8G  114M  94% /
/dev/loop1                       243M  231M  6.5M  98% /android/system
none                             4.0K     0  4.0K   0% /android
tmpfs                            982M  4.0K  982M   1% /etc/fstab
tmpfs                            982M  2.0M  980M   1% /var/lib/lxc/android/rootfs
/dev/disk/by-partlabel/system    3.8G  8.2M  3.8G   1% /android/cache
/dev/disk/by-partlabel/protect1  5.9M   44K  5.6M   1% /android/protect_f
/dev/disk/by-partlabel/protect2  5.9M   44K  5.6M   1% /android/protect_s
none                             4.0K     0  4.0K   0% /sys/fs/cgroup
tmpfs                            982M   16K  982M   1% /tmp
none                             5.0M     0  5.0M   0% /run/lock
none                             982M  244K  981M   1% /run/shm
none                             100M     0  100M   0% /run/user
cgmfs                            100K     0  100K   0% /run/cgmanager/fs
tmpfs                            982M     0  982M   0% /media
tmpfs                            982M     0  982M   0% /var/lib/openvpn/chroot/tmp
tmpfs                            982M     0  982M   0% /var/lib/sudo
tmpfs                            197M   48K  197M   1% /run/user/32011
tmpfs                            197M     0  197M   0% /run/user/0
/dev/mmcblk1                      60G  256K   60G   1% /media/phablet/7FE6-1555

I then ran the commands you suggested, these are the messages they returned:

sudo apt update:

W: No sandbox user '_apt' on the system, can not drop privileges
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.

sudo apt install anbox-ubuntu-touch android-tools-adb:

E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.

NOTE: The device reports 7.8 GB of free space in the settings menu, and this is a fresh installation, so their should be plenty of free space

Источник

  • #1

Hi,

While upgrading to 7.0, in the end this is the output.

Code: 

root@vms:~# apt upgrade
Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.
root@vms:~# df -i
Filesystem                  Inodes   IUsed     IFree IUse% Mounted on
udev                       3052638     742   3051896    1% /dev
tmpfs                      3068300  139840   2928460    5% /run
/dev/mapper/pve-root        966656  207986    758670   22% /
tmpfs                      3068300      95   3068205    1% /dev/shm
tmpfs                      3068300      30   3068270    1% /run/lock
tmpfs                      3068300      18   3068282    1% /sys/fs/cgroup
/dev/sdg1                305201152 1190911 304010241    1% /mnt/pve/TenTB
/dev/sdf1                 61054976    1080  61053896    1% /mnt/S1TB
/dev/sda2                        0       0         0     - /boot/efi
/dev/sdc1                 14655488      63  14655425    1% /mnt/pve/SSDStorage
/dev/sde1                244195328   86234 244109094    1% /mnt/4TB
/dev/sdd1                 61054976    5026  61049950    1% /mnt/pve/WDOneTB
/dev/sdb1                 61054976   11979  61042997    1% /mnt/pve/OneTB
//192.168.40.2/TempShare         0       0         0     - /mnt/pve/TempShare
tmpfs                      3068300      11   3068289    1% /run/user/0
/dev/fuse                   262144      73    262071    1% /etc/pve
root@vms:~#

t.lamprecht


  • #2

And the actual space usage (i.e., the non-inode one?) df?

Nefariousparity


  • #3

I wonder if it is freaking out over the NFSSMB share there.

t.lamprecht


  • #4

Would be strange, as we explicitly check the root mount point «/» only.

  • #5

@t.lamprecht @Nefariousparity Proxmox terminal is very slow over ssh and I not able to login to gui anymore. Login Falied Error. I scroll up the console and found flood of these.

Code: 

cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/bpck.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/dstr.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/epia.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pt.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/on26.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/friq.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/fit3.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pf.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/kbic.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/comm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pcd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/aoe/aoe.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/brd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/floppy.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/mtip32xx/mtip32xx.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/zram/zram.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-tcp.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/ib_core.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/ib_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/iw_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/rdma_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-rdma.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fc.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/target/nvmet.ko': No such file or directory

  • #6

@t.lamprecht @Nefariousparity Found the problem /dev/mapper/pve-root was full in storage. But still can’t login I am getting «

Connection failure. Network error or Proxmox VE services not running?»

Code: 

root@vms:~# df
Filesystem                1K-blocks       Used  Available Use% Mounted on
udev                       12210552          0   12210552   0% /dev
tmpfs                       2454644     588312    1866332  24% /run
/dev/mapper/pve-root       15158232   15141848          0 100% /
tmpfs                      12273200      48772   12224428   1% /dev/shm
tmpfs                          5120          0       5120   0% /run/lock
tmpfs                      12273200          0   12273200   0% /sys/fs/cgroup
/dev/sdg1                9688383588 6723466588 2476578828  74% /mnt/pve/TenTB
/dev/sdf1                 960380648  910283528    1242660 100% /mnt/S1TB
/dev/sda2                    523248        312     522936   1% /boot/efi
/dev/sdc1                 229701520  136949532   81014104  63% /mnt/pve/SSDStorage
/dev/sde1                3844639728 2084415876 1564856636  58% /mnt/4TB
/dev/sdd1                 960380628  516056904  395469264  57% /mnt/pve/WDOneTB
/dev/sdb1                 960380628  460214484  451311684  51% /mnt/pve/OneTB
//192.168.40.2/TempShare   29439884    1921524   27518360   7% /mnt/pve/TempShare
tmpfs                       2454640          0    2454640   0% /run/user/0
/dev/fuse                    131072         36     131036   1% /etc/pve

t.lamprecht


  • #7

So clearly no space left as a normal df call would show: df /

You need to cleanup some files to advance, if your lucky only some log file went crazy and takes up lots of space, those could be deleted relatively safely.

What I do in such cases is using «du» (disk usage) tool to find out the directories with the biggest usages, I normally start on root, tell du to not cross into other file-systems (-x option), output human-readable units (-h), and only report to 1 directory depth -level (-d1). e.g., first du -hxd1 / | sort -h and then if that shows /var as biggest user I use that as new start point: du -hxd1 /var/ | sort -h

If you’re unsure about that I’d recommend posting the following output here, so we can help

Bash: 

du -hxd2 / | sort -h | tail -20

(may need quite a while depending on IO speed)

t.lamprecht


  • #8

Found the problem /dev/mapper/pve-root was full in storage. But still can’t login I am getting »

Yeah, that’s the root /, as indicated by pve6to7 ;) «was» as in «I made some space»?

  • #9

@t.lamprecht @Nefariousparity Thanks. It was /root filled NVIDIA driver files, as was trying pass gpu lxc month back. Forgot to cleanup. Any idea about login issue?

t.lamprecht


  • #10

I think they failed to start during an upgrade.

Check: systemctl list-units --failed

I’d guess that you need to restart at least pveproxy and pveproxy service:
systemctl try-reload-or-restart pveproxy.service pvedaemon.service

There may be other failed services requiring a restart too though.

Last edited: Jul 13, 2021

  • #11

@t.lamprecht This is what it’s showing

Code: 

root@vms:~# systemctl list-units --failed
  UNIT                     LOAD   ACTIVE SUB    DESCRIPTION
● dcservice.service        loaded failed failed Service for ManageEngine UEMS Agent
● nfs-blkmap.service       loaded failed failed pNFS block layout mapping daemon
● pve-daily-update.service loaded failed failed Daily PVE download activities
● pvesr.service            loaded failed failed Proxmox VE replication runner

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
4 loaded units listed.

Code: 

root@vms:~# systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-06-26 18:27:24 EDT; 2 weeks 2 days ago
    Process: 2825741 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
   Main PID: 1905 (pveproxy)
      Tasks: 4 (limit: 28618)
     Memory: 246.6M
     CGroup: /system.slice/pveproxy.service
             ├─   1905 pveproxy
             ├─2911473 pveproxy worker
             ├─2911504 pveproxy worker
             └─2913851 pveproxy worker

Jul 13 02:18:56 vms pveproxy[1905]: worker 2911504 started
Jul 13 02:19:26 vms pveproxy[2911443]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1795.
Jul 13 02:19:26 vms pveproxy[2911443]: error writing access log
Jul 13 02:26:31 vms pveproxy[2911443]: worker exit
Jul 13 02:26:31 vms pveproxy[1905]: worker 2911443 finished
Jul 13 02:26:31 vms pveproxy[1905]: starting 1 worker(s)
Jul 13 02:26:31 vms pveproxy[1905]: worker 2913851 started
Jul 13 02:37:03 vms pveproxy[2913851]: proxy detected vanished client connection
Jul 13 02:39:30 vms pveproxy[2911504]: proxy detected vanished client connection
Jul 13 02:47:44 vms pveproxy[2911473]: proxy detected vanished client connection

Code: 

root@vms:~# systemctl status pvesr.service
● pvesr.service - Proxmox VE replication runner
     Loaded: loaded (/lib/systemd/system/pvesr.service; static)
     Active: failed (Result: timeout) since Tue 2021-07-13 02:43:15 EDT; 13min ago
TriggeredBy: ● pvesr.timer
   Main PID: 2832705
      Tasks: 1 (limit: 28618)
     Memory: 68.0M
     CGroup: /system.slice/pvesr.service
             └─2832705 /usr/bin/perl -T /usr/bin/pvesr run --mail 1

Jul 13 00:28:00 vms systemd[1]: Starting Proxmox VE replication runner...
Jul 13 02:38:45 vms systemd[1]: pvesr.service: State 'stop-sigterm' timed out. Killing.
Jul 13 02:38:45 vms systemd[1]: pvesr.service: Killing process 2832705 (pvesr) with signal SIGKILL.
Jul 13 02:40:15 vms systemd[1]: pvesr.service: Processes still around after SIGKILL. Ignoring.
Jul 13 02:41:45 vms systemd[1]: pvesr.service: State 'final-sigterm' timed out. Killing.
Jul 13 02:41:45 vms systemd[1]: pvesr.service: Killing process 2832705 (pvesr) with signal SIGKILL.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Processes still around after final SIGKILL. Entering failed mode.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Failed with result 'timeout'.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Unit process 2832705 (pvesr) remains running after unit stopped.
Jul 13 02:43:15 vms systemd[1]: Stopped Proxmox VE replication runner.
root@vms:~# systemctl start pvesr.service
Failed to start pvesr.service: Transaction for pvesr.service/start is destructive (cryptsetup.target has 'stop' job queued, but 'start' is included in transaction).
See system logs and 'systemctl status pvesr.service' for details.
root@vms:~# systemctl restart pvesr.service

Last edited: Jul 13, 2021

t.lamprecht


  • #12

systemctl start pvesr.service

pvesr (storage replication) is managed through the pvesr.timer unit, so it cannot be started directly — for it you can just reset the failed state (systemctl reset-failed pvesr.service.

Jul 13 02:18:56 vms pveproxy[1905]: worker 2911504 started
Jul 13 02:19:26 vms pveproxy[2911443]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1795.
Jul 13 02:19:26 vms pveproxy[2911443]: error writing access log
Jul 13 02:26:31 vms pveproxy[2911443]: worker exit

The pveproxy is the main API daemon, and it was definitively affected when space run out, does it work now after a full reload?

Else I’d recommend restarting pveproxy.service and pvedaemon.service via systemctl.

  • #13

@t.lamprecht I was not able to reboot or shutdown the Proxmox server from terminal, It kept waiting for something to stop or something for hours. Then finally I hard rebooted it via power button(physical-one). Then Proxmox went into emergency mode, then I ran apt update and apt upgrade and reboot it and It’s working now, all VMS and LXCs are up.

t.lamprecht


  • #14

There were probably half finished upgrades applied when your host ran out of space, so repeating the apt update and apt full-upgrade (avoid a plain upgrade command, especially on major upgrades) was a good call.

Glad to hear that all is up again.

Источник

Hello,

I have encountered an issue where my normal boot process is failing.
Using the recovery mode, I am able to arrive at ttly and the command prompt to logon however.

In searching the forums, I have been able to see a discussion about a similar issue, and have attempted to run dpkg — repair broken packages via the recovery menu option.

Reference: viewtopic.php?f=46&t=305330&p=1714897&h … e#p1714897

I am using a 2nd computer here, and have typed out the output from running dpkg:

could not resolve ‘archive.ubuntu.com’
some index files failed to download
write error — write (28: no space left on device)
IO error saving source cache
tre package lists or status files could not be parsed or opened…

In using the system summary command, the following info is available:
/ 100% usage 19GB of the 19GB

Prior to this issue surfacing, I was receiving a space warning message, and I believe this is because I had set up timeshift to save to my root /.
There seemed to be another message about 3-4 files [maybe dependency files — or the index files?]
I thought I had resolved the issue by deleting one of my timeshift backups, so there was 1.5GB free, although this is likely not enough space.

Background: using LM 20.02, which is a new install [previously worked very well], 256 GB SSD which is has a dual boot set up.
The dual boot set up set created a lack of space in general, and I set up the linux partitions based on / = 19 GB, Home = 120 GB, swap = 8 GB.
I have moved the timeshift backup location to the larger 120GB home partition, prior to this issue surfacing.

At this point, my objective is to restore the normal boot, perhaps through resolving the index file and space issues.
I suspect I should increase the / root partition from 20 GB to perhaps 40 GB?

I have quite limited technical knowledge, but have been successful at resolve issues, if anyone is willing to offer their expertise and detailed steps to follow.
Thank-you for any expertise the community is willing to offer.

Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 2 times in total.

Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.

Источник

#amazon-web-services #ubuntu #nginx #jenkins #amazon-ec2

Вопрос:

У меня есть экземпляр Ubuntu ec2 под управлением nginx и Дженкинса. Больше нет свободного места для выполнения обновлений, и каждая команда, которую я пытаюсь освободить, не работает. Кроме того, при попытке связаться с Дженкинсом я получаю 502 Плохих шлюза.

Когда я бегу sudo apt-get update , я получаю длинный список ошибок, но главной из них, которая выделялась, была E: Write error - write (28: No space left on device)

Я понятия не имею, почему больше нет места или что это вызвало, но df -h выдает следующий результат:

 Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           394M  732K  393M   1% /run
/dev/xvda1       15G   15G     0 100% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/loop1       56M   56M     0 100% /snap/core18/1988
/dev/loop3       34M   34M     0 100% /snap/amazon-ssm-agent/3552
/dev/loop0      100M  100M     0 100% /snap/core/10958
/dev/loop2       56M   56M     0 100% /snap/core18/1997
/dev/loop4      100M  100M     0 100% /snap/core/10908
/dev/loop5       33M   33M     0 100% /snap/amazon-ssm-agent/2996
tmpfs           394M     0  394M   0% /run/user/1000

Я пытался освободить место, бегая sudo apt-get autoremove , и это дало мне E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

Я побежал sudo dpkg --configure -a и получил pkg: error: failed to write status database record about 'libexpat1-dev:amd64' to '/var/lib/dpkg/status': No space left on device

Наконец, я побежал sudo apt-get clean; sudo apt-get autoclean , и это дало мне следующие ошибки:

 Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.

Мы будем очень признательны за любую помощь, чтобы освободить место и снова запустить сервер.

Комментарии:

1. Почему бы не расширить раздел на 15 ГБ, если вам нужно больше места?

Ответ №1:

В моем случае у меня есть приложение с контейнерами nginx, postgresql и gunicorn. Я выполнил эти шаги, чтобы решить свою проблему,

  1. Я попытался выяснить, какие файлы потребляют мое хранилище больше всего, используя приведенную ниже команду

    sudo найти / -тип f -размер 10 м -exec ls -lh {} ;

  2. Как вы можете видеть на скриншоте, оказывается, что неиспользуемые и связанные с докером контейнеры являются источником

введите описание изображения здесь

  1. Затем я удаляю все неиспользуемые, остановленные или зависшие изображения

    система докеров обрезка -а

В конце концов мне удалось вернуть около 4,4 ГБ!

Ответ №2:

Для сервера (если вы не тестируете Jenkins amp; Nginx), на котором работает Jenkins amp; Nginx, вы должны лучше управлять разделом диска. Ниже приведены несколько возможных способов устранения вашей проблемы.

  1. Увеличьте существующий размер корневого тома EBS EC2 с 15 ГБ до более высокого значения с помощью консоли AWS EBS.

    или

  2. Найдите файлы, занимающие много места на диске, и удалите их, если это не требуется. Скорее всего, файлы журналов занимают место на диске. Вы можете выполнить следующие команды, чтобы узнать места, которые занимают больше места.

    cd /

    du -sch * | grep G

    или

  3. Добавьте дополнительный том EBS в свой экземпляр и смонтируйте его в домашнем каталоге Jenkins или в том месте, где используется больше места на диске.
Источник

Понравилась статья? Поделить с друзьями:
  • Invoke webrequest the underlying connection was closed an unexpected error occurred on a send
  • Invisible boot device windows 10 как исправить
  • Inventor как изменить единицы измерения
  • Invalidargumenterror graph execution error
  • Invalid zip file format twrp как исправить