Error internal error end of file from qemu monitor

Components:

CPU: AMD Ryzen 5 5600G

Motherboard: MSI B550-A Pro

GPU: AMD Radeon RX 560

I’m trying to pass through an AMD GPU to an Ubuntu desktop VM.

The VM works fine in VNC, but when using GPU passthrough, I get this error in the libvirt log:

2022-03-25 21:42:07.299+0000: 19863: error : qemuMonitorIO:578 : internal error: End of file from qemu monitor

It also shows «tainted: high-privileges» and «tainted: host-cpu».

I have also tried creating a Windows VM, and I get exactly the same error message. I have also tried passing it through with and without the sound card on the GPU.

At one point I actually managed to get the passthrough to work, with the Ubuntu login screen appearing on the monitor. But when I rebooted to add a keyboard, it didn’t work anymore, and I have not been able to replicate that (even deleting the VM and recreating the same settings).

Note that I have PCIe ACS override disabled, as it was not needed for separately IOMMU groups. The graphics and sound components of the card are by themselves in IOMMU group 9:

[1002:67ff] 10:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev cf)

[1002:aae0] 10:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]

we have 2 Hardware-Servers with Ubuntu 20.04 LTS on both we run Oracle linux 8 via KVM.
They were working fine until today. Both do not boot correctly with the following error message on the host machine:

internal error: End of file from qemu monitor

We have no clue why and why now. Nothing changed on the VM’s.

We run a backup-script on both and use Cockpit for monitoring the VM and setting them up, but had no problems so far.

This script ist the only thing running on both host-machines, so i thought there might be something with it but can´t find anything.

Have you experienced something simlar?

Backup Script:

#!/bin/bash
set +x
exec < /dev/null

[[ -f /tmp/backup-running ]] && exit 1
touch /tmp/backup-running

# Path, where backups should be stored
PDST=/vmbackup

# Path, where temporary snapshots should be created
SNAP=/var/backups/temp_snapshots

# Check and create working directories
[[ -d ${PDST} ]] || mkdir -p ${PDST}
[[ -d ${SNAP} ]] || mkdir -p ${SNAP}

date=$(date +"%Y%m%d")

#mount /dev/sda1 ${DST} >/dev/null 2>&1

if [[ "$?" -ne 0 ]]; then
    echo "Unable to mount backup drive!"
    exit 1
fi

echo coppy VM: $(virsh list --all --name)

for VM in $(virsh list --all  --name); do
    #Directory path
    DST=${PDST}/$VM
    #Check and create VM Backup Folder
    [[ -d ${DST} ]] || mkdir -p ${DST}
    #add file with date
    rm -f ${PDST}/LAST_TRY_*
    rm -f ${DST}/LATEST_*

    echo "Backup KVM guest ${VM}"

    # Make sure, snapshot merging was successfull
    image_path=$(virsh domblklist ${VM} --details | 
                 grep vda | 
                 awk '{ print $4; }' | 
                 cut -d "/" -f -5)

    if [[ "$image_path" != "/var/lib/libvirt/images" ]]; then
        continue # Existing snapshot. Skipping
    fi

    if [[ ! -f ${SNAP}/backup-snapshot-${VM}.qcow2 ]]; then
        echo "Create Snapshot..."
        virsh snapshot-create-as ${VM} "backup-${VM}" 
            --diskspec vda,file=${SNAP}/backup-snapshot-${VM}.qcow2 
            --disk-only 
            --atomic 
            --no-metadata
    else
        continue # Orphaned snapshot. Skipping
    fi

    echo "Copy guest image to backup drive..."
    cp -f --sparse=always /var/lib/libvirt/images/${VM}.qcow2 ${DST}/

    echo "Merge snapshot with guest image..."
    virsh blockcommit ${VM} vda --wait --active
    virsh blockjob ${VM} ${SNAP}/backup-snapshot-${VM}.qcow2 --pivot

    echo "Copy XML to backup drive..."
    cp -f /etc/libvirt/qemu/${VM}.xml ${DST}/

    # file with Date
    touch ${DST}/LATEST_$date

    # Make sure, snapshot merging was successfull
    image_path=$(virsh domblklist ${VM} --details | 
                 grep vda | 
                 awk '{ print $4; }' | 
                 cut -d "/" -f -5)

    if [[ "$image_path" == "/var/lib/libvirt/images" ]]; then
        echo "Remove snapshot..."
        rm -f ${SNAP}/backup-snapshot-${VM}.qcow2
    fi

    echo "VM as compressed_${VM}.qcow2"
    qemu-img convert -c -f qcow2 ${DST}/${VM}.qcow2 -O qcow2 ${DST}/compressed_${VM}.qcow2
    echo "entferne unkomprimiertes img ..."
    rm ${DST}/${VM}.qcow2
    echo "renaming VM to ${VM}.qcow2"
    mv ${DST}/compressed_${VM}.qcow2 ${DST}/${VM}.qcow2
done

touch ${PDST}/LAST_TRY_$date

echo "Syncing disk..."
sync
#umount ${DST}

rm -f /tmp/backup-running

exit 0

# vim: expandtab ts=4 sw=4

I created a kvm domain,but it always crash after running about a day.

Here are some error messages in libvirtd.log:

2020-08-13 13:46:15.906+0000: 2018: error : qemuMonitorIO:718 : internal error：End of file from qemu monitor
2020-08-13 13:46:15.906+0000: 2018: debug : qemuMonitorIO:765 : Error on monitor internal error：End of file from qemu monitor

and some error messages in the domain’s log:

qemu: qemu_thread_create: Resource temporarily unavailable

I have checked the free ram is enough

Run ulimit -a:

[root@localhost ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 385903
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 204800
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 1024
cpu time               (seconds, -t) unlimited
max user processes              (-u) 204800
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

This is defiantly something wrong with my script to deploy groups of VMs for a cluster.

New server, fresh gluster file space. Fresh KVM environment. Create template RHEL7 VM. Check my VM list to validate no «weird characters»….and now I have ghost VMs.. Symptom I get is when it starts to create the VM.. it fails on attaching the «opt» and «var» volumes to system as it lists a space between hostname and volume… so it fails to attach the the VM

[root@medusa ~]# virsh list —all
Id Name State
—————————————————-
— osmstr01
shut off
— rhel7 shut off

[root@medusa ~]#

Attempt to create issue again for baseline

# List of VMs in cluster to create
[root@medusa ~]# cat /media/sw2_usb_A1/penguinpages_local_cluster/os_cluster_nodes
osmstr01
osmstr02
osmstr03
osinf01
osinf02
osinf03
osworker01
osworker02
osworker03
osetcd01
osetcd02
osetcd03

# check for weird characters
[root@medusa ~]# vi /media/sw2_usb_A1/penguinpages_local_cluster/os_cluster_nodes
osmstr01$
osmstr02$
osmstr03$
osinf01$
osinf02$
osinf03$
osworker01$
osworker02$
osworker03$
osetcd01$
osetcd02$
osetcd03$
~
~
:set list

# Look at deployment script to see if something weird

[root@medusa ~]# vi /media/sw2_usb_A1/penguinpages_local_cluster/vm_deploy_os
#!/bin/bash$
# read in file and run loop against list of VMs to create VM$
# v0.4 2019-10-01$
for vm in $(cat /media/sw2_usb_A1/penguinpages_local_cluster/os_cluster_nodes)$
do$
virt-clone —original rhel7 —name «$vm» —auto-clone$
virsh setvcpus «$vm» 4 —config —maximum$
virsh setvcpus «$vm» 4 —config$
virsh setmaxmem «$vm» 16G —config$
sleep 10$
$
#Create OPT and VAR volumes for servers and mount and move data$
qemu-img create -f qcow2 /data/gv0/vms/»$vm»_opt.qcow2 80G$
virsh attach-disk —persistent —subdriver qcow2 $vm /data/gv0/vms/»$vm»_opt.qcow2 vdb$
sleep 5$
qemu-img create -f qcow2 /data/gv0/vms/»$vm»_var.qcow2 100G$
virsh attach-disk —persistent —subdriver qcow2 $vm /data/gv0/vms/»$vm»_var.qcow2 vdc$
sleep 5$
virt-sysprep -d $vm —hostname «$vm» —root-password password:1q2w3e4r —firstboot /media/sw2_usb_A1/penguinpages_local_cluster/vm_post_deploy_rhel$
virsh start $vm$
$
done $
~
~
:set list

# Nothing odd.. so lets deploy a bunch of VMs!!
[root@medusa ~]# sh /media/sw2_usb_A1/penguinpages_local_cluster/vm_deploy_os
-1.qcow2′ | 30 GB 00:00:17

‘ created successfully.

_opt.qcow2′, fmt=qcow2 size=85899345920 cluster_size=65536 lazy_refcounts=off refcount_bits=16
Disk attached successfully

_var.qcow2′, fmt=qcow2 size=107374182400 cluster_size=65536 lazy_refcounts=off refcount_bits=16
Disk attached successfully

[ 0.0] Examining the guest …
virt-sysprep: error: libguestfs error: /data/gv0/vms/osmstr01 -1.qcow2: No
such file or directory

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

virt-sysprep -v -x […]
error: Failed to start domain osmstr01

error: Cannot access storage file ‘/data/gv0/vms/osmstr01 _var.qcow2’ (as uid:107, gid:107): No such file or directory

-1.qcow2′ 8% [======== ] 306 MB/s | 2.7 GB 00:01:31 ETA
^C.qcow2′ 10% [========= ] 312 MB/s | 3.0 GB 00:01:28 ETA
^C
^C.qcow2′ 18% [================- ] 338 MB/s | 5.4 GB 00:01:14 ETA
Installation aborted at user request 21% [===================- ] 327 MB/s | 6.4 GB 00:01:14 ETA

^C
You have new mail in /var/spool/mail/root
[root@medusa ~]#

#

# List of VMs lies like a rug until it is restart!
[root@medusa ~]# virsh list —all
Id Name State
—————————————————-
— osmstr01
shut off
— osmstr01
shut off
shut off
— rhel7 shut off

[root@medusa ~]# !systemc
systemctl restart libvirtd
[root@medusa ~]# virsh list —all
Id Name State
—————————————————-
— osmstr01
shut off
— osmstr02
shut off
— rhel7 shut off

[root@medusa ~]#

# Now I have more VMs with weird characters

# And I cannot delete them at CLI
[root@medusa ~]# virsh list —all
Id Name State
—————————————————-
— osmstr01
shut off
— osmstr02
shut off
— rhel7 shut off

[root@medusa ~]# virsh undefine osmstr01
error: failed to get domain ‘osmstr01’
error: Domain not found: no domain with matching name ‘osmstr01’

[root@medusa ~]# virsh undefine osmstr02
error: failed to get domain ‘osmstr02’
error: Domain not found: no domain with matching name ‘osmstr02’

[root@medusa ~]#

# I can delete them in GUI… or rename them and do it at CLI

[root@medusa ~]# virsh list —all
Id Name State
—————————————————-
— foo shut off
— rhel7 shut off

[root@medusa ~]# virsh undefine foo
Domain foo has been undefined

[root@medusa ~]# virsh list —all
Id Name State
—————————————————-
— rhel7 shut off

[root@medusa ~]# !system
systemctl restart libvirtd
[root@medusa ~]# virsh list —all
Id Name State
—————————————————-
— osmstr01
shut off
— osmstr02
shut off
— rhel7 shut off

[root@medusa ~]#

Where is KVM storing this and why are they continuing to come back!! Ugh

/var/log/libvirt/libvirtd.log
2018-01-30 04:45:46.716+0000: 4947: info : libvirt version: 3.0.0, package: 4+deb9u1 (Guido Günther <agx@sigxcpu.org> Mon, 16 Oct 2017 22:48:55 +0200)
2018-01-30 04:45:46.716+0000: 4947: info : hostname: deb9
2018-01-30 04:45:46.716+0000: 4947: error : virDBusGetSystemBus:109 : internal error: Unable to get DBus system bus connection: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2018-01-30 04:45:46.716+0000: 4947: warning : networkStateInitialize:784 : DBus not available, disabling firewalld support in bridge_network_driver: internal error: Unable to get DBus system bus connection: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2018-01-30 04:45:47.609+0000: 4947: error : virSysinfoRead:1022 : internal error: Failed to find path for dmidecode binary
2018-01-30 04:45:47.609+0000: 4947: error : virSysinfoRead:1022 : internal error: Failed to find path for dmidecode binary
2018-01-30 04:45:47.616+0000: 4947: error : virNodeSuspendSupportsTarget:332 : internal error: Cannot probe for supported suspend types
2018-01-30 04:45:47.616+0000: 4947: warning : virQEMUCapsInit:1149 : Failed to get host power management capabilities
2018-01-30 04:45:49.205+0000: 4947: error : virSysinfoRead:1022 : internal error: Failed to find path for dmidecode binary
2018-01-30 04:45:49.421+0000: 4947: error : virNodeSuspendSupportsTarget:332 : internal error: Cannot probe for supported suspend types
2018-01-30 04:45:49.421+0000: 4947: warning : virLXCDriverCapsInit:88 : Failed to get host power management capabilities
2018-01-30 04:45:49.421+0000: 4947: error : virNodeSuspendSupportsTarget:332 : internal error: Cannot probe for supported suspend types
2018-01-30 04:45:49.421+0000: 4947: warning : umlCapsInit:74 : Failed to get host power management capabilities
2018-01-30 04:49:41.933+0000: 4931: error : qemuMonitorIO:695 : internal error: End of file from qemu monitor
2018-01-30 04:49:56.941+0000: 6432: error : virProcessKillPainfully:401 : Failed to terminate process 6378 with SIGKILL: Device or resource busy

/var/log/libvirt/libvirtd.log
2018-02-08 01:11:15.231+0000: 5211: info : libvirt version: 3.0.0, package: 4+deb9u1 (Guido Günther <agx@sigxcpu.org> Mon, 16 Oct 2017 22:48:55 +0200)
2018-02-08 01:11:15.231+0000: 5211: info : hostname: deb9
2018-02-08 01:11:15.231+0000: 5211: error : qemuMonitorIO:695 : internal error: End of file from qemu monitor

/var/log/libvirt/virtlogd.log
2018-02-08 01:07:25.208+0000: 5393: info : libvirt version: 3.0.0, package: 4+deb9u1 (Guido Günther <agx@sigxcpu.org> Mon, 16 Oct 2017 22:48:55 +0200)
2018-02-08 01:07:25.208+0000: 5393: info : hostname: deb9
2018-02-08 01:07:25.208+0000: 5393: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
2018-02-08 01:11:15.432+0000: 5393: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
2018-02-08 01:11:29.451+0000: 5393: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error