Host key verification failed task error failed to run vncproxy

[SOLVED] Failed to run vncproxy in all VMs / Windows VMs no longer booting

audioPhil

New Member

we are running Proxmox on a single server (no Cluster) and VMs are stored locally on an LVM storage.
Currently the system is running

10-12 VMs. Most of them are Linux server systems, but we are also running two Windows machines.
During everyday use, all VMs were running fine and no issues occurred (SSH, HTTP, SMB, RDP were all behaving as expected).

Today, my colleague tried to boot up a Windows VM which had been offline for some weeks. He realized he could not get an RDP connection and so he tried to access the machine via the Proxmox noVNC console.
However, this was not working either. The Console tab shows «Verbinden. » (I suspect the English translation would be «Connecting. «) and after a while the following error appears:

This is when I got involved into this issue (I am sort of the main administrator of the Proxmox instance).

So far I found out / tried the following:
— For almost all of the VMs the noVNC console is not working.
— For some non-critical systems I tried to initiate a shutdown via the Web GUI, as well as via qm shutdown . The systems with a non-working noVNC console cannot be shutdown this way. The systems shows VM quit/powerdown failed .
— I logged into some of the Linux VMs via SSH and initiated a shutdown. I then restarted these VMs via the Web GUI and they are now working as expected (noVNC is showing, I can now initiate a shutdown too).
— I tried the same for one of the (up to that point still working) Windows VMs. It does not come back online, is neither accessible via RDP nor noVNC. While a start of the Windows VM seems successful via the Web GUI, starting it via qm start shows

— The problem also occurs if I try to create a new Windows VM from scratch. Not even the initial boot into the installer ISO file is working.
— The problem does not occur if I try to create a new Linux VM from scratch. The installer loads perfectly as expected.
— I installed the latest packages ( apt update && apt upgrade ) but nothing changed.
— I did not yet restart the hypervisor itself. One of the Windows VMs is still running and I would prefer not to lose access to it too, as it is in daily use.
— I searched for the issue on the Proxmox forum and found this thread which describes a similar behaviour regarding the «Failed to run vncproxy» part. However, the issue described there has been resolved with an upgrade to pve-qemu-kvm 4.0.0-7 . We are running pve-qemu-kvm/stable 5.0.0-11 amd64

What could have led to this error?
A week ago I updated the packages on the Proxmox instance. After that, I did not reboot the hypervisor. As we are not using the noVNC feature on a daily base it could very well be that one of the updates led to the problem and has remained undetected until today.
However, I have one Linux VM with an uptime of 6 days (rebooted right after the update of Proxmox) which is working perfectly, while another Linux VM with an uptime of 4 days (rebooted multiple days after the update) is not working.

Likely not related, but something I changed recently:
I introduced named admin accounts for my colleagues and me and deactivated the default root account. This had some unexpected side effects such as updates no longer being possible via the Web GUI. Therefore, I updated via the CLI using apt . To rule this out as a possible reason I reactivated the root account today. The issues persist.

Information on our system:

I am out of ideas and hope that someone might be able to help. If required, I could organize a reboot of the hypervisor relatively quickly. A short downtime of the VMs is no problem but I am afraid that we will then be without any working Windows VM.

Источник

[SOLVED] Failed to run vncproxy

New Member

I have a two node setup. From the first node, when I pull up the console for a VM on the second node, I get this:

Debian GNU/Linux 9
Permission denied (publickey).
TASK ERROR: Failed to run vncproxy.

I can’t tell if this is an SSH issue or a vncproxy issue. I have verified that the authorized_keys are set properly to match the ssh keys on each host.

Any idea on troubleshooting this? Proxmox version 5.1-41.

New Member

Tamilarasan

New Member

how did you solve the issue? could pls provide the steps?

DanielWood

New Member

how did you solve the issue? could pls provide the steps?

I came across a similar issue.

The hint of root ssh access by the original poster caused me to stumble upon the solution.

My problem was when I tried to access the console from the WebUI for one host for a VM on another host. The fix for me was to open the web ssh console (Shell) on the other hosts from the WebUI of the primary host. When I did, I was greeted with a prompt to accept the SSH fingerprint. I suspect this was the problem to begin with. As soon as I accepted the fingerprint, I was able to see the console on other hosts.

In fact, I was able to validate it since I still had more hosts in the cluster. After entering the UI through another host, the problem was back. The fix was as I described above. I recorded the fix in action, but cannot post links as a new user:

Источник

TASK ERROR: Failed to run vncproxy.

mhayhurst

Active Member

I have two Proxmox machines in a cluster (Promox1 and Proxmox2) both running Proxmox 5.3-5. If I log into Proxmox1’s web UI and select any VM console in Proxmox2 then I receive this error:

The same thing happens if I go to Proxmox2’s web UI and select any VM console in Proxmox1. This appears to only be related to the console as everything else is working correctly.

I’ve also tried restarting pveproxy on both Proxmox1 and Proxmox2 and I can SSH from Proxmox1 to Proxmox2 and vice versa.

Would anyone be able to tell me how to correct this issue?

mhayhurst

Active Member

Stoiko Ivanov

Proxmox Staff Member

can you ssh without any password as root from Proxmox1 to Proxmox2 (and the other direction)?
PVE relies on ssh-public-key auth for some of its operations (including the vncproxy).

In any case you can try running `pvecm updatecerts` and see if this helps.

Best regards,
Stoiko

Do you already have a Commercial Support Subscription? — If not, Buy now and read the documentation

mhayhurst

Active Member

can you ssh without any password as root from Proxmox1 to Proxmox2 (and the other direction)?
PVE relies on ssh-public-key auth for some of its operations (including the vncproxy).

In any case you can try running `pvecm updatecerts` and see if this helps.

on both Proxmox1 and Proxmox2 and restarted both of them but that did not fix the issue. I see this in: /var/log/pveproxy/access.log

Stoiko Ivanov

Proxmox Staff Member

The access.log line indicates that the websocket connection seems successful (HTTP code 101).

You could try running the ssh-command the invokes the vncproxy (on Proxmox1):
/usr/bin/ssh -e none -T -o BatchMode=yes 10.10.10.10 /usr/sbin/qm vncproxy $VMID

(replace 10.10.10.10 with Proxmox2 IP)

What’s the output you get?

Best regards,
Stoiko

Do you already have a Commercial Support Subscription? — If not, Buy now and read the documentation

mhayhurst

Active Member

The access.log line indicates that the websocket connection seems successful (HTTP code 101).

You could try running the ssh-command the invokes the vncproxy (on Proxmox1):
/usr/bin/ssh -e none -T -o BatchMode=yes 10.10.10.10 /usr/sbin/qm vncproxy $VMID

(replace 10.10.10.10 with Proxmox2 IP)

What’s the output you get?

I added some verbosity as it only showed:

The private key: id_rsa DOES exist in: /root/.ssh/

Stoiko Ivanov

Proxmox Staff Member

I guess something in the `sshd_config` on the servers might be the culprit — It seems there were some modifications from the shipped defaults
(password-authentication is disabled in your config, but enabled in the default config)

* check `/etc/ssh/sshd_config` for differences from the defaults
* try to let sshd log at level DEBUG (or DEBUG2) and see why it refuses the key
* try with more verbosity (`ssh -vvv` vs. `ssh -v`)

Best regards,
Stoiko

Do you already have a Commercial Support Subscription? — If not, Buy now and read the documentation

mhayhurst

Active Member

I guess something in the `sshd_config` on the servers might be the culprit — It seems there were some modifications from the shipped defaults
(password-authentication is disabled in your config, but enabled in the default config)

* check `/etc/ssh/sshd_config` for differences from the defaults
* try to let sshd log at level DEBUG (or DEBUG2) and see why it refuses the key
* try with more verbosity (`ssh -vvv` vs. `ssh -v`)

I apologize it’s taken so long to get back to this but I was able to dedicate some time yesterday to troubleshooting this but the issue still remains.

I decided to start fresh so I set both Proxmox1 and Proxmox2 to allow password login: PasswordAuthentication yes. I then deleted everything in:

/.ssh/, created new key pairs, copied the public keys to the appropriate box, made sure I was able to login using the key pairs and set: PasswordAuthentication no and rebooted. One thing I did notice is that after a reboot Proxmox created both:
id_rsa.pub and id_rsa key pairs in the

/.ssh directory. which I found a bit odd and not sure why Proxmox did that?

However, here is what I discovered:

Proxmox1 can SSH to Proxmox2 and vice versa using the new SSH keypairs

Источник

Proxmox failed to run vncproxy – Here is how we sort it out

by Keerthi PS | Dec 31, 2019

Are you getting the error failed to run vncproxy while using Proxmox? We can help you fix it.

Usually, this error occurs due to improper symlink configuration.

At Bobcares, we often get requests to fix errors in Proxmox, as a part of our Infrastructure Management Services.

Today, let’s have a deeper look into Proxmox VNC Proxy and see how our Support Engineers fix this error.

VNC Proxy in Proxmox

Proxmox VE is an open-source server virtualization environment. It allows easy managing of VMs, containers, highly available clusters, storage, and network. And we can manage all these using the integrated web interface or via the CLI.

Whereas VNC proxy is the VNC interface for Proxmox VE hosts in the cluster. Once the cluster is set up, it is quite easy to manage it via the VNC Proxy. It provides access to the cluster to anyone with a VNC viewer. That is VNC Proxy is both the VNC client and VNC server.

But what if it shows up error while accessing VM console of other VM? This is a recent error our customer got. The error message appeared as,

So, let’s see how our Support Engineers fix it.

How we fix the Proxmox error failed to run vncproxy?

Initially, we checked whether the VMs are accessible or not. Also, we were able to SSH from one VM to another.

Then we tried to invoke the vncproxy using the command,

And this resulted in permission denied error.

Sometimes, syntax error in the command to invoke vncproxy can cause the same error. But this was not the actual reason here.

On further checking, we found that the symlinks were not configured correctly.

So we corrected these symlinks as, * /root/.ssh/authorized_keys to /etc/pve/priv .

Additionally, we check the key-pair in /root/.ssh/id_rsa(.pub) . And we ensure that the public key in the pair is in /etc/pve/priv/authorized_keys . Thereby we ensure that the PVE synchronizes with the cluster.

[Need assistance in fixing Proxmox errors? – Our Experts are available 24/7.]

Conclusion

In short, Proxmox failed to run vncproxy error occur due to improper symlink configuration. Sometimes invoking the vncproxy can resolve this error. Today, we saw how our Support Engineers fix this error.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

Источник

Failed to run vncproxy

xsancho

New Member

Hello,
I have installed 2 times Proxmox on a pc and when I try to power on a VM says: Failed to run vncproxy

My pveversion-v is:

# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-6
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4

pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Источник

Are you getting the error failed to run vncproxy while using Proxmox? We can help you fix it.

Usually, this error occurs due to improper symlink configuration.

At Bobcares, we often get requests to fix errors in Proxmox, as a part of our Infrastructure Management Services.

Today, let’s have a deeper look into Proxmox VNC Proxy and see how our Support Engineers fix this error.

VNC Proxy in Proxmox

Proxmox VE is an open-source server virtualization environment. It allows easy managing of VMs, containers, highly available clusters, storage, and network. And we can manage all these using the integrated web interface or via the CLI.

Whereas VNC proxy is the VNC interface for Proxmox VE hosts in the cluster. Once the cluster is set up, it is quite easy to manage it via the VNC Proxy. It provides access to the cluster to anyone with a VNC viewer. That is VNC Proxy is both the VNC client and VNC server.

But what if it shows up error while accessing VM console of other VM? This is a recent error our customer got. The error message appeared as,

So, let’s see how our Support Engineers fix it.

How we fix the Proxmox error failed to run vncproxy?

Initially, we checked whether the VMs are accessible or not. Also, we were able to SSH from one VM to another.

Then we tried to invoke the vncproxy using the command,

/usr/bin/ssh -e none -T -o BatchMode=yes  /usr/sbin/qm vncproxy $VMID

And this resulted in permission denied error.

Sometimes, syntax error in the command to invoke vncproxy can cause the same error. But this was not the actual reason here.

On further checking, we found that the symlinks were not configured correctly.

So we corrected these symlinks as, * /root/.ssh/authorized_keys to /etc/pve/priv.

Additionally, we check the key-pair in /root/.ssh/id_rsa(.pub). And we ensure that the public key in the pair is in /etc/pve/priv/authorized_keys.  Thereby we ensure that the PVE synchronizes with the cluster.

[Need assistance in fixing Proxmox errors? – Our Experts are available 24/7.]

Conclusion

In short, Proxmox failed to run vncproxy error occur due to improper symlink configuration. Sometimes invoking the vncproxy can resolve this error. Today, we saw how our Support Engineers fix this error.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;

Hypervisors

02 November 2018

Hits: 21798

2018-11-09 08:48:23 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=PMX01' This email address is being protected from spambots. You need JavaScript enabled to view it. /bin/true
2018-11-09 08:48:23 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2018-11-09 08:48:23 @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
2018-11-09 08:48:23 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2018-11-09 08:48:23 IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
2018-11-09 08:48:23 Someone could be eavesdropping on you right now (man-in-the-middle attack)!
2018-11-09 08:48:23 It is also possible that a host key has just been changed.
2018-11-09 08:48:23 The fingerprint for the RSA key sent by the remote host is
2018-11-09 08:48:23 SHA256:AwjDV7HjOjWaRruzdf4453452223JIkugHk1I7HFcVLfG+lx+wOAg.
2018-11-09 08:48:23 Please contact your system administrator.
2018-11-09 08:48:23 Add correct host key in /root/.ssh/known_hosts to get rid of this message.
2018-11-09 08:48:23 Offending RSA key in /etc/ssh/ssh_known_hosts:11
2018-11-09 08:48:23 remove with:
2018-11-09 08:48:23 ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R pmx01
2018-11-09 08:48:23 RSA host key for pmx01 has changed and you have requested strict checking.
2018-11-09 08:48:23 Host key verification failed.
2018-11-09 08:48:23 ERROR: migration aborted (duration 00:00:00): Can't connect to destination address using public key
TASK ERROR: migration aborted

$ ssh <destination_IP>

X.X.X.X      <hostname>

$ ssh-copy-id <destination_hostname>

$ ssh <destination_hostname>

При использовании ssh-сервера вы можете столкнуться с одной из распространенных ошибок: «Host Key Verification Failed». Чтобы понять, почему возникает эта ошибка, давайте сначала разберемся, как ssh устанавливает соединение.

Когда вы пытаетесь подключиться к удаленному серверу, сервер просит вас подтвердить, пытаетесь ли вы установить соединение с правильным сервером.

Если вы наберете «да», клиент добавит открытый ключ хоста в файл «.ssh/known_hosts». После добавления ключа удаленного сервера в следующий раз, когда вы попытаетесь подключиться к тому же серверу, клиент сравнит ключи с ключами, хранящимися в файле «known_hosts».

Вы не получите никаких предупреждений, если ключ присутствует в файле «known_hosts». Сервер будет подключен сразу.

Почему возникает ошибка «Host Key Verification Failed»

Основная причина, вызывающая ошибку Host Key Verification Failed», заключается в том, что ключ удаленного хоста был изменен и больше не тот, который хранится в файле «known_hosts». Ключ обычно меняется, когда серверы перестраиваются, и вы получаете сообщение об ошибке, как показано ниже:

Как исправить ошибку «Host Key Verification Failed»

Чтобы исправить эту ошибку, нам нужно удалить неверный ключ из файла «known_hosts», находящегося в нашей системе в каталоге «.ssh». Ошибка дает вам IP-адрес удаленного сервера и номер строки, в которой хранится ключ в файле «known_hosts».

В приведенной выше ошибки, «/home/user/.ssh/known_hosts:7», то «: 7» является задеть номер строки. Ниже перечислены несколько подходов к исправлению этой ошибки:

Способ 1:

Первый способ исправить эту ошибку – использовать команду sed. Команда «sed» используется для изменения текстовых файлов для поиска, добавления или удаления чего-либо из файлов. Мы используем его для удаления хоста-нарушителя:

$ sed -i '7d' ~.ssh/known_hosts

Если «7» – это номер строки, показанный в приведенной выше ошибке, ваш номер строки может быть другим; убедитесь, что вы используете правильный номер строки. Команда удалит неправильную строку из файла «known_hosts» и решит проблему.

Способ 2:

Второй подход – открыть файл «known_hosts» в любом редакторе:

$ nano.ssh/known_hosts

И вручную удалите оскорбительную строку и сохраните файл.

Способ 3:

Третий метод – удаление сервера с помощью команды «ssh-keygen». Следуйте синтаксису, указанному ниже:

$ ssh-keygen -R [IP_ADDRESS]

Например, чтобы удалить ключ хоста «192.168.10.116», используйте:

$ ssh-keygen -R 192.168.10.116

Заключение

Ошибка проверки ключа хоста возникает, когда ключ удаленного сервера изменяется, а клиент не проверяет его по сохраненным ключам. Ключи сервера хранятся в файле «known_hosts» на стороне клиента, и после установления соединения клиент проверяет ключ, сравнивая его с ключами, хранящимися в файле «known_host», и в случае сбоя вы получаете “Host key verification failed”.

Чтобы исправить это, удалите хост-нарушитель из файла «known_hosts». В этой статье упоминаются три различных метода удаления вредоносного хоста, и любой метод может использоваться для устранения этой ошибки.

If you’ve ever tried to connect to a remote server using ssh, and received an error message that says “Host key verification failed,” you know how frustrating it can be. This article will show you three ways to fix the problem.

What is a Host Key in SSH?

A Host key is a unique identifier that is used to verify the identity of a remote host. When you connect to a remote host, the Host key is verified against a list of known Host keys. If there is a match, the connection will be allowed to proceed. If there is not a match, the connection will be denied.

The Host key is also used to generate a cryptographic signature for each connection. This signature is used to verify the integrity of the data that is transferred between the client and server.

Understanding error message Host key verification failed

If you receive the error message “Host key verification failed”, it means that the key stored for the host you’re trying to connect to has changed. It is often caused by connecting to a different server than the one you originally connected to (for example, your server has been rebuilt by a new one).

Whenever we connect to a server via SSH, that server’s public key is stored in our home directory. The file is called known_hosts. 

This file is local to the user account and contains the known keys for remote hosts. These are collected from the hosts when connecting for the first time.

As with those keys stored in the file, ~/.ssh/known_hosts, these keys are used to verify the identity of the remote host, thus protecting against impersonation or man-in-the-middle attacks.

When we reconnect to the same server, the SSH connection will verify the current public key matches the one we have saved in our known_hosts file. If there is a match, the connection will proceed. If the match fails, ssh will fail with an error message Host key verification failed happens.

Example of Host key verification failed

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is x. Please contact your system administrator.
Add correct host key in /home/ec2-user/.ssh/known_hosts to get rid of this message.

Offending RSA key in /home/ec2-user.ssh/known_hosts:222 RSA host key for www.howtouselinux.com has changed and you have requested strict checking.Host key verification failed.

Methods to fix problem of Host key verification failed

Host key verification failed error occurs when the server’s host key does not match the key that was expected. This can happen when the server’s key has been changed, or when the key has been compromised. 

Here are three ways to fix this Host key verification failed error.

• Manually edit the “~/.ssh/known_hosts” file and remove the old key for the host you’re trying to connect to. This will allow you to connect to the new server without any problems.
• Use the “ssh-keygen -R” command to remove the old key from your “~/.ssh/known_hosts” file. This will allow you to connect to the new server without any problems.
• Use the “-o StrictHostKeyChecking=no” option when connecting to the server. This will prevent ssh from checking the “~/.ssh/known_hosts” file, and will allow you to connect to the new server without any problems.

Remove old host key info from known_hosts file

The easiest way to fix the problem of Host key verification failed is removing the old host key info and reconnect the server.

We can fix this issue with the following steps.

• Locate our known_hosts file
• open in a general text editor with vi /home/user/.ssh/known_hosts
• search the old host name and press “ESC dd” to delete the line.
• save the changes by pressing “esc” and typing “:wq!”.
• reconnect the server

Remove old host key info with ssh-keygen command

We can also remove the old host key with ssh-keygen command.

Open up a terminal session, and type one of the following

• ssh-keygen -R hostname
• ssh-keygen -R ipaddress
• ssh-keygen -f “/home/ec2-user.ssh/known_hosts” -R “192.168.0.106”

Disable SSH stricthostkeychecking option

The stricthostkeychecking option in SSH is a security feature that verifies the host key information for each connection.

If there is a problem with the host key information, the connection will not be allowed to proceed. This option can be disabled, which will allow the connection to proceed even if there is a problem with the host key information.

• Open up a terminal window.
• Type in the following command: ssh -o StrictHostKeyChecking=no hostname

This command removes the old host key for the device in the known_hosts file and replaces old host key with the new host key.

Understanding SSH known_hosts File with Examples