Task error failed to run vncproxy proxmox - Исправление ошибок и поиск оптимальных решений проблем

Hello everyone!

I have two Proxmox machines in a cluster (Promox1 and Proxmox2) both running Proxmox 5.3-5. If I log into Proxmox1’s web UI and select any VM console in Proxmox2 then I receive this error:

Code:

Permission denied (publickey).

TASK ERROR: Failed to run vncproxy.

The same thing happens if I go to Proxmox2’s web UI and select any VM console in Proxmox1. This appears to only be related to the console as everything else is working correctly.

I’ve also tried restarting pveproxy on both Proxmox1 and Proxmox2 and I can SSH from Proxmox1 to Proxmox2 and vice versa.

Would anyone be able to tell me how to correct this issue?

Would anyone be able to help me out with this?

can you ssh without any password as root from Proxmox1 to Proxmox2 (and the other direction)?
PVE relies on ssh-public-key auth for some of its operations (including the vncproxy).

In any case you can try running `pvecm updatecerts` and see if this helps.

can you ssh without any password as root from Proxmox1 to Proxmox2 (and the other direction)?
PVE relies on ssh-public-key auth for some of its operations (including the vncproxy).

In any case you can try running `pvecm updatecerts` and see if this helps.

Yes, I can SSH from Proxmox1 —> Proxmox2 and vice-versa as root without using a password. I executed:

on both Proxmox1 and Proxmox2 and restarted both of them but that did not fix the issue. I see this in: /var/log/pveproxy/access.log

Code:

"GET /api2/json/nodes/proxmox2/qemu/109/vncwebsocket?port=5900&vncticket=PVEVNC%3A5C24...(random key characters) HTTP/1.1" 101 -

It does not appear there are any errors (at least in that log file) regarding the VNC connection.

The access.log line indicates that the websocket connection seems successful (HTTP code 101).

You could try running the ssh-command the invokes the vncproxy (on Proxmox1):
/usr/bin/ssh -e none -T -o BatchMode=yes 10.10.10.10 /usr/sbin/qm vncproxy $VMID

(replace 10.10.10.10 with Proxmox2 IP)

What’s the output you get?

The access.log line indicates that the websocket connection seems successful (HTTP code 101).

You could try running the ssh-command the invokes the vncproxy (on Proxmox1):
/usr/bin/ssh -e none -T -o BatchMode=yes 10.10.10.10 /usr/sbin/qm vncproxy $VMID

(replace 10.10.10.10 with Proxmox2 IP)

What’s the output you get?

I added some verbosity as it only showed:

Code:

root@proxmox1:~/.ssh# /usr/bin/ssh -e none -T -o BatchMode=yes 192.168.1.5 /usr/sbin/qm vncproxy 104
Permission denied (publickey).

Code:

root@proxmox1:~# /usr/bin/ssh -v -e none -T -o BatchMode=yes 192.168.1.5 /usr/sbin/qm vncproxy 104
OpenSSH_7.4p1 Debian-10+deb9u4, OpenSSL 1.0.2q  20 Nov 2018
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 192.168.1.5 [192.168.1.5] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u4
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.1.5:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:H9ESbHJ9QEJpzJOBhpVs5tJ9skK3LWsp3r1uuuMxNVA
debug1: Host '192.168.1.5' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

The private key: id_rsa DOES exist in: /root/.ssh/

Code:

root@proxmox1:~# ls -alh /root/.ssh/id_rsa
-rw------- 1 root root 1.7K May 22  2018 /root/.ssh/id_rsa

If I change the id_rsa private key filename to: id_dsa then I can view any VM’s console on Proxmox2 from Proxmox1’s web UI (as they are both in a cluster), so that fixed it! But why? A few questions:

It’s not accepting the id_rsa (even though the key works) so I’m assuming that’s a config setting?
If so, I wonder why this changed as I did not alter any config settings?
Why all the complaints of: debug1: key_load_public: No such file or directory is that an issue?

Last edited: Dec 28, 2018

I guess something in the `sshd_config` on the servers might be the culprit — It seems there were some modifications from the shipped defaults
(password-authentication is disabled in your config, but enabled in the default config)

* check `/etc/ssh/sshd_config` for differences from the defaults
* try to let sshd log at level DEBUG (or DEBUG2) and see why it refuses the key
* try with more verbosity (`ssh -vvv` vs. `ssh -v`)

I guess something in the `sshd_config` on the servers might be the culprit — It seems there were some modifications from the shipped defaults
(password-authentication is disabled in your config, but enabled in the default config)

* check `/etc/ssh/sshd_config` for differences from the defaults
* try to let sshd log at level DEBUG (or DEBUG2) and see why it refuses the key
* try with more verbosity (`ssh -vvv` vs. `ssh -v`)

I apologize it’s taken so long to get back to this but I was able to dedicate some time yesterday to troubleshooting this but the issue still remains.

I decided to start fresh so I set both Proxmox1 and Proxmox2 to allow password login: PasswordAuthentication yes. I then deleted everything in: ~/.ssh/, created new key pairs, copied the public keys to the appropriate box, made sure I was able to login using the key pairs and set: PasswordAuthentication no and rebooted. One thing I did notice is that after a reboot Proxmox created both:
id_rsa.pub and id_rsa key pairs in the ~/.ssh directory…which I found a bit odd and not sure why Proxmox did that?

However, here is what I discovered:

Proxmox1 can SSH to Proxmox2 and vice versa using the new SSH keypairs

Executing both:

Code:

/usr/bin/ssh -vvvv -e none -T -o BatchMode=yes proxmox2 /usr/sbin/qm vncproxy 104

Code:

/usr/bin/ssh -vvvv -e none -T -o BatchMode=yes 192.168.1.5 /usr/sbin/qm vncproxy 104

from Proxmox1 works:

Code:

debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
RFB 003.008

Executing both:

Code:

/usr/bin/ssh -vvvv -e none -T -o BatchMode=yes proxmox1 /usr/sbin/qm vncproxy 100

Code:

/usr/bin/ssh -vvvv -e none -T -o BatchMode=yes 192.168.1.4 /usr/sbin/qm vncproxy 100

from Proxmox2 works:

Code:

debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
RFB 003.008

I should mention, I remember these complaining about the /etc/ssh/ssh_known_hosts at first so I removed that file thinking Proxmox would recreate it but it has not. This is also odd as I initially removed this file and it was recreated after some time…not sure what’s going on there?

Logging into Proxmox1’s Web UI I’m able to successfully connect to Proxmox2’s console but none of the console’s for any of Proxmox2’s VM’s

Code:

Host key verification failed.
TASK ERROR: Failed to run vncproxy.

Logging into Proxmox2’s Web UI I’m able to successfully connect to Proxmox1’s console but none of the console’s for any of Proxmox1’s VM’s

Code:

Host key verification failed.
TASK ERROR: Failed to run vncproxy.

This is extremely confusing at this point as I feel Proxmox has built in layers of complication to something that should be simple and straight forward. It appears the CLI SSH, CLI VNProxy SSH and Web UI VNC SSH are all doing something different. Do you or does anyone know what the Web UI is doing differently to connect via VNC to a VM’s console?

This is extremely confusing at this point as I feel Proxmox has built in layers of complication to something that should be simple and straight forward. It appears the CLI SSH, CLI VNProxy SSH and Web UI VNC SSH are all doing something different. Do you or does anyone know what the Web UI is doing differently to connect via VNC to a VM’s console?

The layers of complication PVE adds are the price for the usually quite comfortable cluster you get:
* /root/.ssh/authorized_keys is symlinked into `/etc/pve/priv` so that they are synchronized across the cluster — please don’t change it or make them regular files, else debugging is far harder and issues like the current happen
* the key-pair /root/.ssh/id_rsa(.pub) is the one PVE uses for communicating within the cluster — please make sure that the public-key of that pair is in /etc/pve/priv/authorized_keys (and that this in turn is where /root/.ssh/authorized_keys points
* try logging with user root and the id_rsa private key from/to both nodes

The WebUI should do the same as the qm cli.
please also post the /etc/ssh/sshd_config from both servers

The layers of complication PVE adds are the price for the usually quite comfortable cluster you get:
* /root/.ssh/authorized_keys is symlinked into `/etc/pve/priv` so that they are synchronized across the cluster — please don’t change it or make them regular files, else debugging is far harder and issues like the current happen
* the key-pair /root/.ssh/id_rsa(.pub) is the one PVE uses for communicating within the cluster — please make sure that the public-key of that pair is in /etc/pve/priv/authorized_keys (and that this in turn is where /root/.ssh/authorized_keys points
* try logging with user root and the id_rsa private key from/to both nodes

The WebUI should do the same as the qm cli.
please also post the /etc/ssh/sshd_config from both servers

Thank you for your reply! Looks like those files are not symlinked:

Code:

root@proxmox1:/etc/pve/priv# ls -alh
total 2.5K
drwx------ 2 root www-data    0 Dec 19  2017 .
drwxr-xr-x 2 root www-data    0 Dec 31  1969 ..
-rw------- 1 root www-data 1.7K Dec 19  2017 authkey.key
-rw------- 1 root www-data 1.6K Jan 11 18:37 authorized_keys
-rw------- 1 root www-data 1.6K Jan 11 18:37 known_hosts
drwx------ 2 root www-data    0 Dec 20  2017 lock
-rw------- 1 root www-data 3.2K Dec 19  2017 pve-root-ca.key
-rw------- 1 root www-data    3 Dec 27 08:57 pve-root-ca.srl

I will correct that and try again but had a couple of questions:

1. What is the authkey.key and does it need symlinked?

2. Is what I deleted: /etc/ssh/ssh_known_hosts the symlink of: /etc/pve/priv/known_hosts? If so, should I delete:
/etc/pve/priv/known_hosts as well and let Proxmox recreate it then symlink it?

3. Are the pve-root-ca.key and pve-root-ca.srl the certificates for the Web UI?

4. You said the WebUI should do the same as the qm cli but they appear to do different things as I’m getting different results (you can see that from my previous post). You would think the qm cli would fail since the Web UI fails and I’m missing symlinks but it does not. Am I overlooking something?

Also, here are my /etc/ssh/sshd_config from both Proxmox1 and Proxmox2

Proxmox1:

Code:

#    $OpenBSD: sshd_config,v 1.100 2016/08/15 12:32:04 naddy Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
AuthorizedKeysFile    .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem    sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#    X11Forwarding no
#    AllowTcpForwarding no
#    PermitTTY no
#    ForceCommand cvs server

Proxmox2:

Code:

#    $OpenBSD: sshd_config,v 1.100 2016/08/15 12:32:04 naddy Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
AuthorizedKeysFile    .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem    sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#    X11Forwarding no
#    AllowTcpForwarding no
#    PermitTTY no
#    ForceCommand cvs server

Thank you for your reply! Looks like those files are not symlinked:

it’s the other way around — /root/.ssh/authorized_keys is a symlink to /etc/pve/priv/authorized_keys , i.e. /root/.ssh/autorized_keys -> /etc/pve/priv/authorized_keys.
* pve-root-ca.key, pve-root-ca.srl — are for the CA, which PVE sets up for the certificates for the nodes and their communication
* authkey.key — is for the webGUI — auth-tickets
* /etc/ssh/ssh_known_hosts is a symlink to /etc/pve/priv/known_hosts — just recreate that -and make sure all your cluster-nodes host-keys are in that file

* the ssh config looks ok.

does logging in as root (with id_rsa) work ?

EDIT: rephrased the symlink explanation

Last edited: Feb 21, 2019

it’s the other way around — /root/.ssh/authorized_keys is a symlink to /etc/pve/priv/authorized_keys , i.e. /root/.ssh/autorized_keys -> /etc/pve/priv/authorized_keys.
* pve-root-ca.key, pve-root-ca.srl — are for the CA, which PVE sets up for the certificates for the nodes and their communication
* authkey.key — is for the webGUI — auth-tickets
* /etc/ssh/ssh_known_hosts is a symlink to /etc/pve/priv/known_hosts — just recreate that -and make sure all your cluster-nodes host-keys are in that file

* the ssh config looks ok.

does logging in as root (with id_rsa) work ?

EDIT: rephrased the symlink explanation

Hello again Stoiko,

Believe it or not, starting with a clean slate again and by this I mean:

Code:

rm ~/.ssh/*
rm /etc/ssh/
rm /etc/ssh/ssh_known_hosts
rm /etc/pve/priv/authorized_keys
rm /etc/pve/priv/known_hosts

and then executing

created new SSH keypairs as well as symlinked the appropriate files on both Proxmox1 and Proxmox2

Code:

root@proxmox1:~# ls -alh ~/.ssh/
total 45K
drwxr-xr-x 2 root root    7 Feb 21 09:39 .
drwx------ 6 root root   13 Feb 21 11:19 ..
lrwxrwxrwx 1 root root   29 Feb 21 09:39 authorized_keys -> /etc/pve/priv/authorized_keys
-rw-r----- 1 root root  117 Feb 21 09:39 config
-rw------- 1 root root 1.7K Feb 21 09:39 id_rsa
-rw-r--r-- 1 root root  395 Feb 21 09:39 id_rsa.pub

Code:

root@proxmox1:~# ls -alh /etc/ssh/ssh_known_hosts
lrwxrwxrwx 1 root root 25 Feb 21 09:39 /etc/ssh/ssh_known_hosts -> /etc/pve/priv/known_hosts

Code:

root@proxmox1:~# ls -alh /etc/pve/priv/
total 2.5K
drwx------ 2 root www-data    0 Dec 19  2017 .
drwxr-xr-x 2 root www-data    0 Dec 31  1969 ..
-rw------- 1 root www-data 1.7K Dec 19  2017 authkey.key
-rw------- 1 root www-data  791 Feb 21 09:39 authorized_keys
-rw------- 1 root www-data 1.6K Feb 21 09:39 known_hosts
drwx------ 2 root www-data    0 Dec 20  2017 lock
-rw------- 1 root www-data 3.2K Dec 19  2017 pve-root-ca.key
-rw------- 1 root www-data    3 Dec 27 08:57 pve-root-ca.srl

However, I get this message when I SSH from Proxmox1 to Proxmox2 and vice versa?

Code:

root@proxmox1:~# ssh proxmox2
Warning: the ECDSA host key for 'proxmox2.jam.lan' differs from the key for the IP address '192.168.1.5'
Offending key for IP in /etc/ssh/ssh_known_hosts:4
Matching host key in /root/.ssh/known_hosts:1
Are you sure you want to continue connecting (yes/no)? yes

Not sure why ssh_known_hosts has entries for the hostnames as well as the IP addresses for both Proxmox1 and Proxmox2? The only thing I can think of is it has something to do with how I set the cluster up. I remember Proxmox’s guide stating that IP addresses had to be used instead of hostnames for clusters.

Code:

root@proxmox1:~# cat /etc/ssh/ssh_known_hosts
proxmox1 ssh-rsa AAAAB3NzaC1yc2EA..........
192.168.1.4 ssh-rsa AAAAB3NzaC1.........
proxmox2 ssh-rsa AAAAB3NzaC1yc2EAA........
192.168.1.5 ssh-rsa AAAAB3NzaC1yc2EAA.......

Not sure why ssh_known_hosts has entries for the hostnames as well as the IP addresses for both Proxmox1 and Proxmox2? The only

ssh always saves the host-key for both the ip-address and the DNS-name you connect to (to prevent someone hijacking your DNS-name and presenting you with the wrong box where you enter your password)

Warning: the ECDSA host key for ‘proxmox2.jam.lan’ differs from the key for the IP address ‘192.168.1.5’ Offending key for IP in /etc/ssh/ssh_known_hosts:4 Matching host key in /root/.ssh/known_hosts:1

this is probably the root cause — delete the IP<->key matching and reconnect.
any question of that sort will make vnproxy fail.

Hope this helps!

ssh always saves the host-key for both the ip-address and the DNS-name you connect to (to prevent someone hijacking your DNS-name and presenting you with the wrong box where you enter your password)

this is probably the root cause — delete the IP<->key matching and reconnect.
any question of that sort will make vnproxy fail.

Hope this helps!

Yes, you’ve been a great help…thank you!

I believe that’s initially what put me in this dilemma. I understand I could delete the offending key but I never put that key in there, it appears executing: pvecm updatecerts and a combination of SSH or logging into the Web UI did all of that. So is this something in Proxmox that needs corrected?

Hello forum,

I have a similar problem: TASK ERROR: Failed to run vncproxy.

BUT I also get this error when trying to start a console to a VM on the very proxmox server I am connected to. So I get the error when trying to start a console to ANY VM in my cluster.

Proxmox 5.3-11
4 server cluster with Ceph

Has been running smoothly BEFORE updating from 5.2 to 5.3

Would anybody be able to give a hint what the problem could be?

Greetings,

hans

P.S. ssh works from any server to any other server in the cluster without problem. Only thing I noticed is that host names are NOT resolved. I have to ssh to the IP-address. Is that normal?

This seems like an unrelated problem!
pve needs to be able to resolve the hostnames of the cluster-nodes to ip-addresses and expects that it can:
the first thing would be to add all nodes to ‘/etc/hosts’ on all nodes and try again!

Hello. I too am running into the original problem posted in this thread, specifically I’m getting

Code:

Host key verification failed.

when testing using the vncproxy command given above, yet I can ssh between the nodes using their host names and get details of the remote node and the VMs on it via the Web UI just fine.
I have a dual-stack (IPv4 and IPv6) network and I think the problem may be that the systems prefer IPv6 and there are no entries in /etc/pve/priv/known_hosts for the IPv6 addresses of the nodes. Indeed, if I put the IPv4 address instead of the host name into the vncproxy command, it works correctly.
So is there an automated way for pvecm or another tool to add IPv6 host keys to the shared file or must I add them manually?

Also, /root/.ssh/known_hosts has ecdsa-sha2-nistp256 keys while /etc/pve/priv/known_hosts has only ssh-rsa ones, if that matters.

Last edited: Apr 10, 2019

I just tried fixing this again and now even if I use an IPv6 address in the vncproxy command,

Code:

/usr/bin/ssh -e none -T -o BatchMode=yes IP:V6:address:of:other:node /usr/sbin/qm vncproxy $VMID

the connection from the command line succeeds. (I see RFB 003.008.) I also tried the command using the host name and the FQDN of the other node. Both also succeed (though I had to allow the key in the latter case.) So why the heck is the Web console connection failing?! What else is happening?? Is there a way to get the Web interface to show precisely which user and command it’s trying to execute and on which host?

Last edited: May 15, 2019

Thanks to some folks in the IRC channel, I finally fixed this (at least partially: the remote Web console works again) by doing the following on the node whose Web page I’m connected to:

Code:

ssh -o HostKeyAlias=IPv4.address.of.other.node IPv4.address.of.other.node
ssh -o HostKeyAlias=IPv6.address.of.other.node IPv6.address.of.other.node
ssh -o HostKeyAlias=HostnameOfOtherNode HostnameOfOtherNode

For each one, delete the entry in /root/.ssh/known_hosts it complains about

Источник

Are you getting the error failed to run vncproxy while using Proxmox? We can help you fix it.

Usually, this error occurs due to improper symlink configuration.

At Bobcares, we often get requests to fix errors in Proxmox, as a part of our Infrastructure Management Services.

Today, let’s have a deeper look into Proxmox VNC Proxy and see how our Support Engineers fix this error.

VNC Proxy in Proxmox

Proxmox VE is an open-source server virtualization environment. It allows easy managing of VMs, containers, highly available clusters, storage, and network. And we can manage all these using the integrated web interface or via the CLI.

Whereas VNC proxy is the VNC interface for Proxmox VE hosts in the cluster. Once the cluster is set up, it is quite easy to manage it via the VNC Proxy. It provides access to the cluster to anyone with a VNC viewer. That is VNC Proxy is both the VNC client and VNC server.

But what if it shows up error while accessing VM console of other VM? This is a recent error our customer got. The error message appeared as,

So, let’s see how our Support Engineers fix it.

How we fix the Proxmox error failed to run vncproxy?

Initially, we checked whether the VMs are accessible or not. Also, we were able to SSH from one VM to another.

Then we tried to invoke the vncproxy using the command,

/usr/bin/ssh -e none -T -o BatchMode=yes  /usr/sbin/qm vncproxy $VMID

And this resulted in permission denied error.

Sometimes, syntax error in the command to invoke vncproxy can cause the same error. But this was not the actual reason here.

On further checking, we found that the symlinks were not configured correctly.

So we corrected these symlinks as, * /root/.ssh/authorized_keys to /etc/pve/priv.

Additionally, we check the key-pair in /root/.ssh/id_rsa(.pub). And we ensure that the public key in the pair is in /etc/pve/priv/authorized_keys. Thereby we ensure that the PVE synchronizes with the cluster.

[Need assistance in fixing Proxmox errors? – Our Experts are available 24/7.]

Conclusion

In short, Proxmox failed to run vncproxy error occur due to improper symlink configuration. Sometimes invoking the vncproxy can resolve this error. Today, we saw how our Support Engineers fix this error.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;

Источник

Содержание

[SOLVED] Failed to run vncproxy in all VMs / Windows VMs no longer booting
audioPhil
[SOLVED] Failed to run vncproxy
Tamilarasan
DanielWood
TASK ERROR: Failed to run vncproxy.
mhayhurst
mhayhurst
Stoiko Ivanov
mhayhurst
Stoiko Ivanov
mhayhurst
Stoiko Ivanov
mhayhurst
Proxmox failed to run vncproxy – Here is how we sort it out
VNC Proxy in Proxmox
How we fix the Proxmox error failed to run vncproxy?
Conclusion
PREVENT YOUR SERVER FROM CRASHING!
Failed to run vncproxy
xsancho

[SOLVED] Failed to run vncproxy in all VMs / Windows VMs no longer booting

audioPhil

New Member

we are running Proxmox on a single server (no Cluster) and VMs are stored locally on an LVM storage.
Currently the system is running

10-12 VMs. Most of them are Linux server systems, but we are also running two Windows machines.
During everyday use, all VMs were running fine and no issues occurred (SSH, HTTP, SMB, RDP were all behaving as expected).

Today, my colleague tried to boot up a Windows VM which had been offline for some weeks. He realized he could not get an RDP connection and so he tried to access the machine via the Proxmox noVNC console.
However, this was not working either. The Console tab shows «Verbinden. » (I suspect the English translation would be «Connecting. «) and after a while the following error appears:

This is when I got involved into this issue (I am sort of the main administrator of the Proxmox instance).

So far I found out / tried the following:
— For almost all of the VMs the noVNC console is not working.
— For some non-critical systems I tried to initiate a shutdown via the Web GUI, as well as via qm shutdown . The systems with a non-working noVNC console cannot be shutdown this way. The systems shows VM quit/powerdown failed .
— I logged into some of the Linux VMs via SSH and initiated a shutdown. I then restarted these VMs via the Web GUI and they are now working as expected (noVNC is showing, I can now initiate a shutdown too).
— I tried the same for one of the (up to that point still working) Windows VMs. It does not come back online, is neither accessible via RDP nor noVNC. While a start of the Windows VM seems successful via the Web GUI, starting it via qm start shows

— The problem also occurs if I try to create a new Windows VM from scratch. Not even the initial boot into the installer ISO file is working.
— The problem does not occur if I try to create a new Linux VM from scratch. The installer loads perfectly as expected.
— I installed the latest packages ( apt update && apt upgrade ) but nothing changed.
— I did not yet restart the hypervisor itself. One of the Windows VMs is still running and I would prefer not to lose access to it too, as it is in daily use.
— I searched for the issue on the Proxmox forum and found this thread which describes a similar behaviour regarding the «Failed to run vncproxy» part. However, the issue described there has been resolved with an upgrade to pve-qemu-kvm 4.0.0-7 . We are running pve-qemu-kvm/stable 5.0.0-11 amd64

What could have led to this error?
A week ago I updated the packages on the Proxmox instance. After that, I did not reboot the hypervisor. As we are not using the noVNC feature on a daily base it could very well be that one of the updates led to the problem and has remained undetected until today.
However, I have one Linux VM with an uptime of 6 days (rebooted right after the update of Proxmox) which is working perfectly, while another Linux VM with an uptime of 4 days (rebooted multiple days after the update) is not working.

Likely not related, but something I changed recently:
I introduced named admin accounts for my colleagues and me and deactivated the default root account. This had some unexpected side effects such as updates no longer being possible via the Web GUI. Therefore, I updated via the CLI using apt . To rule this out as a possible reason I reactivated the root account today. The issues persist.

Information on our system:

I am out of ideas and hope that someone might be able to help. If required, I could organize a reboot of the hypervisor relatively quickly. A short downtime of the VMs is no problem but I am afraid that we will then be without any working Windows VM.

Источник

[SOLVED] Failed to run vncproxy

New Member

I have a two node setup. From the first node, when I pull up the console for a VM on the second node, I get this:

Debian GNU/Linux 9
Permission denied (publickey).
TASK ERROR: Failed to run vncproxy.

I can’t tell if this is an SSH issue or a vncproxy issue. I have verified that the authorized_keys are set properly to match the ssh keys on each host.

Any idea on troubleshooting this? Proxmox version 5.1-41.

New Member

Tamilarasan

New Member

how did you solve the issue? could pls provide the steps?

DanielWood

New Member

how did you solve the issue? could pls provide the steps?

I came across a similar issue.

The hint of root ssh access by the original poster caused me to stumble upon the solution.

My problem was when I tried to access the console from the WebUI for one host for a VM on another host. The fix for me was to open the web ssh console (Shell) on the other hosts from the WebUI of the primary host. When I did, I was greeted with a prompt to accept the SSH fingerprint. I suspect this was the problem to begin with. As soon as I accepted the fingerprint, I was able to see the console on other hosts.

In fact, I was able to validate it since I still had more hosts in the cluster. After entering the UI through another host, the problem was back. The fix was as I described above. I recorded the fix in action, but cannot post links as a new user:

Источник

TASK ERROR: Failed to run vncproxy.

mhayhurst

Active Member

I have two Proxmox machines in a cluster (Promox1 and Proxmox2) both running Proxmox 5.3-5. If I log into Proxmox1’s web UI and select any VM console in Proxmox2 then I receive this error:

The same thing happens if I go to Proxmox2’s web UI and select any VM console in Proxmox1. This appears to only be related to the console as everything else is working correctly.

I’ve also tried restarting pveproxy on both Proxmox1 and Proxmox2 and I can SSH from Proxmox1 to Proxmox2 and vice versa.

Would anyone be able to tell me how to correct this issue?

mhayhurst

Active Member

Stoiko Ivanov

Proxmox Staff Member

can you ssh without any password as root from Proxmox1 to Proxmox2 (and the other direction)?
PVE relies on ssh-public-key auth for some of its operations (including the vncproxy).

In any case you can try running `pvecm updatecerts` and see if this helps.

Best regards,
Stoiko

Do you already have a Commercial Support Subscription? — If not, Buy now and read the documentation

mhayhurst

Active Member

can you ssh without any password as root from Proxmox1 to Proxmox2 (and the other direction)?
PVE relies on ssh-public-key auth for some of its operations (including the vncproxy).

In any case you can try running `pvecm updatecerts` and see if this helps.

on both Proxmox1 and Proxmox2 and restarted both of them but that did not fix the issue. I see this in: /var/log/pveproxy/access.log

Stoiko Ivanov

Proxmox Staff Member

The access.log line indicates that the websocket connection seems successful (HTTP code 101).

You could try running the ssh-command the invokes the vncproxy (on Proxmox1):
/usr/bin/ssh -e none -T -o BatchMode=yes 10.10.10.10 /usr/sbin/qm vncproxy $VMID

(replace 10.10.10.10 with Proxmox2 IP)

What’s the output you get?

Best regards,
Stoiko

Do you already have a Commercial Support Subscription? — If not, Buy now and read the documentation

mhayhurst

Active Member

The access.log line indicates that the websocket connection seems successful (HTTP code 101).

You could try running the ssh-command the invokes the vncproxy (on Proxmox1):
/usr/bin/ssh -e none -T -o BatchMode=yes 10.10.10.10 /usr/sbin/qm vncproxy $VMID

(replace 10.10.10.10 with Proxmox2 IP)

What’s the output you get?

I added some verbosity as it only showed:

The private key: id_rsa DOES exist in: /root/.ssh/

Stoiko Ivanov

Proxmox Staff Member

* check `/etc/ssh/sshd_config` for differences from the defaults
* try to let sshd log at level DEBUG (or DEBUG2) and see why it refuses the key
* try with more verbosity (`ssh -vvv` vs. `ssh -v`)

Best regards,
Stoiko

Do you already have a Commercial Support Subscription? — If not, Buy now and read the documentation

mhayhurst

Active Member

* check `/etc/ssh/sshd_config` for differences from the defaults
* try to let sshd log at level DEBUG (or DEBUG2) and see why it refuses the key
* try with more verbosity (`ssh -vvv` vs. `ssh -v`)

I apologize it’s taken so long to get back to this but I was able to dedicate some time yesterday to troubleshooting this but the issue still remains.

I decided to start fresh so I set both Proxmox1 and Proxmox2 to allow password login: PasswordAuthentication yes. I then deleted everything in:

/.ssh/, created new key pairs, copied the public keys to the appropriate box, made sure I was able to login using the key pairs and set: PasswordAuthentication no and rebooted. One thing I did notice is that after a reboot Proxmox created both:
id_rsa.pub and id_rsa key pairs in the

/.ssh directory. which I found a bit odd and not sure why Proxmox did that?

However, here is what I discovered:

Proxmox1 can SSH to Proxmox2 and vice versa using the new SSH keypairs

Источник

Proxmox failed to run vncproxy – Here is how we sort it out

by Keerthi PS | Dec 31, 2019

Are you getting the error failed to run vncproxy while using Proxmox? We can help you fix it.

Usually, this error occurs due to improper symlink configuration.

At Bobcares, we often get requests to fix errors in Proxmox, as a part of our Infrastructure Management Services.

Today, let’s have a deeper look into Proxmox VNC Proxy and see how our Support Engineers fix this error.

VNC Proxy in Proxmox

But what if it shows up error while accessing VM console of other VM? This is a recent error our customer got. The error message appeared as,

So, let’s see how our Support Engineers fix it.

How we fix the Proxmox error failed to run vncproxy?

Initially, we checked whether the VMs are accessible or not. Also, we were able to SSH from one VM to another.

Then we tried to invoke the vncproxy using the command,

And this resulted in permission denied error.

Sometimes, syntax error in the command to invoke vncproxy can cause the same error. But this was not the actual reason here.

On further checking, we found that the symlinks were not configured correctly.

So we corrected these symlinks as, * /root/.ssh/authorized_keys to /etc/pve/priv .

Additionally, we check the key-pair in /root/.ssh/id_rsa(.pub) . And we ensure that the public key in the pair is in /etc/pve/priv/authorized_keys . Thereby we ensure that the PVE synchronizes with the cluster.

[Need assistance in fixing Proxmox errors? – Our Experts are available 24/7.]

Conclusion

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

Источник

Failed to run vncproxy

xsancho

New Member

Hello,
I have installed 2 times Proxmox on a pc and when I try to power on a VM says: Failed to run vncproxy

My pveversion-v is:

# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-6
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4

pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Источник

Having issues, I have following the post from Brnson, but I think I am missing something.

here are the steps I followed

On 7/30/2019 at 12:51 PM, Benson said:

Step almost the same, but I think best should be prepare unraid USB stick first then unplug it before prepare Proxmox.

Below are some detail

1. Unplug well prepare Unraid USB, best ( not must ) confirm it can boot in UEFI mode.

2. Install Proxmox to harddisk, make IOMMU enable.

3. Replug Unraid USB

4. Create VM

4a. Tweak ( Some step may not necessary )

— Click CD/DVD drive, remove it

— Click Hard Disk, detach and remove it

— Add phyical disk controller and physical Unraid USB stick

4b. Tweak

— Edit boot order, only retain CD-ROM in boot device_1

That’s all.

so the steps I have takes. I have created a Test UNRAID USB pen drive. — all works with no issues with a 4TB Hard Drive for storage.

I then installed Proxmox onto a 1TB HD.

so far so good.

then I followed Bensons steps and here is where the fun starts. I will post each step. in the hope someone will see my mistake.

First I check the I have IOMMU Groups

then I created the VM

I did’t choose of OS as I was thinking that as I am passthrough my Unraid Pen drive, I will not need to set this up

working my way through

and it build OK then I when through and remvoe the CDROM and Added the Unraid Pen Drive.

when adding the PCI Controllor I am getting the following — No Ideal How to resovlve this at the moment

then I confirm my boot order

but when I start my VM, then I am getting this

TASK ERROR: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.

and here is my Hard Spec

Источник