Permission denied (publickey) after upgrade to 3.6.2

Hi,

I have just upgraded to Core Tunnel 3.6.2 today and also enabled Core Helper.

I start to get "Permission denied (publickey)" error when I tried starting the tunnel.

I have also followed the trouble shooting topic to add +ssh-rsa option. But it still doesn't work.

Could you help me to take a quick look? Any help would be much appreciated! Thanks!

log attached:
log.txt (11.7 KB)

Hi Yiliang,

Could you please run this command in local shell and send me a copy of output?

ssh -N -vvv -L 8080:localhost:8080 -L 2044:localhost:1044 -L 2009:localhost:2009 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -p 22 yilianx@dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxx.xxx

Kindly Regards,

Yang

Hi Yang,

Thank you for you quick response. Here's the output of running the command and seems like the tunnel has established successfully thru the shell:

❯ ssh -N -vvv -L 8080:localhost:8080 -L 2044:localhost:1044 -L 2009:localhost:2009 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -p 22 yilianx@dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxxx.com
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/yilianx/.ssh/config
debug1: /Users/yilianx/.ssh/config line 19: Applying options for dev-dsk-*.xxxxxxx.com
debug1: /Users/yilianx/.ssh/config line 56: Applying options for *.us-west-2.xxxxxxx.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxxx.com port 22.
debug1: Connection established.
debug1: identity file /Users/yilianx/.ssh/id_rsa type 0
debug1: identity file /Users/yilianx/.ssh/id_rsa-cert type 4
debug1: identity file /Users/yilianx/.ssh/id_dsa type -1
debug1: identity file /Users/yilianx/.ssh/id_dsa-cert type -1
debug1: identity file /Users/yilianx/.ssh/id_ecdsa type -1
debug1: identity file /Users/yilianx/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/yilianx/.ssh/id_ed25519 type -1
debug1: identity file /Users/yilianx/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/yilianx/.ssh/id_xmss type -1
debug1: identity file /Users/yilianx/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1-RHEL7-7.4p1-21 mdy1.0
debug1: match: OpenSSH_7.4p1-RHEL7-7.4p1-21 mdy1.0 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxxx.com:22 as 'yilianx'
debug3: hostkeys_foreach: reading file "/Users/yilianx/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/yilianx/.ssh/known_hosts:44
debug3: load_hostkeys: loaded 1 keys from dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxxx.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:4RVV+aDDtKaD6z9TbH6fNz2XLZalKdYh216GhnKzVu8
debug3: hostkeys_foreach: reading file "/Users/yilianx/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/yilianx/.ssh/known_hosts:44
debug3: load_hostkeys: loaded 1 keys from dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxxx.com
debug3: hostkeys_foreach: reading file "/Users/yilianx/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/yilianx/.ssh/known_hosts:44
debug3: load_hostkeys: loaded 1 keys from 172.16.42.162
debug1: Host 'dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxxx.com' is known and matches the RSA host key.
debug1: Found key in /Users/yilianx/.ssh/known_hosts:44
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/yilianx/.ssh/id_rsa RSA SHA256:K7vaPppxUev1qkr1MqlM49qwS2y6J+s/WblqnnlP/ME
debug1: Will attempt key: /Users/yilianx/.ssh/id_rsa RSA-CERT SHA256:K7vaPppxUev1qkr1MqlM49qwS2y6J+s/WblqnnlP/ME
debug1: Will attempt key: /Users/yilianx/.ssh/id_dsa
debug1: Will attempt key: /Users/yilianx/.ssh/id_ecdsa
debug1: Will attempt key: /Users/yilianx/.ssh/id_ed25519
debug1: Will attempt key: /Users/yilianx/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/yilianx/.ssh/id_rsa RSA SHA256:K7vaPppxUev1qkr1MqlM49qwS2y6J+s/WblqnnlP/ME
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Offering public key: /Users/yilianx/.ssh/id_rsa RSA-CERT SHA256:K7vaPppxUev1qkr1MqlM49qwS2y6J+s/WblqnnlP/ME
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/yilianx/.ssh/id_rsa RSA-CERT SHA256:K7vaPppxUev1qkr1MqlM49qwS2y6J+s/WblqnnlP/ME
debug3: sign_and_send_pubkey: RSA-CERT SHA256:K7vaPppxUev1qkr1MqlM49qwS2y6J+s/WblqnnlP/ME
debug2: sign_and_send_pubkey: using private key "/Users/yilianx/.ssh/id_rsa" for certificate
debug3: sign_and_send_pubkey: signing using ssh-rsa-cert-v01@openssh.com
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxxx.com ([172.16.42.162]:22).
debug1: Local connections to LOCALHOST:8080 forwarded to remote address localhost:8080
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug1: Local forwarding listening on 127.0.0.1 port 8080.
debug2: fd 7 setting O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 8080.
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug1: Local connections to LOCALHOST:2044 forwarded to remote address localhost:1044
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug1: Local forwarding listening on 127.0.0.1 port 2044.
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 2: new [port listener]
debug3: sock_set_v6only: set socket 10 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 2044.
debug2: fd 10 setting O_NONBLOCK
debug3: fd 10 is O_NONBLOCK
debug1: channel 3: new [port listener]
debug1: Local connections to LOCALHOST:2009 forwarded to remote address localhost:2009
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug1: Local forwarding listening on 127.0.0.1 port 2009.
debug2: fd 11 setting O_NONBLOCK
debug3: fd 11 is O_NONBLOCK
debug1: channel 4: new [port listener]
debug3: sock_set_v6only: set socket 12 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 2009.
debug2: fd 12 setting O_NONBLOCK
debug3: fd 12 is O_NONBLOCK
debug1: channel 5: new [port listener]
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

Thank you very much for the output, pease try set PubkeyAcceptedKeyTypes and HostbasedKeyTypes to

+ssh-rsa-cert-v01@openssh.com,+ssh-rsa

And please send me the connection log if you still could not get it to work.

Kindly Regards,

Yang

Hi, I would assume you mean PubkeyAcceptedAlgorithms and HostbasedAcceptedAlgorithms?

Since you mentioned it in the "PubkeyAcceptedKeyTypes and HostbasedKeyTypes directives are disappeared" section of the trouble shooting topic.

I tried setting these two options as below:
image

But I am getting new error saying the option is invalid:

Equivalent Command: ssh -N -vvv -L 8080:localhost:8080 -L 2044:localhost:1044 -L 2009:localhost:2009 -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o HostbasedAcceptedAlgorithms=+ssh-rsa-cert-v01@openssh.com,+ssh-rsa -o ServerAliveCountMax=3 -o PubkeyAcceptedAlgorithms=+ssh-rsa-cert-v01@openssh.com,+ssh-rsa -p 22 yilianx@dev-dsk-yilianx-2b-7205eb1c.us-west-2.xxxxxx.com
05:16:32 Connecting…
05:16:32 Using Core Helper 6.6 (r3386)
05:16:32 command-line line 0: Bad key types '+ssh-rsa-cert-v01@openssh.com,+ssh-rsa'.
05:16:32 Abnormal Disconnect
05:16:32 Connection failed, retry after 3s…

nit: Also it might be worth-mentioning that the app crashed when I was trying to save the two options. But the options are preserved after re-open.

Yes, I'm sorry about the misleading.

Please try again with setting PubkeyAcceptedAlgorithms and HostbasedAcceptedAlgorithms to +ssh-rsa-cert-v01@openssh.com

Yang

Succeed! Thank you for the help!
I am curious to know that if this +ssh-rsa-cert-v01@openssh.com required (instead of a regular +ssh-rsa) because of the host i am trying to connect only takes ssh-rsa-cert-v01@openssh.com auth method?

OpenSSH has obsoleted ssh-rsa (i.e. SHA-1 based) algorithms in version 8.8 because of safety reasons.

This also affects signatures in SHA-1 based certificates, though the official release notes of OpenSSH doesn't say so explicitly.

Your ~/.ssh/id_rsa-cert might use SHA-1, you can examine the certificate use the ssh-keygen command:

ssh-keygen -Lf [CERT_FILE_PATH]

Kindly Regards,

Yang

You are right:

output from ssh-keygen -Lf ~/.ssh/id_rsa-cert.pub:

Type: ssh-rsa-cert-v01@openssh.com user certificate

Thanks for the help again!

You're welcome, feel free to let me know if you have further questions.

Kindly Regards,

Yang