Need help translating command line to Core Tunnel settings


(Dave Anderson) #1

I have the following working tunnel command-line:

ssh -v -A -L8093:localhost:12445 ubuntu@bastion.example.com -t ssh -L12445:localhost:8080 -N core-load-worker-example.com

The purpose of this tunnel is to access REST endpoints on the load worker via localhost:8093/endpoint. I cannot copy the keys from the bastion host to my local machine. Note that the password used is a one-time password. I have tried to get this working via the many posts here, including the one that suggests netcat in the config file. Clearly I am missing something. Can someone please point me in the right direction, and if that fails I can post debug log output from core tunnel?

$ ssh -vvv -A -L8093:localhost:12445 ubuntu@bastion.example.com -t ssh -vvv -L12445:localhost:8080 -N core-load-worker-production.example.com
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/DA/.ssh/config
debug1: /Users/DA/.ssh/config line 1: Applying options for *
debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to bastion.example.com port 22.
debug1: Connection established.
debug1: identity file /Users/DA/.ssh/id_rsa type 0
debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_dsa type -1
debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/DA/.ssh/id_xmss type -1
debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys from 192.168.42.42
debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/DA/.ssh/known_hosts:8
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug1: Will attempt key: /Users/DA/.ssh/id_dsa
debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
debug1: Will attempt key: /Users/DA/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: send packet: type 61
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: send packet: type 61
debug3: receive packet: type 52
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to bastion.example.com ([192.168.42.42]:22).
debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:12445
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 8093.
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8093.
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug1: channel 2: new [client-session]
debug3: ssh_session2_open: channel_new: 2
debug2: channel 2: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 2: request auth-agent-req@openssh.com confirm 0
debug3: send packet: type 98
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
...
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
...
debug3: Ignored env OLDPWD
debug3: Ignored env _
debug1: Sending command: ssh -vvv -L12445:localhost:8080 -N core-load-worker-production.example.com
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/ubuntu/.ssh/config
debug1: /home/ubuntu/.ssh/config line 1: Applying options for *.example.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug1: /etc/ssh/ssh_config line 5: Deprecated option "useroaming"
debug2: resolving "core-load-worker-production.example.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to core-load-worker-production.example.com [10.4.65.210] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to core-load-worker-production.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:127
debug3: load_hostkeys: loaded 1 keys from core-load-worker-production.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:vZy...588
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:127
debug3: load_hostkeys: loaded 1 keys from core-load-worker-production.example.com
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/ubuntu/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.4.65.210
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: Host 'core-load-worker-production.example.com' is known and matches the RSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:127
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug2: fd 13 setting O_NONBLOCK
debug3: fd 13 is O_NONBLOCK
debug1: channel 3: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug3: send packet: type 91
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key:  (0x56151d9fcc80), agent
debug2: key: /home/ubuntu/.ssh/id_rsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_dsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ecdsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key:
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 535
debug2: input_userauth_pk_ok: fp SHA256:3sf...HWE
debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to core-load-worker-production.example.com ([10.4.65.210]:22).
debug1: Local connections to LOCALHOST:12445 forwarded to remote address localhost:8080
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug3: sock_set_v6only: set socket 5 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 12445.
debug2: fd 5 setting O_NONBLOCK
debug3: fd 5 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 12445.
debug2: fd 6 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82


(Yang.Y) #2

Sorry for late.

If I understand you correctly, you're trying to connect to host core-load-worker-example.com and port 8080, between your Mac and the host, you've to hop over bastion.example.com. Here are the settings you may want:

image
image

The key is use ProxyJump to hop over bastion.example.com, you do not have to create a handoff port forwarding with proxy jump.


(Dave Anderson) #3

Thank you for your reply. I tried your solution but it is not working, apparently because I need for the bastion host's keys to be used for the pubkey connection from there to the worker. I set it up as you suggested except for the debug level and it failed with a message indicating that pubkey auth failed. I am pasting the debug output from Core Tunnel here(All logs cleaned):

## Core Load Worker 5 ##

----------------------------------------
Equivalent Command: ssh -NT -A -J ubuntu@bastion.example.com -vvv -L localhost:10105:localhost:8080 -o ServerAliveInterval=15 -o PasswordAuthentication=yes -o ExitOnForwardFailure=yes -o PubkeyAuthentication=no -o ServerAliveCountMax=3 ubuntu@core-load-worker-example.com
22:16:18 Connecting…
22:16:18 Using Core Helper 4.0 (r40)
22:16:18 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
22:16:18 debug1: Reading configuration data /Users/DA/.ssh/config
22:16:18 debug1: /Users/DA/.ssh/config line 1: Applying options for *
22:16:18 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
22:16:18 debug1: Reading configuration data /etc/ssh/ssh_config
22:16:18 debug1: /etc/ssh/ssh_config line 48: Applying options for *
22:16:18 debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l ubuntu -vvv -W '[%h]:%p' bastion.example.com
22:16:18 debug1: Executing proxy xpc
22:16:18 Jumping…
22:16:18 debug1: identity file /Users/DA/.ssh/id_rsa type 0
22:16:18 debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_dsa type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_xmss type -1
22:16:18 debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
22:16:18 debug1: Local version string SSH-2.0-OpenSSH_7.9
22:16:18 Using Core Helper 4.0 (r40)
22:16:18 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
22:16:18 debug1: Reading configuration data /Users/DA/.ssh/config
22:16:18 debug1: /Users/DA/.ssh/config line 1: Applying options for *
22:16:18 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
22:16:18 debug1: Reading configuration data /etc/ssh/ssh_config
22:16:18 debug1: /etc/ssh/ssh_config line 48: Applying options for *
22:16:18 debug2: resolving "bastion.example.com" port 22
22:16:18 [bastion.example.com] debug2: ssh_connect_direct
22:16:18 [bastion.example.com] debug1: Connecting to bastion.example.com [192.168.42.42] port 22.
22:16:18 [bastion.example.com] debug1: Connection established.
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_rsa type 0
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_dsa type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_xmss type -1
22:16:18 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
22:16:18 [bastion.example.com] debug1: Local version string SSH-2.0-OpenSSH_7.9
22:16:18 [bastion.example.com] debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
22:16:18 [bastion.example.com] debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
22:16:18 [bastion.example.com] debug2: fd 5 setting O_NONBLOCK
22:16:18 [bastion.example.com] debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
22:16:18 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:16:18 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
22:16:18 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from bastion.example.com
22:16:18 [bastion.example.com] debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
22:16:18 [bastion.example.com] debug3: send packet: type 20
22:16:18 [bastion.example.com] debug1: SSH2_MSG_KEXINIT sent
22:16:18 [bastion.example.com] debug3: receive packet: type 20
22:16:18 [bastion.example.com] debug1: SSH2_MSG_KEXINIT received
22:16:18 [bastion.example.com] debug2: local client KEXINIT proposal
22:16:18 [bastion.example.com] debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
22:16:18 [bastion.example.com] debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
22:16:18 [bastion.example.com] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:18 [bastion.example.com] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:18 [bastion.example.com] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:18 [bastion.example.com] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:18 [bastion.example.com] debug2: compression ctos: none,zlib@openssh.com,zlib
22:16:18 [bastion.example.com] debug2: compression stoc: none,zlib@openssh.com,zlib
22:16:18 [bastion.example.com] debug2: languages ctos:
22:16:18 [bastion.example.com] debug2: languages stoc:
22:16:18 [bastion.example.com] debug2: first_kex_follows 0
22:16:18 [bastion.example.com] debug2: reserved 0
22:16:18 [bastion.example.com] debug2: peer server KEXINIT proposal
22:16:18 [bastion.example.com] debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
22:16:18 [bastion.example.com] debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
22:16:18 [bastion.example.com] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:18 [bastion.example.com] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:18 [bastion.example.com] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:18 [bastion.example.com] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:18 [bastion.example.com] debug2: compression ctos: none,zlib@openssh.com
22:16:18 [bastion.example.com] debug2: compression stoc: none,zlib@openssh.com
22:16:18 [bastion.example.com] debug2: languages ctos:
22:16:18 [bastion.example.com] debug2: languages stoc:
22:16:18 [bastion.example.com] debug2: first_kex_follows 0
22:16:18 [bastion.example.com] debug2: reserved 0
22:16:18 [bastion.example.com] debug1: kex: algorithm: curve25519-sha256@libssh.org
22:16:18 [bastion.example.com] debug1: kex: host key algorithm: ecdsa-sha2-nistp256
22:16:18 [bastion.example.com] debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:16:18 [bastion.example.com] debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:16:18 [bastion.example.com] debug3: send packet: type 30
22:16:18 [bastion.example.com] debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
22:16:18 [bastion.example.com] debug3: receive packet: type 31
22:16:18 [bastion.example.com] debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
22:16:18 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:16:18 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
22:16:18 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from bastion.example.com
22:16:18 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:16:18 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
22:16:18 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from 192.168.42.42
22:16:18 [bastion.example.com] debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
22:16:18 [bastion.example.com] debug1: Found key in /Users/DA/.ssh/known_hosts:8
22:16:18 [bastion.example.com] debug3: send packet: type 21
22:16:18 [bastion.example.com] debug2: set_newkeys: mode 1
22:16:18 [bastion.example.com] debug1: rekey after 134217728 blocks
22:16:18 [bastion.example.com] debug1: SSH2_MSG_NEWKEYS sent
22:16:18 [bastion.example.com] debug1: expecting SSH2_MSG_NEWKEYS
22:16:18 [bastion.example.com] debug3: receive packet: type 21
22:16:18 [bastion.example.com] debug1: SSH2_MSG_NEWKEYS received
22:16:18 [bastion.example.com] debug2: set_newkeys: mode 0
22:16:18 [bastion.example.com] debug1: rekey after 134217728 blocks
22:16:18 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:16:18 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_dsa
22:16:18 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
22:16:18 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
22:16:18 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_xmss
22:16:18 [bastion.example.com] debug2: pubkey_prepare: done
22:16:18 [bastion.example.com] debug3: send packet: type 5
22:16:18 [bastion.example.com] debug3: receive packet: type 7
22:16:18 [bastion.example.com] debug1: SSH2_MSG_EXT_INFO received
22:16:18 [bastion.example.com] debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
22:16:19 [bastion.example.com] debug3: receive packet: type 6
22:16:19 [bastion.example.com] debug2: service_accept: ssh-userauth
22:16:19 [bastion.example.com] debug1: SSH2_MSG_SERVICE_ACCEPT received
22:16:19 [bastion.example.com] debug3: send packet: type 50
22:16:19 [bastion.example.com] debug3: receive packet: type 51
22:16:19 [bastion.example.com] debug1: Authentications that can continue: publickey
22:16:19 [bastion.example.com] debug3: start over, passed a different list publickey
22:16:19 [bastion.example.com] debug3: preferred publickey,keyboard-interactive,password
22:16:19 [bastion.example.com] debug3: authmethod_lookup publickey
22:16:19 [bastion.example.com] debug3: remaining preferred: keyboard-interactive,password
22:16:19 [bastion.example.com] debug3: authmethod_is_enabled publickey
22:16:19 [bastion.example.com] debug1: Next authentication method: publickey
22:16:19 [bastion.example.com] debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:16:19 [bastion.example.com] debug3: send packet: type 50
22:16:19 [bastion.example.com] debug2: we sent a publickey packet, wait for reply
22:16:19 [bastion.example.com] debug3: receive packet: type 60
22:16:19 [bastion.example.com] debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:16:19 [bastion.example.com] debug3: sign_and_send_pubkey: RSA SHA256:3sf...9qk+HWE
22:16:19 [bastion.example.com] debug3: sign_and_send_pubkey: signing using rsa-sha2-512
22:16:19 [bastion.example.com] debug3: send packet: type 50
22:16:19 [bastion.example.com] debug3: receive packet: type 51
22:16:19 [bastion.example.com] Authenticated with partial success.
22:16:19 [bastion.example.com] debug1: Authentications that can continue: keyboard-interactive
22:16:19 [bastion.example.com] debug3: start over, passed a different list keyboard-interactive
22:16:19 [bastion.example.com] debug3: preferred publickey,keyboard-interactive,password
22:16:19 [bastion.example.com] debug3: authmethod_lookup keyboard-interactive
22:16:19 [bastion.example.com] debug3: remaining preferred: password
22:16:19 [bastion.example.com] debug3: authmethod_is_enabled keyboard-interactive
22:16:19 [bastion.example.com] debug1: Next authentication method: keyboard-interactive
22:16:19 [bastion.example.com] debug2: userauth_kbdint
22:16:19 [bastion.example.com] debug3: send packet: type 50
22:16:19 [bastion.example.com] debug2: we sent a keyboard-interactive packet, wait for reply
22:16:19 [bastion.example.com] debug3: receive packet: type 60
22:16:19 [bastion.example.com] debug2: input_userauth_info_req
22:16:19 [bastion.example.com] debug2: input_userauth_info_req: num_prompts 1
22:16:19 [bastion.example.com] debug1: read_passphrase: can't open /dev/tty: Device not configured
22:16:24 [bastion.example.com] debug3: send packet: type 61
22:16:25 [bastion.example.com] debug3: receive packet: type 60
22:16:25 [bastion.example.com] debug2: input_userauth_info_req
22:16:25 [bastion.example.com] debug2: input_userauth_info_req: num_prompts 0
22:16:25 [bastion.example.com] debug3: send packet: type 61
22:16:25 [bastion.example.com] debug3: receive packet: type 52
22:16:25 [bastion.example.com] debug1: Authentication succeeded (keyboard-interactive).
22:16:25 [bastion.example.com] Authenticated to bastion.example.com ([192.168.42.42]:22).
22:16:25 [bastion.example.com] debug3: ssh_init_stdio_forwarding: core-load-worker-example.com:22
22:16:25 [bastion.example.com] debug1: channel_connect_stdio_fwd core-load-worker-example.com:22
22:16:25 [bastion.example.com] debug1: channel 0: new [stdio-forward]
22:16:25 [bastion.example.com] debug2: fd 7 setting O_NONBLOCK
22:16:25 [bastion.example.com] debug2: fd 8 setting O_NONBLOCK
22:16:25 [bastion.example.com] debug1: getpeername failed: Bad file descriptor
22:16:25 [bastion.example.com] debug3: send packet: type 90
22:16:25 [bastion.example.com] debug2: fd 5 setting TCP_NODELAY
22:16:25 [bastion.example.com] debug3: ssh_packet_set_tos: set IP_TOS 0x48
22:16:25 [bastion.example.com] debug1: Requesting no-more-sessions@openssh.com
22:16:25 [bastion.example.com] debug3: send packet: type 80
22:16:25 [bastion.example.com] debug1: Entering interactive session.
22:16:25 [bastion.example.com] debug1: pledge: network
22:16:25 [bastion.example.com] debug2: fd 9 setting O_NONBLOCK
22:16:25 [bastion.example.com] debug2: fd 10 setting O_NONBLOCK
22:16:25 [bastion.example.com] debug3: receive packet: type 80
22:16:25 [bastion.example.com] debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
22:16:25 [bastion.example.com] debug3: receive packet: type 91
22:16:25 [bastion.example.com] debug2: channel_input_open_confirmation: channel 0: callback start
22:16:25 [bastion.example.com] debug2: channel_input_open_confirmation: channel 0: callback done
22:16:25 [bastion.example.com] debug2: channel 0: open confirm rwindow 2097152 rmax 32768
22:16:25 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
22:16:25 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
22:16:25 debug2: fd 5 setting O_NONBLOCK
22:16:25 debug2: fd 4 setting O_NONBLOCK
22:16:25 debug1: Authenticating to core-load-worker-example.com:22 as 'ubuntu'
22:16:25 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:16:25 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:9
22:16:25 Authenticating…
22:16:25 debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
22:16:25 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
22:16:25 debug3: send packet: type 20
22:16:25 debug1: SSH2_MSG_KEXINIT sent
22:16:25 debug3: receive packet: type 20
22:16:25 debug1: SSH2_MSG_KEXINIT received
22:16:25 debug2: local client KEXINIT proposal
22:16:25 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
22:16:25 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
22:16:25 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:25 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:25 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:25 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:25 debug2: compression ctos: none,zlib@openssh.com,zlib
22:16:25 debug2: compression stoc: none,zlib@openssh.com,zlib
22:16:25 debug2: languages ctos:
22:16:25 debug2: languages stoc:
22:16:25 debug2: first_kex_follows 0
22:16:25 debug2: reserved 0
22:16:25 debug2: peer server KEXINIT proposal
22:16:25 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
22:16:25 debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
22:16:25 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:25 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:16:25 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:25 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:16:25 debug2: compression ctos: none,zlib@openssh.com
22:16:25 debug2: compression stoc: none,zlib@openssh.com
22:16:25 debug2: languages ctos:
22:16:25 debug2: languages stoc:
22:16:25 debug2: first_kex_follows 0
22:16:25 debug2: reserved 0
22:16:25 debug1: kex: algorithm: curve25519-sha256@libssh.org
22:16:25 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
22:16:25 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:16:25 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:16:25 debug3: send packet: type 30
22:16:25 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
22:16:25 debug3: receive packet: type 31
22:16:25 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:B7Smr9ZjBpuvl8kWjdF0m9h12D0sAyaxzalpWSJEXJc
22:16:25 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:16:25 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:9
22:16:25 debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
22:16:25 debug1: Host 'core-load-worker-example.com' is known and matches the ECDSA host key.
22:16:25 debug1: Found key in /Users/DA/.ssh/known_hosts:9
22:16:25 debug3: send packet: type 21
22:16:25 debug2: set_newkeys: mode 1
22:16:25 debug1: rekey after 134217728 blocks
22:16:25 debug1: SSH2_MSG_NEWKEYS sent
22:16:25 debug1: expecting SSH2_MSG_NEWKEYS
22:16:25 debug3: receive packet: type 21
22:16:25 debug1: SSH2_MSG_NEWKEYS received
22:16:25 debug2: set_newkeys: mode 0
22:16:25 debug1: rekey after 134217728 blocks
22:16:25 debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:16:25 debug1: Will attempt key: /Users/DA/.ssh/id_dsa
22:16:25 debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
22:16:25 debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
22:16:25 debug1: Will attempt key: /Users/DA/.ssh/id_xmss
22:16:25 debug2: pubkey_prepare: done
22:16:25 debug3: send packet: type 5
22:16:25 debug3: receive packet: type 7
22:16:25 debug1: SSH2_MSG_EXT_INFO received
22:16:25 debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
22:16:25 debug3: receive packet: type 6
22:16:25 debug2: service_accept: ssh-userauth
22:16:25 debug1: SSH2_MSG_SERVICE_ACCEPT received
22:16:25 debug3: send packet: type 50
22:16:25 debug3: receive packet: type 51
22:16:25 debug1: Authentications that can continue: publickey
22:16:25 debug3: start over, passed a different list publickey
22:16:25 debug3: preferred keyboard-interactive,password
22:16:25 debug1: No more authentication methods to try.
22:16:25 ubuntu@core-load-worker-example.com: Permission denied (publickey).
22:16:25 [bastion.example.com] debug2: channel 0: read<=0 rfd 7 len 0
22:16:25 Abnormal Disconnect
22:16:25 [bastion.example.com] debug2: channel 0: read failed
22:16:25 [bastion.example.com] debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 7 efd -1 [closed])
22:16:25 [bastion.example.com] debug2: channel 0: input open -> drain
22:16:25 [bastion.example.com] debug2: channel 0: ibuf empty
22:16:25 [bastion.example.com] debug2: channel 0: send eof
22:16:25 [bastion.example.com] debug3: send packet: type 96
22:16:25 [bastion.example.com] debug2: channel 0: input drain -> closed
22:16:26 [bastion.example.com] debug3: receive packet: type 96
22:16:26 [bastion.example.com] debug2: channel 0: rcvd eof
22:16:26 [bastion.example.com] debug2: channel 0: output open -> drain
22:16:26 [bastion.example.com] debug2: channel 0: obuf empty
22:16:26 [bastion.example.com] debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 8 efd -1 [closed])
22:16:26 [bastion.example.com] debug2: channel 0: output drain -> closed
22:16:26 [bastion.example.com] debug3: receive packet: type 97

(Dave Anderson) #4

Note that I get a pubkey auth error as well if I uncheck "Forward Agent"; that log output is next:

## Core Load Worker 5 ##

----------------------------------------
Equivalent Command: ssh -NT -J ubuntu@bastion.example.com -vvv -L localhost:10105:localhost:8080 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o PubkeyAuthentication=no -o ServerAliveCountMax=3 -o PasswordAuthentication=yes ubuntu@core-load-worker-example.com
22:44:21 Connecting…
22:44:21 Using Core Helper 4.0 (r40)
22:44:21 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
22:44:21 debug1: Reading configuration data /Users/DA/.ssh/config
22:44:21 debug1: /Users/DA/.ssh/config line 1: Applying options for *
22:44:21 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
22:44:21 debug1: Reading configuration data /etc/ssh/ssh_config
22:44:21 debug1: /etc/ssh/ssh_config line 48: Applying options for *
22:44:21 debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l ubuntu -vvv -W '[%h]:%p' bastion.example.com
22:44:21 debug1: Executing proxy xpc
22:44:21 Jumping…
22:44:21 debug1: identity file /Users/DA/.ssh/id_rsa type 0
22:44:21 debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
22:44:21 debug1: identity file /Users/DA/.ssh/id_dsa type -1
22:44:21 debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
22:44:21 debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
22:44:21 debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
22:44:21 debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
22:44:21 debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
22:44:21 debug1: identity file /Users/DA/.ssh/id_xmss type -1
22:44:21 Using Core Helper 4.0 (r40)
22:44:21 debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
22:44:21 debug1: Local version string SSH-2.0-OpenSSH_7.9
22:44:21 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
22:44:21 debug1: Reading configuration data /Users/DA/.ssh/config
22:44:21 debug1: /Users/DA/.ssh/config line 1: Applying options for *
22:44:21 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
22:44:21 debug1: Reading configuration data /etc/ssh/ssh_config
22:44:21 debug1: /etc/ssh/ssh_config line 48: Applying options for *
22:44:21 debug2: resolving "bastion.example.com" port 22
22:44:21 [bastion.example.com] debug2: ssh_connect_direct
22:44:21 [bastion.example.com] debug1: Connecting to bastion.example.com [192.168.42.42] port 22.
22:44:21 [bastion.example.com] debug1: Connection established.
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_rsa type 0
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_dsa type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_xmss type -1
22:44:21 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
22:44:21 [bastion.example.com] debug1: Local version string SSH-2.0-OpenSSH_7.9
22:44:21 [bastion.example.com] debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
22:44:21 [bastion.example.com] debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
22:44:21 [bastion.example.com] debug2: fd 5 setting O_NONBLOCK
22:44:21 [bastion.example.com] debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
22:44:21 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:44:21 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
22:44:21 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from bastion.example.com
22:44:21 [bastion.example.com] debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
22:44:21 [bastion.example.com] debug3: send packet: type 20
22:44:21 [bastion.example.com] debug1: SSH2_MSG_KEXINIT sent
22:44:21 [bastion.example.com] debug3: receive packet: type 20
22:44:21 [bastion.example.com] debug1: SSH2_MSG_KEXINIT received
22:44:21 [bastion.example.com] debug2: local client KEXINIT proposal
22:44:21 [bastion.example.com] debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
22:44:21 [bastion.example.com] debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
22:44:21 [bastion.example.com] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:21 [bastion.example.com] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:21 [bastion.example.com] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:21 [bastion.example.com] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:21 [bastion.example.com] debug2: compression ctos: none,zlib@openssh.com,zlib
22:44:21 [bastion.example.com] debug2: compression stoc: none,zlib@openssh.com,zlib
22:44:21 [bastion.example.com] debug2: languages ctos:
22:44:21 [bastion.example.com] debug2: languages stoc:
22:44:21 [bastion.example.com] debug2: first_kex_follows 0
22:44:21 [bastion.example.com] debug2: reserved 0
22:44:21 [bastion.example.com] debug2: peer server KEXINIT proposal
22:44:21 [bastion.example.com] debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
22:44:21 [bastion.example.com] debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
22:44:21 [bastion.example.com] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:21 [bastion.example.com] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:21 [bastion.example.com] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:21 [bastion.example.com] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:22 [bastion.example.com] debug2: compression ctos: none,zlib@openssh.com
22:44:22 [bastion.example.com] debug2: compression stoc: none,zlib@openssh.com
22:44:22 [bastion.example.com] debug2: languages ctos:
22:44:22 [bastion.example.com] debug2: languages stoc:
22:44:22 [bastion.example.com] debug2: first_kex_follows 0
22:44:22 [bastion.example.com] debug2: reserved 0
22:44:22 [bastion.example.com] debug1: kex: algorithm: curve25519-sha256@libssh.org
22:44:22 [bastion.example.com] debug1: kex: host key algorithm: ecdsa-sha2-nistp256
22:44:22 [bastion.example.com] debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:44:22 [bastion.example.com] debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:44:22 [bastion.example.com] debug3: send packet: type 30
22:44:22 [bastion.example.com] debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
22:44:22 [bastion.example.com] debug3: receive packet: type 31
22:44:22 [bastion.example.com] debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
22:44:22 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:44:22 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
22:44:22 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from bastion.example.com
22:44:22 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:44:22 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
22:44:22 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from 192.168.42.42
22:44:22 [bastion.example.com] debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
22:44:22 [bastion.example.com] debug1: Found key in /Users/DA/.ssh/known_hosts:8
22:44:22 [bastion.example.com] debug3: send packet: type 21
22:44:22 [bastion.example.com] debug2: set_newkeys: mode 1
22:44:22 [bastion.example.com] debug1: rekey after 134217728 blocks
22:44:22 [bastion.example.com] debug1: SSH2_MSG_NEWKEYS sent
22:44:22 [bastion.example.com] debug1: expecting SSH2_MSG_NEWKEYS
22:44:22 [bastion.example.com] debug3: receive packet: type 21
22:44:22 [bastion.example.com] debug1: SSH2_MSG_NEWKEYS received
22:44:22 [bastion.example.com] debug2: set_newkeys: mode 0
22:44:22 [bastion.example.com] debug1: rekey after 134217728 blocks
22:44:22 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:44:22 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_dsa
22:44:22 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
22:44:22 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
22:44:22 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_xmss
22:44:22 [bastion.example.com] debug2: pubkey_prepare: done
22:44:22 [bastion.example.com] debug3: send packet: type 5
22:44:22 [bastion.example.com] debug3: receive packet: type 7
22:44:22 [bastion.example.com] debug1: SSH2_MSG_EXT_INFO received
22:44:22 [bastion.example.com] debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
22:44:22 [bastion.example.com] debug3: receive packet: type 6
22:44:22 [bastion.example.com] debug2: service_accept: ssh-userauth
22:44:22 [bastion.example.com] debug1: SSH2_MSG_SERVICE_ACCEPT received
22:44:22 [bastion.example.com] debug3: send packet: type 50
22:44:22 [bastion.example.com] debug3: receive packet: type 51
22:44:22 [bastion.example.com] debug1: Authentications that can continue: publickey
22:44:22 [bastion.example.com] debug3: start over, passed a different list publickey
22:44:22 [bastion.example.com] debug3: preferred publickey,keyboard-interactive,password
22:44:22 [bastion.example.com] debug3: authmethod_lookup publickey
22:44:22 [bastion.example.com] debug3: remaining preferred: keyboard-interactive,password
22:44:22 [bastion.example.com] debug3: authmethod_is_enabled publickey
22:44:22 [bastion.example.com] debug1: Next authentication method: publickey
22:44:22 [bastion.example.com] debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:44:22 [bastion.example.com] debug3: send packet: type 50
22:44:22 [bastion.example.com] debug2: we sent a publickey packet, wait for reply
22:44:22 [bastion.example.com] debug3: receive packet: type 60
22:44:22 [bastion.example.com] debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:44:22 [bastion.example.com] debug3: sign_and_send_pubkey: RSA SHA256:3sf...9qk+HWE
22:44:22 [bastion.example.com] debug3: sign_and_send_pubkey: signing using rsa-sha2-512
22:44:22 [bastion.example.com] debug3: send packet: type 50
22:44:22 [bastion.example.com] debug3: receive packet: type 51
22:44:22 [bastion.example.com] Authenticated with partial success.
22:44:22 [bastion.example.com] debug1: Authentications that can continue: keyboard-interactive
22:44:22 [bastion.example.com] debug3: start over, passed a different list keyboard-interactive
22:44:22 [bastion.example.com] debug3: preferred publickey,keyboard-interactive,password
22:44:22 [bastion.example.com] debug3: authmethod_lookup keyboard-interactive
22:44:22 [bastion.example.com] debug3: remaining preferred: password
22:44:22 [bastion.example.com] debug3: authmethod_is_enabled keyboard-interactive
22:44:22 [bastion.example.com] debug1: Next authentication method: keyboard-interactive
22:44:22 [bastion.example.com] debug2: userauth_kbdint
22:44:22 [bastion.example.com] debug3: send packet: type 50
22:44:22 [bastion.example.com] debug2: we sent a keyboard-interactive packet, wait for reply
22:44:23 [bastion.example.com] debug3: receive packet: type 60
22:44:23 [bastion.example.com] debug2: input_userauth_info_req
22:44:23 [bastion.example.com] debug2: input_userauth_info_req: num_prompts 1
22:44:23 [bastion.example.com] debug1: read_passphrase: can't open /dev/tty: Device not configured
22:44:27 [bastion.example.com] debug3: send packet: type 61
22:44:27 [bastion.example.com] debug3: receive packet: type 60
22:44:27 [bastion.example.com] debug2: input_userauth_info_req
22:44:27 [bastion.example.com] debug2: input_userauth_info_req: num_prompts 0
22:44:27 [bastion.example.com] debug3: send packet: type 61
22:44:28 [bastion.example.com] debug3: receive packet: type 52
22:44:28 [bastion.example.com] debug1: Authentication succeeded (keyboard-interactive).
22:44:28 [bastion.example.com] Authenticated to bastion.example.com ([192.168.42.42]:22).
22:44:28 [bastion.example.com] debug3: ssh_init_stdio_forwarding: core-load-worker-example.com:22
22:44:28 [bastion.example.com] debug1: channel_connect_stdio_fwd core-load-worker-example.com:22
22:44:28 [bastion.example.com] debug1: channel 0: new [stdio-forward]
22:44:28 [bastion.example.com] debug2: fd 7 setting O_NONBLOCK
22:44:28 [bastion.example.com] debug2: fd 8 setting O_NONBLOCK
22:44:28 [bastion.example.com] debug1: getpeername failed: Bad file descriptor
22:44:28 [bastion.example.com] debug3: send packet: type 90
22:44:28 [bastion.example.com] debug2: fd 5 setting TCP_NODELAY
22:44:28 [bastion.example.com] debug3: ssh_packet_set_tos: set IP_TOS 0x48
22:44:28 [bastion.example.com] debug1: Requesting no-more-sessions@openssh.com
22:44:28 [bastion.example.com] debug3: send packet: type 80
22:44:28 [bastion.example.com] debug1: Entering interactive session.
22:44:28 [bastion.example.com] debug1: pledge: network
22:44:28 [bastion.example.com] debug2: fd 9 setting O_NONBLOCK
22:44:28 [bastion.example.com] debug2: fd 10 setting O_NONBLOCK
22:44:28 [bastion.example.com] debug3: receive packet: type 80
22:44:28 [bastion.example.com] debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
22:44:28 [bastion.example.com] debug3: receive packet: type 91
22:44:28 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
22:44:28 [bastion.example.com] debug2: channel_input_open_confirmation: channel 0: callback start
22:44:28 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
22:44:28 [bastion.example.com] debug2: channel_input_open_confirmation: channel 0: callback done
22:44:28 debug2: fd 5 setting O_NONBLOCK
22:44:28 [bastion.example.com] debug2: channel 0: open confirm rwindow 2097152 rmax 32768
22:44:28 debug2: fd 4 setting O_NONBLOCK
22:44:28 debug1: Authenticating to core-load-worker-example.com:22 as 'ubuntu'
22:44:28 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:44:28 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:9
22:44:28 debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
22:44:28 Authenticating…
22:44:28 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
22:44:28 debug3: send packet: type 20
22:44:28 debug1: SSH2_MSG_KEXINIT sent
22:44:28 debug3: receive packet: type 20
22:44:28 debug1: SSH2_MSG_KEXINIT received
22:44:28 debug2: local client KEXINIT proposal
22:44:28 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
22:44:28 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
22:44:28 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:28 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:28 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:28 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:28 debug2: compression ctos: none,zlib@openssh.com,zlib
22:44:28 debug2: compression stoc: none,zlib@openssh.com,zlib
22:44:28 debug2: languages ctos:
22:44:28 debug2: languages stoc:
22:44:28 debug2: first_kex_follows 0
22:44:28 debug2: reserved 0
22:44:28 debug2: peer server KEXINIT proposal
22:44:28 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
22:44:28 debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
22:44:28 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:28 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
22:44:28 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:28 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
22:44:28 debug2: compression ctos: none,zlib@openssh.com
22:44:28 debug2: compression stoc: none,zlib@openssh.com
22:44:28 debug2: languages ctos:
22:44:28 debug2: languages stoc:
22:44:28 debug2: first_kex_follows 0
22:44:28 debug2: reserved 0
22:44:28 debug1: kex: algorithm: curve25519-sha256@libssh.org
22:44:28 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
22:44:28 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:44:28 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
22:44:28 debug3: send packet: type 30
22:44:28 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
22:44:28 debug3: receive packet: type 31
22:44:28 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:B7Smr9ZjBpuvl8kWjdF0m9h12D0sAyaxzalpWSJEXJc
22:44:28 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
22:44:28 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:9
22:44:28 debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
22:44:28 debug1: Host 'core-load-worker-example.com' is known and matches the ECDSA host key.
22:44:28 debug1: Found key in /Users/DA/.ssh/known_hosts:9
22:44:28 debug3: send packet: type 21
22:44:28 debug2: set_newkeys: mode 1
22:44:28 debug1: rekey after 134217728 blocks
22:44:28 debug1: SSH2_MSG_NEWKEYS sent
22:44:28 debug1: expecting SSH2_MSG_NEWKEYS
22:44:28 debug3: receive packet: type 21
22:44:28 debug1: SSH2_MSG_NEWKEYS received
22:44:28 debug2: set_newkeys: mode 0
22:44:28 debug1: rekey after 134217728 blocks
22:44:28 debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...9qk+HWE agent
22:44:28 debug1: Will attempt key: /Users/DA/.ssh/id_dsa
22:44:28 debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
22:44:28 debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
22:44:28 debug1: Will attempt key: /Users/DA/.ssh/id_xmss
22:44:28 debug2: pubkey_prepare: done
22:44:28 debug3: send packet: type 5
22:44:28 debug3: receive packet: type 7
22:44:28 debug1: SSH2_MSG_EXT_INFO received
22:44:28 debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
22:44:28 debug3: receive packet: type 6
22:44:28 debug2: service_accept: ssh-userauth
22:44:28 debug1: SSH2_MSG_SERVICE_ACCEPT received
22:44:28 debug3: send packet: type 50
22:44:28 debug3: receive packet: type 51
22:44:28 debug1: Authentications that can continue: publickey
22:44:28 debug3: start over, passed a different list publickey
22:44:28 debug3: preferred keyboard-interactive,password
22:44:28 debug1: No more authentication methods to try.
22:44:28 ubuntu@core-load-worker-example.com: Permission denied (publickey).
22:44:28 [bastion.example.com] debug2: channel 0: read<=0 rfd 7 len 0
22:44:28 Abnormal Disconnect
22:44:28 [bastion.example.com] debug2: channel 0: read failed
22:44:28 [bastion.example.com] debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 7 efd -1 [closed])
22:44:28 [bastion.example.com] debug2: channel 0: input open -> drain
22:44:28 [bastion.example.com] debug2: channel 0: ibuf empty
22:44:28 [bastion.example.com] debug2: channel 0: send eof
22:44:28 [bastion.example.com] debug3: send packet: type 96
22:44:28 [bastion.example.com] debug2: channel 0: input drain -> closed
22:44:29 [bastion.example.com] debug3: receive packet: type 96
22:44:29 [bastion.example.com] debug2: channel 0: rcvd eof
22:44:29 [bastion.example.com] debug2: channel 0: output open -> drain
22:44:29 [bastion.example.com] debug2: channel 0: obuf empty
22:44:29 [bastion.example.com] debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 8 efd -1 [closed])
22:44:29 [bastion.example.com] debug2: channel 0: output drain -> closed
22:44:29 [bastion.example.com] debug3: receive packet: type 97


(Dave Anderson) #5

Below is output from successful connection from bastion to worker at terminal using ssh -vvv core-load-worker-example.com showing use of the bastion's keys for pubkey auth.

Last login: Wed May 22 16:38:06 on ttys013
DA@DAs-MacBook-Pro ~  $ ssh pmc
Password:
ubuntu@bastion.example.com:~$ ssh -vvv core-load-worker-example.com
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/ubuntu/.ssh/config
debug1: /home/ubuntu/.ssh/config line 1: Applying options for *.example.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug1: /etc/ssh/ssh_config line 5: Deprecated option "useroaming"
debug2: resolving "core-load-worker-example.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to core-load-worker-example.com [10.9.8.7] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to core-load-worker-example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:133
debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:vZy...588
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:133
debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/ubuntu/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.9.8.7
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: Host 'core-load-worker-example.com' is known and matches the RSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:133
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key:  (0x56215b033310), agent
debug2: key: /home/ubuntu/.ssh/id_rsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_dsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ecdsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key:
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 535
debug2: input_userauth_pk_ok: fp SHA256:3sf...9qk+HWE
debug3: sign_and_send_pubkey: RSA SHA256:3sf...9qk+HWE
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to core-load-worker-example.com ([10.9.8.7]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 0: request auth-agent-req@openssh.com confirm 0
debug3: send packet: type 98
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
ubuntu@core-load-worker-example.com:~$ debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80


(Yang.Y) #6

Forgot to mention, please also enable Core Helper if you would like to forward auth agent:

image


(Dave Anderson) #7

Yes, that is enabled. I don't think I want to forward, I think I want to let bastion do its own pubkey auth, so I will try disabling core helper.


(Dave Anderson) #8

Disabling core helper did not change anything.


(Yang.Y) #9

ProxyJump directive seems not forward auth agent, could you please try again with ProxyCommand, set its value to ssh ubuntu@bastion.example.com -W %h:%p:

Also, in order to use ProxyCommand, you have to enable Core Helper and clear ProxyJump option.


(Yang.Y) #10

If you still out of luck, please edit ~/.ssh/config and make sure it contains following lines:

Host bastion.example.com
    ForwardAgent yes

Host core-load-worker-example.com
    ForwardAgent yes

(Dave Anderson) #11

I followed your instructions exactly, including modifying my local ~/.ssh/config to include the settings for the worker - bastion already had forwarding enabled. With this setup I can't even get the first hop authenticated, as evidenced by the log output below. Note that I tried:

with the config entry for worker and without;
with the ProxyCommand written exactly as you show and without the colon after the port number.

I have tried all four possible combinations of those two changes and the only difference I saw in the logs is that the line beginning with >>:>>:>> appeared when I had the worker entry in config. Read past the log for an interesting experiment.

Log:

## Core Load Worker ##

----------------------------------------
Equivalent Command: ssh -NT -vvv -L localhost:10105:localhost:8080 -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o PubkeyAuthentication=no -o PasswordAuthentication=yes -o ServerAliveCountMax=3 -o ProxyCommand="ssh ubuntu@bastion.example.com -W %h:%p:" ubuntu@core-load-worker-example.com
22:33:20 Connecting…
22:33:20 Using Core Helper 4.0 (r40)
22:33:20 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
22:33:20 debug1: Reading configuration data /Users/DA/.ssh/config
22:33:20 debug1: /Users/DA/.ssh/config line 1: Applying options for *
22:33:20 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
>>:>>:>> debug1: /Users/DA/.ssh/config line 53: Applying options for ubuntu@core-load-worker-example.com
22:33:20 debug1: Executing proxy command: exec ssh ubuntu@bastion.example.com -W core-load-worker-example.com:22
22:33:20 Jumping…
22:33:20 debug1: identity file /Users/DA/.ssh/id_rsa type 0
22:33:20 debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_dsa type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_xmss type -1
22:33:20 debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
22:33:20 debug1: Local version string SSH-2.0-OpenSSH_7.9
ubuntu@bastion.example.com: Permission denied (keyboard-interactive).
22:33:22 ssh_exchange_identification: Connection closed by remote host
22:33:22 Abnormal Disconnect
22:33:22 Connection failed, retry after 3s…
----------------------------------------
Equivalent Command: ssh -NT -vvv -L localhost:10105:localhost:8080 -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o PubkeyAuthentication=no -o PasswordAuthentication=yes -o ServerAliveCountMax=3 -o ProxyCommand="ssh ubuntu@bastion.example.com -W %h:%p:" ubuntu@core-load-worker-example.com
22:33:25 Connecting…
22:33:25 Using Core Helper 4.0 (r40)
22:33:25 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
22:33:25 debug1: Reading configuration data /Users/DA/.ssh/config
22:33:25 debug1: /Users/DA/.ssh/config line 1: Applying options for *
22:33:25 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
22:33:25 debug1: Executing proxy command: exec ssh ubuntu@bastion.example.com -W core-load-worker-example.com:22
22:33:25 Jumping…
22:33:25 debug1: identity file /Users/DA/.ssh/id_rsa type 0
22:33:25 debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_dsa type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_xmss type -1
22:33:25 debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
22:33:25 debug1: Local version string SSH-2.0-OpenSSH_7.9
ubuntu@bastion.example.com: Permission denied (keyboard-interactive).
22:33:27 ssh_exchange_identification: Connection closed by remote host
22:33:27 Abnormal Disconnect
22:33:27 Connection failed, retry after 3s…
22:33:29 Disconnected

I copied the ssh command from the log and ran it in my terminal:

 $ ssh ubuntu@bastion.example.com -W core-load-worker-example.com:22
Password:
SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
^C $

From the terminal I was prompted for the OTP as expected - Core Tunnel never prompted me for my password since the ProxyCommand change. The helper has been enabled through this entire process except yesterday when I briefly disabled it.

It's curious that I can run the command from a terminal and it appears to create a working pipe(note the Ctrl-C I used to kill it) yet this command when exec'ed by Core Tunnel fails. Any other data I can gather to help characterize this?


(Yang.Y) #12

There might has a bug, ssh_askpass should be invoked when executing ProxyCommand, I'll investigate this issue.

Could you please clear ProxyCommand and set ProxyJump to ubuntu@bastion.example.com and try again?


(Dave Anderson) #13

That did ask for a password, but the jump failed.

I can't help but think we are going about this wrong. When I made the connection directly from bastion to worker above using a terminal session, bastion's keys were used(/home/ubuntu/...). All of these logs from Core Tunnel seem to be trying to use my own local keys to authenticate on the worker and I don't see how that can work. As I mentioned at the outset, I'm not supposed to have those keys locally.

In any case, here is the log from setting as you requested, ProxyCommand cleared and ProxyJump set as described.

## Core Load Worker 5 ##

----------------------------------------
Equivalent Command: ssh -NT -J ubuntu@bastion.example.com -vvv -L localhost:10105:localhost:8080 -o ServerAliveCountMax=3 -o PubkeyAuthentication=no -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o PasswordAuthentication=yes ubuntu@core-load-worker-example.com
17:08:28 Connecting…
17:08:28 Using Core Helper 4.0 (r40)
17:08:28 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
17:08:28 debug1: Reading configuration data /Users/DA/.ssh/config
17:08:28 debug1: /Users/DA/.ssh/config line 1: Applying options for *
17:08:28 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
17:08:28 debug1: /Users/DA/.ssh/config line 53: Applying options for core-load-worker-example.com
17:08:28 debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l ubuntu -F /Users/DA/.ssh/config -vvv -W '[%h]:%p' bastion.example.com
17:08:28 debug1: Executing proxy xpc
17:08:28 Jumping…
17:08:28 debug1: identity file /Users/DA/.ssh/id_rsa type 0
17:08:28 debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
17:08:28 debug1: identity file /Users/DA/.ssh/id_dsa type -1
17:08:28 debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
17:08:28 debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
17:08:28 debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
17:08:28 debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
17:08:28 debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
17:08:28 Using Core Helper 4.0 (r40)
17:08:28 debug1: identity file /Users/DA/.ssh/id_xmss type -1
17:08:28 debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
17:08:28 debug1: Local version string SSH-2.0-OpenSSH_7.9
17:08:28 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
17:08:28 debug1: Reading configuration data /Users/DA/.ssh/config
17:08:28 debug1: /Users/DA/.ssh/config line 1: Applying options for *
17:08:28 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
17:08:28 debug2: resolving "bastion.example.com" port 22
17:08:28 [bastion.example.com] debug2: ssh_connect_direct
17:08:28 [bastion.example.com] debug1: Connecting to bastion.example.com [192.168.42.42] port 22.
17:08:28 [bastion.example.com] debug1: Connection established.
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_rsa type 0
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_dsa type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_xmss type -1
17:08:28 [bastion.example.com] debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
17:08:28 [bastion.example.com] debug1: Local version string SSH-2.0-OpenSSH_7.9
17:08:29 [bastion.example.com] debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
17:08:29 [bastion.example.com] debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
17:08:29 [bastion.example.com] debug2: fd 5 setting O_NONBLOCK
17:08:29 [bastion.example.com] debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
17:08:29 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
17:08:29 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
17:08:29 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from bastion.example.com
17:08:29 [bastion.example.com] debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
17:08:29 [bastion.example.com] debug3: send packet: type 20
17:08:29 [bastion.example.com] debug1: SSH2_MSG_KEXINIT sent
17:08:29 [bastion.example.com] debug3: receive packet: type 20
17:08:29 [bastion.example.com] debug1: SSH2_MSG_KEXINIT received
17:08:29 [bastion.example.com] debug2: local client KEXINIT proposal
17:08:29 [bastion.example.com] debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
17:08:29 [bastion.example.com] debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
17:08:29 [bastion.example.com] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:08:29 [bastion.example.com] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:08:29 [bastion.example.com] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:08:29 [bastion.example.com] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:08:29 [bastion.example.com] debug2: compression ctos: none,zlib@openssh.com,zlib
17:08:29 [bastion.example.com] debug2: compression stoc: none,zlib@openssh.com,zlib
17:08:29 [bastion.example.com] debug2: languages ctos:
17:08:29 [bastion.example.com] debug2: languages stoc:
17:08:29 [bastion.example.com] debug2: first_kex_follows 0
17:08:29 [bastion.example.com] debug2: reserved 0
17:08:29 [bastion.example.com] debug2: peer server KEXINIT proposal
17:08:29 [bastion.example.com] debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
17:08:29 [bastion.example.com] debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
17:08:29 [bastion.example.com] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:08:29 [bastion.example.com] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:08:29 [bastion.example.com] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:08:29 [bastion.example.com] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:08:29 [bastion.example.com] debug2: compression ctos: none,zlib@openssh.com
17:08:29 [bastion.example.com] debug2: compression stoc: none,zlib@openssh.com
17:08:29 [bastion.example.com] debug2: languages ctos:
17:08:29 [bastion.example.com] debug2: languages stoc:
17:08:29 [bastion.example.com] debug2: first_kex_follows 0
17:08:29 [bastion.example.com] debug2: reserved 0
17:08:29 [bastion.example.com] debug1: kex: algorithm: curve25519-sha256@libssh.org
17:08:29 [bastion.example.com] debug1: kex: host key algorithm: ecdsa-sha2-nistp256
17:08:29 [bastion.example.com] debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
17:08:29 [bastion.example.com] debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
17:08:29 [bastion.example.com] debug3: send packet: type 30
17:08:29 [bastion.example.com] debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
17:08:29 [bastion.example.com] debug3: receive packet: type 31
17:08:29 [bastion.example.com] debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
17:08:29 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
17:08:29 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
17:08:29 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from bastion.example.com
17:08:29 [bastion.example.com] debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
17:08:29 [bastion.example.com] debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
17:08:29 [bastion.example.com] debug3: load_hostkeys: loaded 1 keys from 192.168.42.42
17:08:29 [bastion.example.com] debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
17:08:29 [bastion.example.com] debug1: Found key in /Users/DA/.ssh/known_hosts:7
17:08:29 [bastion.example.com] debug3: send packet: type 21
17:08:29 [bastion.example.com] debug2: set_newkeys: mode 1
17:08:29 [bastion.example.com] debug1: rekey after 134217728 blocks
17:08:29 [bastion.example.com] debug1: SSH2_MSG_NEWKEYS sent
17:08:29 [bastion.example.com] debug1: expecting SSH2_MSG_NEWKEYS
17:08:29 [bastion.example.com] debug3: receive packet: type 21
17:08:29 [bastion.example.com] debug1: SSH2_MSG_NEWKEYS received
17:08:29 [bastion.example.com] debug2: set_newkeys: mode 0
17:08:29 [bastion.example.com] debug1: rekey after 134217728 blocks
17:08:29 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
17:08:29 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_dsa
17:08:29 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
17:08:29 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
17:08:29 [bastion.example.com] debug1: Will attempt key: /Users/DA/.ssh/id_xmss
17:08:29 [bastion.example.com] debug2: pubkey_prepare: done
17:08:29 [bastion.example.com] debug3: send packet: type 5
17:08:29 [bastion.example.com] debug3: receive packet: type 7
17:08:29 [bastion.example.com] debug1: SSH2_MSG_EXT_INFO received
17:08:29 [bastion.example.com] debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
17:08:29 [bastion.example.com] debug3: receive packet: type 6
17:08:29 [bastion.example.com] debug2: service_accept: ssh-userauth
17:08:29 [bastion.example.com] debug1: SSH2_MSG_SERVICE_ACCEPT received
17:08:29 [bastion.example.com] debug3: send packet: type 50
17:08:29 [bastion.example.com] debug3: receive packet: type 51
17:08:29 [bastion.example.com] debug1: Authentications that can continue: publickey
17:08:29 [bastion.example.com] debug3: start over, passed a different list publickey
17:08:29 [bastion.example.com] debug3: preferred publickey,keyboard-interactive,password
17:08:29 [bastion.example.com] debug3: authmethod_lookup publickey
17:08:29 [bastion.example.com] debug3: remaining preferred: keyboard-interactive,password
17:08:29 [bastion.example.com] debug3: authmethod_is_enabled publickey
17:08:29 [bastion.example.com] debug1: Next authentication method: publickey
17:08:29 [bastion.example.com] debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
17:08:29 [bastion.example.com] debug3: send packet: type 50
17:08:29 [bastion.example.com] debug2: we sent a publickey packet, wait for reply
17:08:29 [bastion.example.com] debug3: receive packet: type 60
17:08:29 [bastion.example.com] debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
17:08:29 [bastion.example.com] debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
17:08:29 [bastion.example.com] debug3: sign_and_send_pubkey: signing using rsa-sha2-512
17:08:29 [bastion.example.com] debug3: send packet: type 50
17:08:29 [bastion.example.com] debug3: receive packet: type 51
17:08:29 [bastion.example.com] Authenticated with partial success.
17:08:29 [bastion.example.com] debug1: Authentications that can continue: keyboard-interactive
17:08:29 [bastion.example.com] debug3: start over, passed a different list keyboard-interactive
17:08:29 [bastion.example.com] debug3: preferred publickey,keyboard-interactive,password
17:08:29 [bastion.example.com] debug3: authmethod_lookup keyboard-interactive
17:08:29 [bastion.example.com] debug3: remaining preferred: password
17:08:29 [bastion.example.com] debug3: authmethod_is_enabled keyboard-interactive
17:08:29 [bastion.example.com] debug1: Next authentication method: keyboard-interactive
17:08:29 [bastion.example.com] debug2: userauth_kbdint
17:08:29 [bastion.example.com] debug3: send packet: type 50
17:08:29 [bastion.example.com] debug2: we sent a keyboard-interactive packet, wait for reply
17:08:29 [bastion.example.com] debug3: receive packet: type 60
17:08:29 [bastion.example.com] debug2: input_userauth_info_req
17:08:29 [bastion.example.com] debug2: input_userauth_info_req: num_prompts 1
17:08:29 [bastion.example.com] debug1: read_passphrase: can't open /dev/tty: Device not configured
17:08:34 [bastion.example.com] debug3: send packet: type 61
17:08:34 [bastion.example.com] debug3: receive packet: type 51
17:08:34 [bastion.example.com] debug1: Authentications that can continue: keyboard-interactive
17:08:34 [bastion.example.com] debug2: userauth_kbdint
17:08:34 [bastion.example.com] debug3: send packet: type 50
17:08:34 [bastion.example.com] debug2: we sent a keyboard-interactive packet, wait for reply
17:08:34 [bastion.example.com] debug3: receive packet: type 60
17:08:34 [bastion.example.com] debug2: input_userauth_info_req
17:08:34 [bastion.example.com] debug2: input_userauth_info_req: num_prompts 1
17:08:34 [bastion.example.com] debug1: read_passphrase: can't open /dev/tty: Device not configured
17:09:09 [bastion.example.com] debug3: send packet: type 61
17:09:09 [bastion.example.com] debug3: receive packet: type 60
17:09:09 [bastion.example.com] debug2: input_userauth_info_req
17:09:09 [bastion.example.com] debug2: input_userauth_info_req: num_prompts 0
17:09:09 [bastion.example.com] debug3: send packet: type 61
17:09:09 [bastion.example.com] debug3: receive packet: type 52
17:09:09 [bastion.example.com] debug1: Authentication succeeded (keyboard-interactive).
17:09:09 [bastion.example.com] Authenticated to bastion.example.com ([192.168.42.42]:22).
17:09:09 [bastion.example.com] debug3: ssh_init_stdio_forwarding: core-load-worker-example.com:22
17:09:09 [bastion.example.com] debug1: channel_connect_stdio_fwd core-load-worker-example.com:22
17:09:09 [bastion.example.com] debug1: channel 0: new [stdio-forward]
17:09:09 [bastion.example.com] debug2: fd 7 setting O_NONBLOCK
17:09:09 [bastion.example.com] debug2: fd 8 setting O_NONBLOCK
17:09:09 [bastion.example.com] debug1: getpeername failed: Bad file descriptor
17:09:09 [bastion.example.com] debug3: send packet: type 90
17:09:09 [bastion.example.com] debug2: fd 5 setting TCP_NODELAY
17:09:09 [bastion.example.com] debug3: ssh_packet_set_tos: set IP_TOS 0x48
17:09:09 [bastion.example.com] debug1: Requesting no-more-sessions@openssh.com
17:09:09 [bastion.example.com] debug3: send packet: type 80
17:09:09 [bastion.example.com] debug1: Entering interactive session.
17:09:09 [bastion.example.com] debug1: pledge: network
17:09:09 [bastion.example.com] debug2: fd 9 setting O_NONBLOCK
17:09:09 [bastion.example.com] debug2: fd 10 setting O_NONBLOCK
17:09:10 [bastion.example.com] debug3: receive packet: type 80
17:09:10 [bastion.example.com] debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
17:09:10 [bastion.example.com] debug3: receive packet: type 91
17:09:10 [bastion.example.com] debug2: channel_input_open_confirmation: channel 0: callback start
17:09:10 [bastion.example.com] debug2: channel_input_open_confirmation: channel 0: callback done
17:09:10 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
17:09:10 [bastion.example.com] debug2: channel 0: open confirm rwindow 2097152 rmax 32768
17:09:10 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
17:09:10 debug2: fd 5 setting O_NONBLOCK
17:09:10 debug2: fd 4 setting O_NONBLOCK
17:09:10 debug1: Authenticating to core-load-worker-example.com:22 as 'ubuntu'
17:09:10 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
17:09:10 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
17:09:10 Authenticating…
17:09:10 debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
17:09:10 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
17:09:10 debug3: send packet: type 20
17:09:10 debug1: SSH2_MSG_KEXINIT sent
17:09:10 debug3: receive packet: type 20
17:09:10 debug1: SSH2_MSG_KEXINIT received
17:09:10 debug2: local client KEXINIT proposal
17:09:10 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
17:09:10 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
17:09:10 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:09:10 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:09:10 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:09:10 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:09:10 debug2: compression ctos: none,zlib@openssh.com,zlib
17:09:10 debug2: compression stoc: none,zlib@openssh.com,zlib
17:09:10 debug2: languages ctos:
17:09:10 debug2: languages stoc:
17:09:10 debug2: first_kex_follows 0
17:09:10 debug2: reserved 0
17:09:10 debug2: peer server KEXINIT proposal
17:09:10 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
17:09:10 debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
17:09:10 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:09:10 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
17:09:10 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:09:10 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
17:09:10 debug2: compression ctos: none,zlib@openssh.com
17:09:10 debug2: compression stoc: none,zlib@openssh.com
17:09:10 debug2: languages ctos:
17:09:10 debug2: languages stoc:
17:09:10 debug2: first_kex_follows 0
17:09:10 debug2: reserved 0
17:09:10 debug1: kex: algorithm: curve25519-sha256@libssh.org
17:09:10 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
17:09:10 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
17:09:10 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
17:09:10 debug3: send packet: type 30
17:09:10 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
17:09:10 debug3: receive packet: type 31
17:09:10 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:B7Smr9ZjBpuvl8kWjdF0m9h12D0sAyaxzalpWSJEXJc
17:09:10 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
17:09:10 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:8
17:09:10 debug3: load_hostkeys: loaded 1 keys from core-load-worker-example.com
17:09:10 debug1: Host 'core-load-worker-example.com' is known and matches the ECDSA host key.
17:09:10 debug1: Found key in /Users/DA/.ssh/known_hosts:8
17:09:10 debug3: send packet: type 21
17:09:10 debug2: set_newkeys: mode 1
17:09:10 debug1: rekey after 134217728 blocks
17:09:10 debug1: SSH2_MSG_NEWKEYS sent
17:09:10 debug1: expecting SSH2_MSG_NEWKEYS
17:09:10 debug3: receive packet: type 21
17:09:10 debug1: SSH2_MSG_NEWKEYS received
17:09:10 debug2: set_newkeys: mode 0
17:09:10 debug1: rekey after 134217728 blocks
17:09:10 debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
17:09:10 debug1: Will attempt key: /Users/DA/.ssh/id_dsa
17:09:10 debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
17:09:10 debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
17:09:10 debug1: Will attempt key: /Users/DA/.ssh/id_xmss
17:09:10 debug2: pubkey_prepare: done
17:09:10 debug3: send packet: type 5
17:09:10 debug3: receive packet: type 7
17:09:10 debug1: SSH2_MSG_EXT_INFO received
17:09:10 debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
17:09:10 debug3: receive packet: type 6
17:09:10 debug2: service_accept: ssh-userauth
17:09:10 debug1: SSH2_MSG_SERVICE_ACCEPT received
17:09:10 debug3: send packet: type 50
17:09:10 debug3: receive packet: type 51
17:09:10 debug1: Authentications that can continue: publickey
17:09:10 debug3: start over, passed a different list publickey
17:09:10 debug3: preferred keyboard-interactive,password
17:09:10 debug1: No more authentication methods to try.
17:09:10 [bastion.example.com] debug2: channel 0: read<=0 rfd 7 len 0
17:09:10 ubuntu@core-load-worker-example.com: Permission denied (publickey).
17:09:10 [bastion.example.com] debug2: channel 0: read failed
17:09:10 [bastion.example.com] debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 7 efd -1 [closed])
17:09:10 Abnormal Disconnect
17:09:10 [bastion.example.com] debug2: channel 0: input open -> drain
17:09:10 [bastion.example.com] debug2: channel 0: ibuf empty
17:09:10 [bastion.example.com] debug2: channel 0: send eof
17:09:10 [bastion.example.com] debug3: send packet: type 96
17:09:10 [bastion.example.com] debug2: channel 0: input drain -> closed
17:09:10 [bastion.example.com] debug3: receive packet: type 96
17:09:10 [bastion.example.com] debug2: channel 0: rcvd eof
17:09:10 [bastion.example.com] debug2: channel 0: output open -> drain
17:09:10 [bastion.example.com] debug2: channel 0: obuf empty
17:09:10 [bastion.example.com] debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 8 efd -1 [closed])
17:09:10 [bastion.example.com] debug2: channel 0: output drain -> closed
17:09:10 [bastion.example.com] debug3: receive packet: type 97


(Yang.Y) #14

You're right, and I'm sorry for my fault. I thought you use agent forwarding for authentication, but obviously you need the key on bastion.example.com.

Both ProxyCommand and ProxyJump could not use the key on bastion, is it possible for you to copy that key to local Mac computer?


(Dave Anderson) #15

I'm afraid that would compromise my company's security posture. As I mentioned in the OP it's not allowed.

Moreover, I am sure that it's not necessary to do so as I demonstrated with the command and log in the OP.

Worst case it seems it should be possible to wrap that command and feed it parameters from the frontend of Core Tunnel. That may not support every Core Tunnel feature but at least it would allow configuring/launching tunnels from the same UI.


(Yang.Y) #16

I will investigate this use case, and try to find out a solution, ASAP.


(Dave Anderson) #17

That sounds awesome. Thanks for looking into this!


(Yang.Y) #18

Dave, please upgrade to latest Core Tunnel (version 1.7), and clear ProxyCommand and ProxyJump fields.

Then set RemoteCommand to:

ssh -L12445:localhost:8080 -N core-load-worker-example.com

I did some tests, and with version 1.7 this solution works for me. But it has one caveat you should aware:

  • Authentication methods on host core-load-worker-example.com that require keyboard inputs (aka passwords, private key passphrases) are not supported

I would recommend you use auth agent on core-load-worker-example.com, so that the command ssh -L12445:localhost:8080 -N core-load-worker-example.com won't ask you for password/passphrase.


(Dave Anderson) #19

Thanks for the update. Where do I enter the Jump server in this case?


(Yang.Y) #20

First create a tunnel point to bastion host:
image

Then set its RemoteCommand option to:

ssh -L12445:localhost:8080 -N core-load-worker-example.com

image