Need help translating command line to Core Tunnel settings


(Dave Anderson) #21

Quick Feedback: As configured it's not asking for a password like it should at the outset. I have to run, will run some detailed experiments a bit later tonight and will post the output.


(Dave Anderson) #22

Hi Yang,

I have attached a file that has output from eight different tunnel attempts. The first four are from the command line and the last four are from Core Tunnel. Each group of four was run with the following variations and each variation was labeled as shown below:

[-vvv-vvv] --> Full debug output on both ssh commands
[-vvv-___] --> Full debug output only on first ssh command
[-___-vvv] --> Full debug output only on second ssh command
[-___-___] --> No debug output

I organized it this way so that it is more apparent what is happening at each stage of the connection. In the case of Core Tunnel with the settings that you suggested, I was never prompted for the password to bastion so the connection failed there.

Note that setting up the host and user exactly as you describe results in the string ubuntu@ubuntu@bastion.example.com appearing in the connect string. I set host to bastion.example.com and user to ubuntu and things behaved marginally better.

I am not sure why Core Tunnel never asks for a password with this setup, but it does for the simpler connections that I use Core Tunnel for.

CombinedDebug.txt (69.3 KB)


(Yang.Y) #23

Please clear PasswordAuthentication and PubkeyAuthentication in your tunnel's advanced options.

image


(Dave Anderson) #24

Unfortunately, while that seems to have made the initial connection to bastion, it seems that it never attempted to pick up the keys from bastion to connect to the worker. In fact it appears that the attempt to connect to worker caused the connection to bastion to close:


----------------------------------------
Equivalent Command: ssh -vvv -L 8093:localhost:14242 -o ExitOnForwardFailure=yes -o RemoteCommand="ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com" -o ServerAliveInterval=15 -o ServerAliveCountMax=3 ubuntu@bastion.example.com
10:20:47 Connectingā€¦
10:20:47 Using Core Helper 4.0 (r40)
10:20:47 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
10:20:47 debug1: Reading configuration data /Users/DA/.ssh/config
10:20:47 debug1: /Users/DA/.ssh/config line 1: Applying options for *
10:20:47 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
10:20:47 debug3: expanding RemoteCommand: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
10:20:47 debug3: expanded RemoteCommand: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
10:20:47 debug2: resolving "bastion.example.com" port 22
10:20:47 debug2: ssh_connect_direct
10:20:47 debug1: Connecting to bastion.example.com [123.45.67.89] port 22.
10:20:48 debug1: Connection established.
10:20:48 debug1: identity file /Users/DA/.ssh/id_rsa type 0
10:20:48 debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_dsa type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_xmss type -1
10:20:48 debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
10:20:48 debug1: Local version string SSH-2.0-OpenSSH_7.9
10:20:48 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
10:20:48 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
10:20:48 debug2: fd 5 setting O_NONBLOCK
10:20:48 debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
10:20:48 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
10:20:48 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
10:20:48 Authenticatingā€¦
10:20:48 debug3: load_hostkeys: loaded 1 keys from bastion.example.com
10:20:48 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
10:20:48 debug3: send packet: type 20
10:20:48 debug1: SSH2_MSG_KEXINIT sent
10:20:48 debug3: receive packet: type 20
10:20:48 debug1: SSH2_MSG_KEXINIT received
10:20:48 debug2: local client KEXINIT proposal
10:20:48 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
10:20:48 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
10:20:48 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
10:20:48 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
10:20:48 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
10:20:48 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
10:20:48 debug2: compression ctos: none,zlib@openssh.com,zlib
10:20:48 debug2: compression stoc: none,zlib@openssh.com,zlib
10:20:48 debug2: languages ctos:
10:20:48 debug2: languages stoc:
10:20:48 debug2: first_kex_follows 0
10:20:48 debug2: reserved 0
10:20:48 debug2: peer server KEXINIT proposal
10:20:48 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
10:20:48 debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
10:20:48 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
10:20:48 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
10:20:48 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
10:20:48 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
10:20:48 debug2: compression ctos: none,zlib@openssh.com
10:20:48 debug2: compression stoc: none,zlib@openssh.com
10:20:48 debug2: languages ctos:
10:20:48 debug2: languages stoc:
10:20:48 debug2: first_kex_follows 0
10:20:48 debug2: reserved 0
10:20:48 debug1: kex: algorithm: curve25519-sha256@libssh.org
10:20:48 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
10:20:48 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
10:20:48 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
10:20:48 debug3: send packet: type 30
10:20:48 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
10:20:48 debug3: receive packet: type 31
10:20:48 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
10:20:48 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
10:20:48 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
10:20:48 debug3: load_hostkeys: loaded 1 keys from bastion.example.com
10:20:48 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
10:20:48 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
10:20:48 debug3: load_hostkeys: loaded 1 keys from 123.45.67.89
10:20:48 debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
10:20:48 debug1: Found key in /Users/DA/.ssh/known_hosts:7
10:20:48 debug3: send packet: type 21
10:20:48 debug2: set_newkeys: mode 1
10:20:48 debug1: rekey after 134217728 blocks
10:20:48 debug1: SSH2_MSG_NEWKEYS sent
10:20:48 debug1: expecting SSH2_MSG_NEWKEYS
10:20:48 debug3: receive packet: type 21
10:20:48 debug1: SSH2_MSG_NEWKEYS received
10:20:48 debug2: set_newkeys: mode 0
10:20:48 debug1: rekey after 134217728 blocks
10:20:48 debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
10:20:48 debug1: Will attempt key: /Users/DA/.ssh/id_dsa
10:20:48 debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
10:20:48 debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
10:20:48 debug1: Will attempt key: /Users/DA/.ssh/id_xmss
10:20:48 debug2: pubkey_prepare: done
10:20:48 debug3: send packet: type 5
10:20:48 debug3: receive packet: type 7
10:20:48 debug1: SSH2_MSG_EXT_INFO received
10:20:48 debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
10:20:48 debug3: receive packet: type 6
10:20:48 debug2: service_accept: ssh-userauth
10:20:48 debug1: SSH2_MSG_SERVICE_ACCEPT received
10:20:48 debug3: send packet: type 50
10:20:48 debug3: receive packet: type 51
10:20:48 debug1: Authentications that can continue: publickey
10:20:48 debug3: start over, passed a different list publickey
10:20:48 debug3: preferred publickey,keyboard-interactive,password
10:20:48 debug3: authmethod_lookup publickey
10:20:48 debug3: remaining preferred: keyboard-interactive,password
10:20:48 debug3: authmethod_is_enabled publickey
10:20:48 debug1: Next authentication method: publickey
10:20:48 debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
10:20:48 debug3: send packet: type 50
10:20:48 debug2: we sent a publickey packet, wait for reply
10:20:48 debug3: receive packet: type 60
10:20:48 debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
10:20:48 debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
10:20:48 debug3: sign_and_send_pubkey: signing using rsa-sha2-512
10:20:48 debug3: send packet: type 50
10:20:48 debug3: receive packet: type 51
10:20:48 Authenticated with partial success.
10:20:48 debug1: Authentications that can continue: keyboard-interactive
10:20:48 debug3: start over, passed a different list keyboard-interactive
10:20:48 debug3: preferred publickey,keyboard-interactive,password
10:20:48 debug3: authmethod_lookup keyboard-interactive
10:20:48 debug3: remaining preferred: password
10:20:48 debug3: authmethod_is_enabled keyboard-interactive
10:20:48 debug1: Next authentication method: keyboard-interactive
10:20:48 debug2: userauth_kbdint
10:20:48 debug3: send packet: type 50
10:20:48 debug2: we sent a keyboard-interactive packet, wait for reply
10:20:49 debug3: receive packet: type 60
10:20:49 debug2: input_userauth_info_req
10:20:49 debug2: input_userauth_info_req: num_prompts 1
10:20:49 debug1: read_passphrase: can't open /dev/tty: Device not configured
10:20:53 debug3: send packet: type 61
10:20:53 debug3: receive packet: type 60
10:20:53 debug2: input_userauth_info_req
10:20:53 debug2: input_userauth_info_req: num_prompts 0
10:20:53 debug3: send packet: type 61
10:20:53 debug3: receive packet: type 52
10:20:53 debug1: Authentication succeeded (keyboard-interactive).
10:20:53 Authenticated to bastion.example.com ([123.45.67.89]:22).
10:20:53 debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:14242
10:20:53 debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
10:20:53 debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
10:20:53 debug1: Local forwarding listening on ::1 port 8093.
10:20:53 debug2: fd 8 setting O_NONBLOCK
10:20:53 debug3: fd 8 is O_NONBLOCK
10:20:53 debug1: channel 0: new [port listener]
10:20:53 debug1: Local forwarding listening on 127.0.0.1 port 8093.
10:20:53 debug2: fd 9 setting O_NONBLOCK
10:20:53 debug3: fd 9 is O_NONBLOCK
10:20:53 debug1: channel 1: new [port listener]
10:20:53 debug2: fd 10 setting O_NONBLOCK
10:20:53 debug2: fd 11 setting O_NONBLOCK
10:20:53 debug2: fd 12 setting O_NONBLOCK
10:20:53 debug1: channel 2: new [client-session]
10:20:53 debug3: ssh_session2_open: channel_new: 2
10:20:53 debug2: channel 2: send open
10:20:53 debug3: send packet: type 90
10:20:53 debug1: Requesting no-more-sessions@openssh.com
10:20:53 debug3: send packet: type 80
10:20:53 debug1: Entering interactive session.
10:20:53 debug1: pledge: network
10:20:53 debug2: fd 13 setting O_NONBLOCK
10:20:53 debug2: fd 14 setting O_NONBLOCK
10:20:53 Connected
10:20:53 debug3: receive packet: type 80
10:20:53 debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
10:20:53 debug3: receive packet: type 91
10:20:53 debug2: channel_input_open_confirmation: channel 2: callback start
10:20:53 debug2: fd 5 setting TCP_NODELAY
10:20:53 debug3: ssh_packet_set_tos: set IP_TOS 0x48
10:20:53 debug2: client_session2_setup: id 2
10:20:53 debug2: channel 2: request pty-req confirm 1
10:20:53 debug3: send packet: type 98
10:20:53 debug1: Sending command: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
10:20:53 debug2: channel 2: request exec confirm 1
10:20:53 debug3: send packet: type 98
10:20:53 debug2: channel_input_open_confirmation: channel 2: callback done
10:20:53 debug2: channel 2: open confirm rwindow 0 rmax 32768
10:20:54 debug3: receive packet: type 99
10:20:54 debug2: channel_input_status_confirm: type 99 id 2
10:20:54 debug2: PTY allocation request accepted on channel 2
10:20:54 debug2: channel 2: rcvd adjust 2097152
10:20:54 debug3: receive packet: type 99
10:20:54 debug2: channel_input_status_confirm: type 99 id 2
10:20:54 debug2: exec request accepted on channel 2
10:20:56 debug3: receive packet: type 96
10:20:56 debug2: channel 2: rcvd eof
Connection to bastion.example.com closed.
10:20:56 debug2: channel 2: output open -> drain
10:20:56 debug3: receive packet: type 98
10:20:56 debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
10:20:56 debug3: receive packet: type 98
10:20:56 debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
10:20:56 debug2: channel 2: rcvd eow
10:20:56 debug2: channel 2: chan_shutdown_read (i0 o1 sock -1 wfd 10 efd 12 [write])
10:20:56 debug2: channel 2: input open -> closed
10:20:56 debug3: receive packet: type 97
10:20:56 debug2: channel 2: rcvd close
10:20:56 debug3: channel 2: will not send data after close
10:20:56 debug3: channel 2: will not send data after close
10:20:56 debug2: channel 2: obuf empty
10:20:56 debug2: channel 2: chan_shutdown_write (i3 o1 sock -1 wfd 11 efd 12 [write])
10:20:56 debug2: channel 2: output drain -> closed
10:20:56 debug2: channel 2: almost dead
10:20:56 debug2: channel 2: gc: notify user
10:20:56 debug2: channel 2: gc: user detached
10:20:56 debug2: channel 2: send close
10:20:56 debug3: send packet: type 97
10:20:56 debug2: channel 2: is dead
10:20:56 debug2: channel 2: garbage collecting
10:20:56 debug1: channel 2: free: client-session, nchannels 3
10:20:56 debug3: channel 2: status: The following connections are open:
  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/12 sock -1 cc -1)
10:20:56 debug3: send packet: type 1
10:20:56 debug1: channel 0: free: port listener, nchannels 2
10:20:56 debug3: channel 0: status: The following connections are open:
10:20:56 debug1: channel 1: free: port listener, nchannels 1
10:20:56 debug3: channel 1: status: The following connections are open:
10:20:56 debug1: fd 0 clearing O_NONBLOCK
10:20:56 debug3: fd 1 is not O_NONBLOCK
10:20:56 debug1: fd 2 clearing O_NONBLOCK
10:20:56 Transferred: sent 4060, received 12516 bytes, in 2.6 seconds
10:20:56 Bytes per second: sent 1584.4, received 4884.3
10:20:56 debug1: Exit status 255
10:20:56 Abnormal Disconnect
10:20:56 Connection failed, retry after 3sā€¦

(Yang.Y) #25

Dave, do you have upgraded Core Tunnel to version 1.7? I did some testes, and it won't break bastion connection on version 1.7.


(Dave Anderson) #26

I definitely have 1.7. Is there a new helper? I did not see any place to check the version of the helper.


(Dave Anderson) #27

Here is an export of the profile, sanitized hostnames of course.

2019-06-08 17:58:02.coressh copy (876 Bytes)


(Yang.Y) #28

Could you please change RemoteCommand to following value and try again?

bash --init-file <(ssh -L12445:localhost:8080 -N core-load-worker-example.com)

(Dave Anderson) #29

Progress! Connection succeeded but I was not able to use the tunnel successfully. The http://localhost:8093/api endpoint could not get any response. Connection completed around 11:03:29 and the attempt to use the tunnel occurred at about 11:03:46:


----------------------------------------
Equivalent Command: ssh -vvv -L 8093:localhost:14242 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 -o RemoteCommand="bash --init-file <(ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com)" ubuntu@bastion.example.com
11:03:23 Connectingā€¦
11:03:23 Using Core Helper 4.0 (r40)
11:03:23 OpenSSH_7.9p1, OpenSSL 1.0.2q  20 Nov 2018
11:03:23 debug1: Reading configuration data /Users/DA/.ssh/config
11:03:23 debug1: /Users/DA/.ssh/config line 1: Applying options for *
11:03:23 debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
11:03:23 debug3: expanding RemoteCommand: bash --init-file <(ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com)
11:03:23 debug3: expanded RemoteCommand: bash --init-file <(ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com)
11:03:23 debug2: resolving "bastion.example.com" port 22
11:03:23 debug2: ssh_connect_direct
11:03:23 debug1: Connecting to bastion.example.com [123.12.3.23] port 22.
11:03:23 debug1: Connection established.
11:03:23 debug1: identity file /Users/DA/.ssh/id_rsa type 0
11:03:23 debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_dsa type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_xmss type -1
11:03:23 debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
11:03:23 debug1: Local version string SSH-2.0-OpenSSH_7.9
11:03:23 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
11:03:23 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
11:03:23 debug2: fd 5 setting O_NONBLOCK
11:03:23 debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
11:03:23 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
11:03:23 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
11:03:23 Authenticatingā€¦
11:03:23 debug3: load_hostkeys: loaded 1 keys from bastion.example.com
11:03:23 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
11:03:23 debug3: send packet: type 20
11:03:23 debug1: SSH2_MSG_KEXINIT sent
11:03:24 debug3: receive packet: type 20
11:03:24 debug1: SSH2_MSG_KEXINIT received
11:03:24 debug2: local client KEXINIT proposal
11:03:24 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
11:03:24 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
11:03:24 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
11:03:24 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
11:03:24 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
11:03:24 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
11:03:24 debug2: compression ctos: none,zlib@openssh.com,zlib
11:03:24 debug2: compression stoc: none,zlib@openssh.com,zlib
11:03:24 debug2: languages ctos:
11:03:24 debug2: languages stoc:
11:03:24 debug2: first_kex_follows 0
11:03:24 debug2: reserved 0
11:03:24 debug2: peer server KEXINIT proposal
11:03:24 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
11:03:24 debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
11:03:24 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
11:03:24 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
11:03:24 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
11:03:24 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
11:03:24 debug2: compression ctos: none,zlib@openssh.com
11:03:24 debug2: compression stoc: none,zlib@openssh.com
11:03:24 debug2: languages ctos:
11:03:24 debug2: languages stoc:
11:03:24 debug2: first_kex_follows 0
11:03:24 debug2: reserved 0
11:03:24 debug1: kex: algorithm: curve25519-sha256@libssh.org
11:03:24 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
11:03:24 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
11:03:24 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
11:03:24 debug3: send packet: type 30
11:03:24 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
11:03:24 debug3: receive packet: type 31
11:03:24 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
11:03:24 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
11:03:24 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
11:03:24 debug3: load_hostkeys: loaded 1 keys from bastion.example.com
11:03:24 debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
11:03:24 debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
11:03:24 debug3: load_hostkeys: loaded 1 keys from 123.12.3.23
11:03:24 debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
11:03:24 debug1: Found key in /Users/DA/.ssh/known_hosts:7
11:03:24 debug3: send packet: type 21
11:03:24 debug2: set_newkeys: mode 1
11:03:24 debug1: rekey after 134217728 blocks
11:03:24 debug1: SSH2_MSG_NEWKEYS sent
11:03:24 debug1: expecting SSH2_MSG_NEWKEYS
11:03:24 debug3: receive packet: type 21
11:03:24 debug1: SSH2_MSG_NEWKEYS received
11:03:24 debug2: set_newkeys: mode 0
11:03:24 debug1: rekey after 134217728 blocks
11:03:24 debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
11:03:24 debug1: Will attempt key: /Users/DA/.ssh/id_dsa
11:03:24 debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
11:03:24 debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
11:03:24 debug1: Will attempt key: /Users/DA/.ssh/id_xmss
11:03:24 debug2: pubkey_prepare: done
11:03:24 debug3: send packet: type 5
11:03:24 debug3: receive packet: type 7
11:03:24 debug1: SSH2_MSG_EXT_INFO received
11:03:24 debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
11:03:24 debug3: receive packet: type 6
11:03:24 debug2: service_accept: ssh-userauth
11:03:24 debug1: SSH2_MSG_SERVICE_ACCEPT received
11:03:24 debug3: send packet: type 50
11:03:24 debug3: receive packet: type 51
11:03:24 debug1: Authentications that can continue: publickey
11:03:24 debug3: start over, passed a different list publickey
11:03:24 debug3: preferred publickey,keyboard-interactive,password
11:03:24 debug3: authmethod_lookup publickey
11:03:24 debug3: remaining preferred: keyboard-interactive,password
11:03:24 debug3: authmethod_is_enabled publickey
11:03:24 debug1: Next authentication method: publickey
11:03:24 debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
11:03:24 debug3: send packet: type 50
11:03:24 debug2: we sent a publickey packet, wait for reply
11:03:24 debug3: receive packet: type 60
11:03:24 debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
11:03:24 debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
11:03:24 debug3: sign_and_send_pubkey: signing using rsa-sha2-512
11:03:24 debug3: send packet: type 50
11:03:24 debug3: receive packet: type 51
11:03:24 Authenticated with partial success.
11:03:24 debug1: Authentications that can continue: keyboard-interactive
11:03:24 debug3: start over, passed a different list keyboard-interactive
11:03:24 debug3: preferred publickey,keyboard-interactive,password
11:03:24 debug3: authmethod_lookup keyboard-interactive
11:03:24 debug3: remaining preferred: password
11:03:24 debug3: authmethod_is_enabled keyboard-interactive
11:03:24 debug1: Next authentication method: keyboard-interactive
11:03:24 debug2: userauth_kbdint
11:03:24 debug3: send packet: type 50
11:03:24 debug2: we sent a keyboard-interactive packet, wait for reply
11:03:24 debug3: receive packet: type 60
11:03:24 debug2: input_userauth_info_req
11:03:24 debug2: input_userauth_info_req: num_prompts 1
11:03:24 debug1: read_passphrase: can't open /dev/tty: Device not configured
11:03:28 debug3: send packet: type 61
11:03:29 debug3: receive packet: type 60
11:03:29 debug2: input_userauth_info_req
11:03:29 debug2: input_userauth_info_req: num_prompts 0
11:03:29 debug3: send packet: type 61
11:03:29 debug3: receive packet: type 52
11:03:29 debug1: Authentication succeeded (keyboard-interactive).
11:03:29 Authenticated to bastion.example.com ([123.12.3.23]:22).
11:03:29 debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:14242
11:03:29 debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
11:03:29 debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
11:03:29 debug1: Local forwarding listening on ::1 port 8093.
11:03:29 debug2: fd 8 setting O_NONBLOCK
11:03:29 debug3: fd 8 is O_NONBLOCK
11:03:29 debug1: channel 0: new [port listener]
11:03:29 debug1: Local forwarding listening on 127.0.0.1 port 8093.
11:03:29 debug2: fd 9 setting O_NONBLOCK
11:03:29 debug3: fd 9 is O_NONBLOCK
11:03:29 debug1: channel 1: new [port listener]
11:03:29 debug2: fd 10 setting O_NONBLOCK
11:03:29 debug2: fd 11 setting O_NONBLOCK
11:03:29 debug2: fd 12 setting O_NONBLOCK
11:03:29 debug1: channel 2: new [client-session]
11:03:29 debug3: ssh_session2_open: channel_new: 2
11:03:29 debug2: channel 2: send open
11:03:29 debug3: send packet: type 90
11:03:29 debug1: Requesting no-more-sessions@openssh.com
11:03:29 debug3: send packet: type 80
11:03:29 debug1: Entering interactive session.
11:03:29 debug1: pledge: network
11:03:29 debug2: fd 13 setting O_NONBLOCK
11:03:29 debug2: fd 14 setting O_NONBLOCK
11:03:29 Connected
11:03:29 debug3: receive packet: type 80
11:03:29 debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
11:03:29 debug3: receive packet: type 91
11:03:29 debug2: channel_input_open_confirmation: channel 2: callback start
11:03:29 debug2: fd 5 setting TCP_NODELAY
11:03:29 debug3: ssh_packet_set_tos: set IP_TOS 0x48
11:03:29 debug2: client_session2_setup: id 2
11:03:29 debug2: channel 2: request pty-req confirm 1
11:03:29 debug3: send packet: type 98
11:03:29 debug1: Sending command: bash --init-file <(ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com)
11:03:29 debug2: channel 2: request exec confirm 1
11:03:29 debug3: send packet: type 98
11:03:29 debug2: channel_input_open_confirmation: channel 2: callback done
11:03:29 debug2: channel 2: open confirm rwindow 0 rmax 32768
11:03:29 debug3: receive packet: type 99
11:03:29 debug2: channel_input_status_confirm: type 99 id 2
11:03:29 debug2: PTY allocation request accepted on channel 2
11:03:29 debug2: channel 2: rcvd adjust 2097152
11:03:29 debug3: receive packet: type 99
11:03:29 debug2: channel_input_status_confirm: type 99 id 2
11:03:29 debug2: exec request accepted on channel 2
11:03:46 debug3: send packet: type 80
11:03:46 debug3: receive packet: type 82
11:03:50 debug1: Connection to port 8093 forwarding to localhost port 14242 requested.
11:03:50 debug2: fd 15 setting TCP_NODELAY
11:03:50 debug3: fd 15 is O_NONBLOCK
11:03:50 debug3: fd 15 is O_NONBLOCK
11:03:50 debug1: channel 3: new [direct-tcpip]
11:03:50 debug3: send packet: type 90
11:03:50 debug1: Connection to port 8093 forwarding to localhost port 14242 requested.
11:03:50 debug2: fd 16 setting TCP_NODELAY
11:03:50 debug3: fd 16 is O_NONBLOCK
11:03:50 debug3: fd 16 is O_NONBLOCK
11:03:50 debug1: channel 4: new [direct-tcpip]
11:03:50 debug3: send packet: type 90
11:03:50 debug3: receive packet: type 92
11:03:50 channel 3: open failed: connect failed: Connection refused
11:03:50 debug3: receive packet: type 92
11:03:50 channel 4: open failed: connect failed: Connection refused
11:03:50 debug2: channel 3: zombie
11:03:50 debug2: channel 3: garbage collecting
11:03:50 debug1: channel 3: free: direct-tcpip: listening port 8093 for localhost port 14242, connect from ::1 port 53039 to ::1 port 8093, nchannels 5
11:03:50 debug3: channel 3: status: The following connections are open:
  #2 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 10/11/12 sock -1 cc -1)
11:03:50 debug2: channel 4: zombie
11:03:50 debug2: channel 4: garbage collecting
11:03:50 debug1: channel 4: free: direct-tcpip: listening port 8093 for localhost port 14242, connect from ::1 port 53040 to ::1 port 8093, nchannels 4
11:03:50 debug3: channel 4: status: The following connections are open:
  #2 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 10/11/12 sock -1 cc -1)
11:04:05 debug3: send packet: type 80
11:04:05 debug3: receive packet: type 82
11:04:06 Disconnectingā€¦
11:04:06 Disconnected


(Yang.Y) #30

No new Core Helper was released, I tried hard to find out the real cause from log, but simply could not reproduce the issue on my Mac.

Could you please run this command in Terminal.app?

Equivalent Command: ssh -ttt -vvv -L 8093:localhost:14242 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 -o RemoteCommand="ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com" ubuntu@bastion.example.com

(Dave Anderson) #31

I appreciate you trying to help. I think we must be close, I am able to create a tunnel in the app -- it just does not tunnel traffic. :slight_smile: Below is the output from the command that you requested... also please check your direct messages.

Attempt 1:

Last login: Tue Jun 11 13:57:14 on ttys005
DA@DAs-MacBook-Pro ~  $ ssh -ttt -vvv -L 8093:localhost:14242 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 -o RemoteCommand="ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com" ubuntu@bastion.example.com
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/DA/.ssh/config
debug1: /Users/DA/.ssh/config line 1: Applying options for *
debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug3: expanding RemoteCommand: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
debug3: expanded RemoteCommand: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
debug1: Connecting to bastion.example.com port 22.
debug1: Connection established.
debug1: identity file /Users/DA/.ssh/id_rsa type 0
debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_dsa type -1
debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/DA/.ssh/id_xmss type -1
debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from 123.45.67.89
debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/DA/.ssh/known_hosts:7
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug1: Will attempt key: /Users/DA/.ssh/id_dsa
debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
debug1: Will attempt key: /Users/DA/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: send packet: type 61
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: send packet: type 61
debug3: receive packet: type 52
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to bastion.example.com ([123.45.67.89]:22).
debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:14242
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 8093.
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8093.
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug1: channel 2: new [client-session]
debug3: ssh_session2_open: channel_new: 2
debug2: channel 2: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env NVM_CD_FLAGS
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env Apple_PubSub_Socket_Render
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env NVM_DIR
debug3: Ignored env USER
debug3: Ignored env COMMAND_MODE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env FARKIN_JEFFROY_PW
debug3: Ignored env PATH
debug3: Ignored env FARKIN_USER_PW
debug3: Ignored env FARKIN_USER_PW_02
debug3: Ignored env FARKIN_USER_PW_03
debug3: Ignored env PWD
debug3: Ignored env FARKIN_USER_PW_06
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env FARKIN_USER_PW_07
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env FARKIN_USER_PW_04
debug3: Ignored env FARKIN_USER_PW_05
debug3: Ignored env FARKIN_USER_LOGIN
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PS1
debug3: Ignored env FARKIN_USER_PW_08
debug3: Ignored env FARKIN_USER_PW_09
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env LOGNAME
debug3: Ignored env NVM_BIN
debug3: Ignored env VAULT_USER
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env COLORTERM
debug3: Ignored env _
debug1: Sending command: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/ubuntu/.ssh/config
debug1: /home/ubuntu/.ssh/config line 1: Applying options for *.clari.io
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug1: /etc/ssh/ssh_config line 5: Deprecated option "useroaming"
debug2: resolving "core-load-worker.example.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to core-load-worker.example.com [10.9.8.7] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to core-load-worker.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:142
debug3: load_hostkeys: loaded 1 keys from core-load-worker.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug2: channel 2: rcvd eow
debug2: channel 2: chan_shutdown_read (i0 o1 sock -1 wfd 10 efd 12 [write])
debug2: channel 2: input open -> closed
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:vZyI380QQqjEfMDjLA+/WZEW0KAq5iCxOGiEDz40588
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:142
debug3: load_hostkeys: loaded 1 keys from core-load-worker.example.com
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/ubuntu/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.9.8.7
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: Host 'core-load-worker.example.com' is known and matches the RSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:142
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/ubuntu/.ssh/id_rsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_dsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ecdsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ubuntu/.ssh/id_rsa
debug3: no such identity: /home/ubuntu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_dsa
debug3: no such identity: /home/ubuntu/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa
debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519
debug3: no such identity: /home/ubuntu/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
debug3: channel 2: will not send data after close
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i3 o1 sock -1 wfd 11 efd 12 [write])
debug2: channel 2: output drain -> closed
debug2: channel 2: almost dead
debug2: channel 2: gc: notify user
debug2: channel 2: gc: user detached
debug2: channel 2: send close
debug3: send packet: type 97
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug3: channel 2: status: The following connections are open:
  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/12 sock -1 cc -1)

debug3: send packet: type 1
debug1: channel 0: free: port listener, nchannels 2
debug3: channel 0: status: The following connections are open:

debug1: channel 1: free: port listener, nchannels 1
debug3: channel 1: status: The following connections are open:

debug3: fd 1 is not O_NONBLOCK
Connection to bastion.example.com closed.
Transferred: sent 4128, received 12876 bytes, in 2.5 seconds
Bytes per second: sent 1651.2, received 5150.4
debug1: Exit status 255
DA@DAs-MacBook-Pro ~  $

(Dave Anderson) #32
=========================
Attempt 2:


DA@DAs-MacBook-Pro ~  $ ssh -tt -vvv -L 8093:localhost:14242 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 -o RemoteCommand="ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com" ubuntu@bastion.example.com
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/DA/.ssh/config
debug1: /Users/DA/.ssh/config line 1: Applying options for *
debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug3: expanding RemoteCommand: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
debug3: expanded RemoteCommand: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
debug1: Connecting to bastion.example.com port 22.
debug1: Connection established.
debug1: identity file /Users/DA/.ssh/id_rsa type 0
debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_dsa type -1
debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/DA/.ssh/id_xmss type -1
debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from 123.45.67.89
debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/DA/.ssh/known_hosts:7
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug1: Will attempt key: /Users/DA/.ssh/id_dsa
debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
debug1: Will attempt key: /Users/DA/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: send packet: type 61
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: send packet: type 61
debug3: receive packet: type 52
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to bastion.example.com ([123.45.67.89]:22).
debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:14242
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 8093.
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8093.
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug1: channel 2: new [client-session]
debug3: ssh_session2_open: channel_new: 2
debug2: channel 2: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env NVM_CD_FLAGS
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env Apple_PubSub_Socket_Render
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env NVM_DIR
debug3: Ignored env USER
debug3: Ignored env COMMAND_MODE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env FARKIN_JEFFROY_PW
debug3: Ignored env PATH
debug3: Ignored env FARKIN_USER_PW
debug3: Ignored env FARKIN_USER_PW_02
debug3: Ignored env FARKIN_USER_PW_03
debug3: Ignored env PWD
debug3: Ignored env FARKIN_USER_PW_06
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env FARKIN_USER_PW_07
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env FARKIN_USER_PW_04
debug3: Ignored env FARKIN_USER_PW_05
debug3: Ignored env FARKIN_USER_LOGIN
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PS1
debug3: Ignored env FARKIN_USER_PW_08
debug3: Ignored env FARKIN_USER_PW_09
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env LOGNAME
debug3: Ignored env NVM_BIN
debug3: Ignored env VAULT_USER
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env COLORTERM
debug3: Ignored env _
debug1: Sending command: ssh -vvv -L14242:localhost:8080 -N core-load-worker.example.com
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/ubuntu/.ssh/config
debug1: /home/ubuntu/.ssh/config line 1: Applying options for *.clari.io
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug1: /etc/ssh/ssh_config line 5: Deprecated option "useroaming"
debug2: resolving "core-load-worker.example.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to core-load-worker.example.com [10.9.8.7] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to core-load-worker.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:142
debug3: load_hostkeys: loaded 1 keys from core-load-worker.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:vZyI380QQqjEfMDjLA+/WZEW0KAq5iCxOGiEDz40588
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:142
debug3: load_hostkeys: loaded 1 keys from core-load-worker.example.com
debug3: hostkeys_foreach: reading file "/home/ubuntu/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/ubuntu/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.9.8.7
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: Host 'core-load-worker.example.com' is known and matches the RSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:142
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/ubuntu/.ssh/id_rsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_dsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ecdsa ((nil))
debug2: key: /home/ubuntu/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ubuntu/.ssh/id_rsa
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug2: channel 2: rcvd eow
debug2: channel 2: chan_shutdown_read (i0 o1 sock -1 wfd 10 efd 12 [write])
debug2: channel 2: input open -> closed
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug3: no such identity: /home/ubuntu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_dsa
debug3: no such identity: /home/ubuntu/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa
debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519
debug3: no such identity: /home/ubuntu/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
debug3: channel 2: will not send data after close
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i3 o1 sock -1 wfd 11 efd 12 [write])
debug2: channel 2: output drain -> closed
debug2: channel 2: almost dead
debug2: channel 2: gc: notify user
debug2: channel 2: gc: user detached
debug2: channel 2: send close
debug3: send packet: type 97
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug3: channel 2: status: The following connections are open:
  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/12 sock -1 cc -1)

debug3: send packet: type 1
debug1: channel 0: free: port listener, nchannels 2
debug3: channel 0: status: The following connections are open:

debug1: channel 1: free: port listener, nchannels 1
debug3: channel 1: status: The following connections are open:

debug3: fd 1 is not O_NONBLOCK
Connection to bastion.example.com closed.
Transferred: sent 4128, received 12848 bytes, in 2.5 seconds
Bytes per second: sent 1664.7, received 5181.2
debug1: Exit status 255
DA@DAs-MacBook-Pro ~  $


(Dave Anderson) #33

I was going to post this in a PM/DM but it doesn't seem sensitive once I changed the hostnames. Maybe this will help understand what is going on:

Here is /home/ubuntu/.ssh/config(bastion.example.com):

Host *.example.com
User ubuntu
ServerAliveInterval 10
ServerAliveCountMax 120
ForwardAgent yes

Interesting files from /etc/ssh:

ubuntu@bastion-production-1:/etc/ssh$ cat ssh_config
# This file was generated by Chef for bastion-production-1.example.com
# Do NOT modify this file by hand!

Host *
UseRoaming no
ubuntu@bastion-production-1:/etc/ssh$ cat sshd_config
# This file was generated by Chef for bastion-production-1.example.com
# Do NOT modify this file by hand!

ChallengeResponseAuthentication yes
UsePAM yes
AcceptEnv LANG LC_*
AuthenticationMethods publickey,keyboard-interactive
ForceCommand /etc/ssh/ssh_wrapper.sh
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_rsa_key
HostbasedAuthentication no
IgnoreRhosts yes
KeyRegenerationInterval 3600
LogLevel VERBOSE
LoginGraceTime 120
PasswordAuthentication no
PermitEmptyPasswords no
PermitRootLogin no
PermitUserEnvironment yes
Port 22
PrintLastLog yes
PrintMotd no
Protocol 2
PubkeyAuthentication yes
RsaAuthentication yes
RhostsRsaauthentication no
ServerKeyBits 768
StrictModes yes
Subsystem sftp /usr/lib/openssh/sftp-server
SyslogFacility AUTH
TcpKeepAlive yes
UsePam yes
UsePrivilegeSeparation yes
X11DisplayOffset 10
X11Forwarding yes

ubuntu@bastion-production-1:/etc/ssh$

This is the wrapper referenced above:

ubuntu@bastion-production-1:/etc/ssh$ cat ssh_wrapper.sh
#!/bin/bash

# Exec the login reporter script
pid=$$
bash -c "exec /etc/ssh/ssh_logger.rb $pid >/dev/null 2>&1" &

# If user is has tunnel access only, exit without invoking more commands
if [ "$TUNNEL_ONLY" == "true" ]; then
  echo "This key can only be used for ssh tunnels."
  exit
fi

# sshd sets the SSH_ORIGINAL_COMMAND IF ForceCommand is set in the sshd_config
# (which it must be, or this script wouldn't get called) AND the ssh client is
# passing a specific command (e.g. ssh host some_command_to_run)
#
# If there is no command, we just exec a login shell
if [ "$SSH_ORIGINAL_COMMAND" == "" ]; then
  exec bash -l
else
  # This is annoying, but the ssh_logger script needs this process to stick
  # around a second or two or it fails to log
  sleep 2
  exec bash -c "${SSH_ORIGINAL_COMMAND}"
fi

(Yang.Y) #34

So, the command also failed on Terminal.app?

Could you please try again with this command in Terminal.app?

ssh -ttt -vvv -L 8093:localhost:14242 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 ubuntu@bastion.example.com ssh -L 14242:localhost:8080 -N core-load-worker-example.com

(Dave Anderson) #35

The prior command succeeded in appearing to have a tunnel open; it just did not tunnel traffic. This latest command that you posted ~1 hr ago did not hold the connection open, it ended back at my local shell prompt with no intervention from me or attempt to send traffic through it. Here is the full output of that command:

Last login: Wed Jun 12 14:47:53 on ttys012
DA@DAs-MacBook-Pro ~  $ ssh -ttt -vvv -L 8093:localhost:14242 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 ubuntu@bastion.example.com ssh -L 14242:localhost:8080 -N core-load-worker.example.com
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/DA/.ssh/config
debug1: /Users/DA/.ssh/config line 1: Applying options for *
debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to bastion.example.com port 22.
debug1: Connection established.
debug1: identity file /Users/DA/.ssh/id_rsa type 0
debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_dsa type -1
debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/DA/.ssh/id_xmss type -1
debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from 123.45.67.89
debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/DA/.ssh/known_hosts:7
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug1: Will attempt key: /Users/DA/.ssh/id_dsa
debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
debug1: Will attempt key: /Users/DA/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: send packet: type 61
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: send packet: type 61
debug3: receive packet: type 52
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to bastion.example.com ([123.45.67.89]:22).
debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:14242
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 8093.
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8093.
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug1: channel 2: new [client-session]
debug3: ssh_session2_open: channel_new: 2
debug2: channel 2: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env NVM_CD_FLAGS
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env Apple_PubSub_Socket_Render
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env NVM_DIR
debug3: Ignored env USER
debug3: Ignored env COMMAND_MODE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env FARKIN_JEFFROY_PW
debug3: Ignored env PATH
debug3: Ignored env FARKIN_USER_PW
debug3: Ignored env FARKIN_USER_PW_02
debug3: Ignored env FARKIN_USER_PW_03
debug3: Ignored env PWD
debug3: Ignored env FARKIN_USER_PW_06
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env FARKIN_USER_PW_07
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env FARKIN_USER_PW_04
debug3: Ignored env FARKIN_USER_PW_05
debug3: Ignored env FARKIN_USER_LOGIN
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PS1
debug3: Ignored env FARKIN_USER_PW_08
debug3: Ignored env FARKIN_USER_PW_09
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env LOGNAME
debug3: Ignored env NVM_BIN
debug3: Ignored env VAULT_USER
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env COLORTERM
debug3: Ignored env _
debug1: Sending command: ssh -L 14242:localhost:8080 -N core-load-worker.example.com
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
Permission denied (publickey).
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i0 o1 sock -1 wfd 11 efd 12 [write])
debug2: channel 2: output drain -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug2: channel 2: rcvd eow
debug2: channel 2: chan_shutdown_read (i0 o3 sock -1 wfd 10 efd 12 [write])
debug2: channel 2: input open -> closed
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug2: channel 2: almost dead
debug2: channel 2: gc: notify user
debug2: channel 2: gc: user detached
debug2: channel 2: send close
debug3: send packet: type 97
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug3: channel 2: status: The following connections are open:
  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/12 sock -1 cc -1)

debug3: send packet: type 1
debug1: channel 0: free: port listener, nchannels 2
debug3: channel 0: status: The following connections are open:

debug1: channel 1: free: port listener, nchannels 1
debug3: channel 1: status: The following connections are open:

debug3: fd 1 is not O_NONBLOCK
Connection to bastion.example.com closed.
Transferred: sent 4128, received 3176 bytes, in 2.6 seconds
Bytes per second: sent 1558.0, received 1198.7
debug1: Exit status 255
DA@DAs-MacBook-Pro ~  $

(Yang.Y) #36

Thanks a lot for the testing, could you please try again with this command?

ssh -ttt -vvv -L 8093:localhost:14242 ubuntu@bastion.example.com ssh -L 14242:localhost:8080 -N core-load-worker-example.com

(Dave Anderson) #37

You're welcome, happy to work toward a solution. This latest command also exited on its own. Again, it did not seem to look for the keys on bastion:

Last login: Wed Jun 12 20:32:57 on ttys014
DA@DAs-MacBook-Pro ~  $ ssh -ttt -vvv -L 8093:localhost:14242 ubuntu@bastion.example.com ssh -L 14242:localhost:8080 -N core-load-worker.example.com
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/DA/.ssh/config
debug1: /Users/DA/.ssh/config line 1: Applying options for *
debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to bastion.example.com port 22.
debug1: Connection established.
debug1: identity file /Users/DA/.ssh/id_rsa type 0
debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_dsa type -1
debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/DA/.ssh/id_xmss type -1
debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from 123.45.67.89
debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/DA/.ssh/known_hosts:7
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug1: Will attempt key: /Users/DA/.ssh/id_dsa
debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
debug1: Will attempt key: /Users/DA/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: send packet: type 61
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: send packet: type 61
debug3: receive packet: type 52
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to bastion.example.com ([123.45.67.89]:22).
debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:14242
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 8093.
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8093.
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug1: channel 2: new [client-session]
debug3: ssh_session2_open: channel_new: 2
debug2: channel 2: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env NVM_CD_FLAGS
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env Apple_PubSub_Socket_Render
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env NVM_DIR
debug3: Ignored env USER
debug3: Ignored env COMMAND_MODE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env FARKIN_JEFFROY_PW
debug3: Ignored env PATH
debug3: Ignored env FARKIN_USER_PW
debug3: Ignored env FARKIN_USER_PW_02
debug3: Ignored env FARKIN_USER_PW_03
debug3: Ignored env PWD
debug3: Ignored env FARKIN_USER_PW_06
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env FARKIN_USER_PW_07
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env FARKIN_USER_PW_04
debug3: Ignored env FARKIN_USER_PW_05
debug3: Ignored env FARKIN_USER_LOGIN
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PS1
debug3: Ignored env FARKIN_USER_PW_08
debug3: Ignored env FARKIN_USER_PW_09
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env LOGNAME
debug3: Ignored env NVM_BIN
debug3: Ignored env VAULT_USER
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env COLORTERM
debug3: Ignored env _
debug1: Sending command: ssh -L 14242:localhost:8080 -N core-load-worker.example.com
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
Permission denied (publickey).
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i0 o1 sock -1 wfd 11 efd 12 [write])
debug2: channel 2: output drain -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug2: channel 2: rcvd eow
debug2: channel 2: chan_shutdown_read (i0 o3 sock -1 wfd 10 efd 12 [write])
debug2: channel 2: input open -> closed
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug2: channel 2: almost dead
debug2: channel 2: gc: notify user
debug2: channel 2: gc: user detached
debug2: channel 2: send close
debug3: send packet: type 97
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug3: channel 2: status: The following connections are open:
  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/12 sock -1 cc -1)

debug3: send packet: type 1
debug1: channel 0: free: port listener, nchannels 2
debug3: channel 0: status: The following connections are open:

debug1: channel 1: free: port listener, nchannels 1
debug3: channel 1: status: The following connections are open:

debug3: fd 1 is not O_NONBLOCK
Connection to bastion.example.com closed.
Transferred: sent 4128, received 3176 bytes, in 2.5 seconds
Bytes per second: sent 1632.6, received 1256.1
debug1: Exit status 255
DA@DAs-MacBook-Pro ~  $

(Yang.Y) #38

Well, I guess I find out the reason, please try again:

ssh -ttt -A -vvv -L 8093:localhost:14242 ubuntu@bastion.example.com ssh -L 14242:localhost:8080 -N core-load-worker-example.com

(Dave Anderson) #39

OK, that worked! I was able to successfully get my expected response from the worker endpoint. I marked the spot in the log below where it had finished establishing the tunnel(look for '<----'); after that the output was the result of my using the tunnel.

Last login: Thu Jun 13 00:37:04 on ttys016
DA@DAs-MacBook-Pro ~  $ ssh -ttt -A -vvv -L 8093:localhost:14242 ubuntu@bastion.example.com ssh -L 14242:localhost:8080 -N core-load-worker.example.com
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/DA/.ssh/config
debug1: /Users/DA/.ssh/config line 1: Applying options for *
debug1: /Users/DA/.ssh/config line 4: Deprecated option "useroaming"
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to bastion.example.com port 22.
debug1: Connection established.
debug1: identity file /Users/DA/.ssh/id_rsa type 0
debug1: identity file /Users/DA/.ssh/id_rsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_dsa type -1
debug1: identity file /Users/DA/.ssh/id_dsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa type -1
debug1: identity file /Users/DA/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/DA/.ssh/id_ed25519 type -1
debug1: identity file /Users/DA/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/DA/.ssh/id_xmss type -1
debug1: identity file /Users/DA/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to bastion.example.com:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oiy...wfY
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from bastion.example.com
debug3: hostkeys_foreach: reading file "/Users/DA/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/DA/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from 123.45.67.89
debug1: Host 'bastion.example.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/DA/.ssh/known_hosts:7
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug1: Will attempt key: /Users/DA/.ssh/id_dsa
debug1: Will attempt key: /Users/DA/.ssh/id_ecdsa
debug1: Will attempt key: /Users/DA/.ssh/id_ed25519
debug1: Will attempt key: /Users/DA/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/DA/.ssh/id_rsa RSA SHA256:3sf...HWE agent
debug3: sign_and_send_pubkey: RSA SHA256:3sf...HWE
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 51
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: send packet: type 61
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: send packet: type 61
debug3: receive packet: type 52
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to bastion.example.com ([123.45.67.89]:22).
debug1: Local connections to LOCALHOST:8093 forwarded to remote address localhost:14242
debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
debug3: sock_set_v6only: set socket 8 IPV6_V6ONLY
debug1: Local forwarding listening on ::1 port 8093.
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 8093.
debug2: fd 9 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 1: new [port listener]
debug1: channel 2: new [client-session]
debug3: ssh_session2_open: channel_new: 2
debug2: channel 2: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 2: request auth-agent-req@openssh.com confirm 0
debug3: send packet: type 98
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env NVM_CD_FLAGS
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env Apple_PubSub_Socket_Render
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env NVM_DIR
debug3: Ignored env USER
debug3: Ignored env COMMAND_MODE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env FARKIN_JEFFROY_PW
debug3: Ignored env PATH
debug3: Ignored env FARKIN_USER_PW
debug3: Ignored env FARKIN_USER_PW_02
debug3: Ignored env FARKIN_USER_PW_03
debug3: Ignored env PWD
debug3: Ignored env FARKIN_USER_PW_06
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env FARKIN_USER_PW_07
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env FARKIN_USER_PW_04
debug3: Ignored env FARKIN_USER_PW_05
debug3: Ignored env FARKIN_USER_LOGIN
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PS1
debug3: Ignored env FARKIN_USER_PW_08
debug3: Ignored env FARKIN_USER_PW_09
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env LOGNAME
debug3: Ignored env NVM_BIN
debug3: Ignored env VAULT_USER
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env COLORTERM
debug3: Ignored env _
debug1: Sending command: ssh -L 14242:localhost:8080 -N core-load-worker.example.com
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug2: fd 13 setting O_NONBLOCK
debug3: fd 13 is O_NONBLOCK
debug1: channel 3: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug3: send packet: type 91 <---- Last output before I tested tunnel
debug1: Connection to port 8093 forwarding to localhost port 14242 requested.
debug2: fd 14 setting TCP_NODELAY
debug3: fd 14 is O_NONBLOCK
debug3: fd 14 is O_NONBLOCK
debug1: channel 4: new [direct-tcpip]
debug3: send packet: type 90
debug1: Connection to port 8093 forwarding to localhost port 14242 requested.
debug2: fd 15 setting TCP_NODELAY
debug3: fd 15 is O_NONBLOCK
debug3: fd 15 is O_NONBLOCK
debug1: channel 5: new [direct-tcpip]
debug3: send packet: type 90
debug3: receive packet: type 91
debug2: channel 4: open confirm rwindow 2097152 rmax 32768
debug3: receive packet: type 91
debug2: channel 5: open confirm rwindow 2097152 rmax 32768


(Yang.Y) #40

I'm sorry for my fault, forgot that Agent Forwarding must be enabled for your case:

image

Please enable Forward Agent and make sure your bastion host is configured properly:
image

Then set its RemoteCommand option to:

ssh -L12445:localhost:8080 -N core-load-worker-example.com

image