Disconnect does not seem to function

[bmorgenthaler]

Running Core Tunnel 3.0.1 on macOS 11.5.2 (non m1) and I'm noticing issues with Disconnects.

1. Disconnect just doesn't work. Pressing Disconnect in the Main Window on a tunnel doesn't stop it. Selecting Disconnect All from the tray menu doesn't disconnect anything. The only way I've found to disconnect anything is to quite Core Tunnel and then kill any of the io.coressh processes.
2. I'm assuming it's tied into the disconnect issue but when my system sleeps and wakes, the ssh tunnels say they are connected but I know they can't be because one of the connects if actually disconnected requires me to enter an OTP to reestablish the connection.

[yang (Yang.Y)]

This is a known issue, caused by multiplexing function, and I'm setting out to fix it.

Thank you for reporting.

Yang

[yang (Yang.Y)]

We just released Core Tunnel 3.5 Beta, and invited you join our Beta Program.

Core Tunnel 3.5 beta now gains full support for multiplexing directives (i.e. ControlPath and ControlMaster).

Please let me know if you find any bugs on multiplexing support.

Thanks you very much,

Yang

[bmorgenthaler]

I just downloaded Core Tunnel 3.5 Beta, I quit Core Tunnel and then started Core Tunnel (Beta) and everything was empty. Do I need to recreate my configs in the Beta app? Also is there a helper side app like there is for Core Tunnel?

[bmorgenthaler]

Okay I stopped the helper app and tried to setup a new connection based upon a name in my ~/.ssh/config file. It connected and I can see the ControlMaster socket created under ~/.ssh like I've specified and other connections utilizing it function.

However looking at the log for the connection in Core Tunnel (Beta) I see the following:
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

Equivalent Command: ssh -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 hub_external
15:14:39 Connecting…
15:14:39 Using Embeded Helper 6.0 (r3257)
15:14:40 Authenticating…
15:14:41 Connected
15:14:42 [mux] bind [127.0.0.1]:28080: Address already in use
15:14:42 [mux] channel_setup_fwd_listener_tcpip: cannot listen to port: 28080
15:14:42 [mux] bind [127.0.0.1]:33394: Address already in use
15:14:42 [mux] channel_setup_fwd_listener_tcpip: cannot listen to port: 33394
15:14:42 [mux] Could not request local forwarding.
15:15:10 Abnormal Disconnect
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
Equivalent Command: ssh -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -o ServerAliveCountMax=3 hub_external
15:15:33 Connecting…
15:15:33 Using Embeded Helper 6.0 (r3257)
15:15:33 Authenticating…
15:15:34 Connected
15:15:35 [mux] bind [127.0.0.1]:28080: Address already in use
15:15:35 [mux] channel_setup_fwd_listener_tcpip: cannot listen to port: 28080
15:15:35 [mux] bind [127.0.0.1]:33394: Address already in use
15:15:35 [mux] channel_setup_fwd_listener_tcpip: cannot listen to port: 33394
15:15:35 [mux] Could not request local forwarding.

When I clicked disconnect the green connection indicator finally went red and then displayed [mux] Could not request local forwarding. I checked with netstat and lsof and those ports are open and no process was accessing them before connecting the tunnel so I don't know why they aren't available.

[bmorgenthaler]

When I manually setup a tunnel and duplicate what I have in my ~/.ssh/config file I'm still seeing the same issue:

15:23:48 [mux] bind [127.0.0.1]:28080: Address already in use
15:23:48 [mux] channel_setup_fwd_listener_tcpip: cannot listen to port: 28080
15:23:48 [mux] Could not request local forwarding.

However it continues to connect even though I have set ExitOnForwardFailure to yes.

Another bit I just noticed is that ControlPersist only has a yes/no option but per the man pages it also allows numerical values to specific how long the connection should persist.

[bmorgenthaler]

3rd update from testing.

I pointed at a bogus empty ssh config file and then manually setup my controlmaster tunnel, here is the equivalent command:

ssh -i ~/.ssh/id_rsa -o TCPKeepAlive=yes -o ControlPath=~/.ssh/sockets/@%h-%p -o PubkeyAuthentication=yes -o HostKeyAlias=jumphost -o ExitOnForwardFailure=yes -o ControlPersist=yes -o ServerAliveCountMax=3 -o ControlMaster=yes -o ServerAliveInterval=15 -o StrictHostKeyChecking=yes bmorgenthaler@aaa.bbb.ccc.ddd

It appears to connect and I see the socket created and can use it with other connections but now the GUI shows a green connected indicator BUT the "Connect" button didn't change to "Disconnect" and the log shows:

Equivalent Command: ssh -i ~/.ssh/id_rsa -o TCPKeepAlive=yes -o ControlPath=~/.ssh/sockets/@%h-%p -o PubkeyAuthentication=yes -o HostKeyAlias=jumphost -o ExitOnForwardFailure=yes -o ControlPersist=yes -o ServerAliveCountMax=3 -o ControlMaster=yes -o ServerAliveInterval=15 -o StrictHostKeyChecking=yes bmorgenthaler@aaa.bbb.ccc.ddd
15:36:09 Connecting…
15:36:09 Using Embeded Helper 6.0 (r3257)
15:36:09 Authenticating…
15:36:11 Connected
15:36:12 [mux] ControlSocket /Users/bmorgenthaler/.ssh/sockets/@aaa.bbb.ccc.ddd-22 already exists, disabling multiplexing

[nigelm (Nigel Metheringham)]

I've seen similar problems since the 11.5.2 update - typically manifesting as SSH or tunnel port forwardings failing because a previous login has left the tunnel in a zombie state.

Nigel

[bmorgenthaler]

In testing the beta I made sure all processes were kill and ports available. So no zombie processes, sockets, etc. were present (unless they were hiding from ls, lsof, ps, etc.)

[yang (Yang.Y)]

Do I need to recreate my configs in the Beta app?

Yes you have recreate the configs or just copy them from public release, here has a how-to for drag & drop pasting:

Codinn Beta Program

Welcome to the Codinn Beta Program. This post explains how to install the betas and report problems. Thanks for helping us test our apps and make the next releases of Codinn apps our best yet. Three steps to becoming a Beta Program member The betas of Codinn apps are distributed via TestFlight. When you apply the beta program you’ll automatically be emailed a TestFlight invitation. Apply for Codinn Beta Program. You receive an invitation from TestFlight to try it out. Open this invitation…

Also is there a helper side app like there is for Core Tunnel?

A Core Helper is embedded in beta app so you don't have to download or upgrade Core Helper.

When I clicked disconnect the green connection indicator finally went red and then displayed [mux] Could not request local forwarding.

Could you please share the part of your ~/.ssh/config which can reproduce the issue? I'll investigate this deeper.

Another bit I just noticed is that ControlPersist only has a yes/no option but per the man pages it also allows numerical values to specific how long the connection should persist.

Good catch ControlPersist is useless in Core Shell/ Tunnel, since both apps has no dedicate ssh process to keep a master connection, the master ssh connection closed along with the tunnel or shell terminal that act as the master connection.

ControlPersist is useful in command line, ssh command run master ssh process in background (daemon) to hold the master connection.

So maybe we should remove ControlPersist option.

GUI shows a green connected indicator BUT the "Connect" button didn't change to "Disconnect"

Will investigate this issue, thank you for the connection log.

Yang

[bmorgenthaler]

@yang

Could you please share the part of your ~/.ssh/config which can reproduce the issue? I'll investigate this deeper.

I work with multiple environments so I use nested ssh configs through Include statements but here is a sanitized version that I tested with.

~/.ssh/config

Include ~/.ssh/config.d/jumphosts.config
Host *
    ForwardAgent                no
    ForwardX11                  no
    ForwardX11Trusted           no
    ServerAliveInterval         30
    ServerAliveCountMax         20
    Compression                 no
    HashKnownHosts              no
    LogLevel                    ERROR
    TCPKeepAlive                yes
    PasswordAuthentication      yes
    BatchMode                   no
    CheckHostIP                 yes
    EscapeChar                  ~
    GSSAPIAuthentication        no
    PermitLocalCommand          yes
    ExitOnForwardFailure        yes

    StrictHostKeyChecking       no
    UserKnownHostsFile          /dev/null

    ControlMaster               no
    ControlPath                 ~/.ssh/sockets/%r@%h-%p
    ControlPersist              300

    # OSX Specific Crap
    UseKeychain                 yes

    # SSH Identities
    AddKeysToAgent              yes
    IdentitiesOnly              yes
    PubkeyAuthentication        no

~/.ssh/config.d/jumphosts.config

## External Jump Host
Host jumphost
    HostName AAA.BBB.CCC.DDD
    HostKeyAlias jumphost
    User bmorgenthaler
    StrictHostKeyChecking yes
    UserKnownHostsFile ~/.ssh/known_hosts
    PubkeyAuthentication yes
    IdentityFile ~/.ssh/id_rsa
    DynamicForward 127.0.0.1:18080
    LocalForward localhost:13389 desktop.local:3389
    ControlMaster yes

## Internal Jump Host
Host jump_mgmt
    HostName EEE.FFF.GGG.HHH
    HostKeyAlias jump_mgmt
    User bmorgenthlaer
    StrictHostKeyChecking yes
    UserKnownHostsFile ~/.ssh/known_hosts
    PubkeyAuthentication yes
    IdentityFile ~/.ssh/id_rsa2
    DynamicForward 127.0.0.1:28080
    ProxyJump bmorgenthaler@AAA.BBB.CCC.DDD

[yang (Yang.Y)]

Thank you very much, will investigate it ASAP.

[yang (Yang.Y)]

Hi,

This issue has been fixed in fresh baked Core Tunnel 3.5 Beta 2, please let me know if you still run into this problem.

Thank you very much for helping.

Yang

[bmorgenthaler]

So far testing of my controlmaster setup with double jumphosts hasn't shown any more issues with 3.5 Beta 2. Thanks!