Opensc-pkcs11.so

I am running macOS 14.5 on an M2 Studio.

I have installed opensc-pkcs11 using Homebrew. I specifically need this to access a private key on a hardware device (Nitrokey). I am able to successfully access the private key on the Nitrokey using openssh by specifiying "PKCS11Provider /Library/OpenSC/lib/opensc-pkcs11.so" in my ~/.ssh/config. I currently use iTerm2 for my default terminal.

However, when I specify the PKCS11Provider to this same value in Core Tunnel or Core Shell (or just let it be read from my ~/.ssh/config), I receive an error.

16:45:26 dlopen /Library/OpenSC/lib/opensc-pkcs11.so failed: dlopen(/Library/OpenSC/lib/opensc-pkcs11.so, 0x0002): tried: '/Library/OpenSC/lib/opensc-pkcs11.so' (code signature in <41F8E9B6-0716-3993-83CC-A985568638A4> '/Library/OpenSC/lib/opensc-pkcs11.so' not valid for use in process: mapping process and mapped file (non-platform) have different Team IDs), '/System/Volumes/Preboot/Cryptexes/OS/Library/OpenSC/lib/opensc-pkcs11.so' (no such file), '/Library/OpenSC/lib/opensc-pkcs11.so' (code signature in <41F8E9B6-0716-3993-83CC-A985568638A4> '/Library/OpenSC/lib/opensc-pkcs11.so' not valid for use in process: mapping process and mapped file (non-platform) have different Team IDs)

This reads to me like a security feature disallowing two signed (?) files from two different applications to communicate. This strikes me as odd as I am using opensc-pkcs11.so and openssh, both installed from Homebrew.

Am I missing something? Am I doing this incorrectly or is there a workaround?

Thanks!

Thanks a lot for reporting issue, version 3.8.8 is released for fix the problem, please do update let me know if this version not function.

Kindly Regards,

Yang

Thanks for the update.

It appears that pkcs11.so is now accessed and communicates successfully! However, it finds no keys on the hardware device.

11:43:23 debug1: Connection established.
11:43:24 debug1: provider /Library/OpenSC/lib/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.25
11:43:25 debug1: pkcs11_register_provider: provider /Library/OpenSC/lib/opensc-pkcs11.so returned no slots

I receive a dialog for Password Authentication in Core Tunnel/Shell, but do not receive a dialog to interact with the hardware device (PIN entry or Multiplex Autoask Yes/No). I suspect this is preventing the hardware key from returning slot (key) information.

Thanks!

Could you please try again with this revised version:

Core Shell 3.8.8-PKCS11.zip

Kindly Regards,

Yang

BTW, does this work if run the equivalent command in Core Shell?

I tested in both Tunnel and Shell with the same result.

I tried the updated Core Shell binary in the zip file. Same issue, no dialog to enter PIN or Multiplex Autoask yes/no.

13:00:39 debug1: Connection established.
13:00:40 debug1: provider /Library/OpenSC/lib/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.25
13:00:41 debug1: pkcs11_register_provider: provider /Library/OpenSC/lib/opensc-pkcs11.so returned no slots

Do you try paste the ssh command to a Local shell of Core Shell? Does it work?

Yang

Please give the second revise version a try:
Core Shell 3.8.8-PKCS11-2.zip

Thank you,

Yang

Yes, if I launch 'Local Default' terminal in Core Shell and then run the command, it is successful.

ssh -v -o ServerAliveCountMax=3 -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -p XX example
...
debug1: Offering public key: cardno:XXXX ED25519 SHA256:XXXX agent
debug1: Server accepts key: cardno:XXXX ED25519 SHA256:XXXX agent
...
debug1: auto-mux: Trying existing master at '/Users/uthyr/.ssh/jump.sock'
debug1: mux_client_request_session: master session id: 4

The second revision is still unsuccessful. Now I am wondering if my dot files are being sourced to create a login environment which includes communication with my running ssh-agent?

Could you please send me desensitized verbose connection logs? Both success and failure logs, these logs could help me identified problem.

Thank you,

Yang

Well, I just got a YubiKey 4, will update this thread after the issue is reproduced.

Yang

I did try YubiKey 4 with Core Tunnel version 3.8.8 (53E56), and seemed every thing works. Here are the steps how I use YubiKey for testing:

  1. Reset then initialize the YubiKey, follow steps in Authenticating SSH with PIV and PKCS#11 (client) to create keys.

  2. Install OpenSC and create a tunnel with PKCS11Provider directive set to /Library/OpenSC/lib/opensc-pkcs11.so

  3. Connect the tunnel, and it shows using the YubiKey for authentication.

The PIN dialog:

image

Full connection log:

## 127.0.0.11 – PKCS11Provider ##

Equivalent Command: ssh -N -vvv -L 33333:codinn.com:22 -L 33334:localhost:22 -R localhost:3435:localhost:22 -o CheckHostIP=yes -o ServerAliveCountMax=3 -o ServerAliveInterval=15 -o PKCS11Provider=/Library/OpenSC/lib/opensc-pkcs11.so -o ConnectTimeout=15 -o TCPKeepAlive=no -o ExitOnForwardFailure=yes -p 22 yang@127.0.0.1
03:22:51 Connecting...
03:22:51 OpenSSH_8.8p1, OpenSSL 1.1.1l  24 Aug 2021
03:22:51 debug1: Reading configuration data /Users/yang/.ssh/config
03:22:51 debug1: Reading configuration data /etc/ssh/ssh_config
03:22:51 debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
03:22:51 debug1: /etc/ssh/ssh_config line 54: Applying options for *
03:22:51 debug2: resolve_canonicalize: hostname 127.0.0.1 is address
03:22:51 debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/yang/.ssh/known_hosts'
03:22:51 debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/yang/.ssh/known_hosts2'
03:22:51 debug3: ssh_connect_direct: entering
03:22:51 debug1: Connecting to 127.0.0.1 [127.0.0.1] port 22.
03:22:51 debug3: set_sock_tos: set socket 3 IP_TOS 0x48
03:22:51 debug2: fd 3 setting O_NONBLOCK
03:22:51 debug1: fd 3 clearing O_NONBLOCK
03:22:51 debug1: Connection established.
03:22:51 debug3: timeout: 15000 ms remain after connect
03:22:51 debug1: provider /Library/OpenSC/lib/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.25
03:22:51 debug1: provider /Library/OpenSC/lib/opensc-pkcs11.so slot 0: label <ssh> manufacturerID <piv_II> model <PKCS#15 emulate> serial <00000000> flags 0x40d
03:22:51 debug2: pkcs11_fetch_keys: provider /Library/OpenSC/lib/opensc-pkcs11.so slot 0: RSA SHA256:PdQXEDk9CVLzDOvOq1RuLFQ4FRCqe+CcpyKMCXLE4KE
03:22:51 debug1: have 1 keys
03:22:51 debug2: pkcs11_fetch_certs: provider /Library/OpenSC/lib/opensc-pkcs11.so slot 0: RSA SHA256:PdQXEDk9CVLzDOvOq1RuLFQ4FRCqe+CcpyKMCXLE4KE
03:22:51 debug2: pkcs11_fetch_certs: key already included
03:22:51 debug1: pkcs11_k11_free: parent 0x600003858160 ptr 0x600000f19960 idx 1
03:22:51 debug1: pkcs11_provider_unref: provider "/Library/OpenSC/lib/opensc-pkcs11.so" refcount 2
03:22:51 debug1: identity file /Users/yang/.ssh/id_rsa type 0
03:22:51 debug1: identity file /Users/yang/.ssh/id_rsa-cert type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_dsa type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_dsa-cert type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_ecdsa type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_ecdsa-cert type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_ecdsa_sk type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_ecdsa_sk-cert type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_ed25519 type 3
03:22:51 debug1: identity file /Users/yang/.ssh/id_ed25519-cert type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_ed25519_sk type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_ed25519_sk-cert type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_xmss type -1
03:22:51 debug1: identity file /Users/yang/.ssh/id_xmss-cert type -1
03:22:51 debug1: Local version string SSH-2.0-OpenSSH_8.8
03:22:51 debug1: Remote protocol version 2.0, remote software version OpenSSH_9.6
03:22:51 debug1: compat_banner: match: OpenSSH_9.6 pat OpenSSH* compat 0x04000000
03:22:51 debug2: fd 3 setting O_NONBLOCK
03:22:51 debug1: Authenticating to 127.0.0.1:22 as 'yang'
03:22:51 Authenticating...
03:22:51 debug3: record_hostkey: found key type ED25519 in file /Users/yang/.ssh/known_hosts:24
03:22:51 debug3: record_hostkey: found key type RSA in file /Users/yang/.ssh/known_hosts:25
03:22:51 debug3: record_hostkey: found key type ECDSA in file /Users/yang/.ssh/known_hosts:26
03:22:51 debug3: load_hostkeys_file: loaded 3 keys from 127.0.0.1
03:22:51 debug1: load_hostkeys: fopen /Users/yang/.ssh/known_hosts2: No such file or directory
03:22:51 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
03:22:51 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
03:22:51 debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
03:22:51 debug1: pkcs11_k11_free: parent 0x60000385c210 ptr 0x0 idx 1
03:22:51 debug3: send packet: type 20
03:22:51 debug1: SSH2_MSG_KEXINIT sent
03:22:51 debug3: receive packet: type 20
03:22:51 debug1: SSH2_MSG_KEXINIT received
03:22:51 debug2: local client KEXINIT proposal
03:22:51 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
03:22:51 debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
03:22:51 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
03:22:51 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
03:22:51 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
03:22:51 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
03:22:51 debug2: compression ctos: none,zlib@openssh.com,zlib
03:22:51 debug2: compression stoc: none,zlib@openssh.com,zlib
03:22:51 debug2: languages ctos: 
03:22:51 debug2: languages stoc: 
03:22:51 debug2: first_kex_follows 0 
03:22:51 debug2: reserved 0 
03:22:51 debug2: peer server KEXINIT proposal
03:22:51 debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-s,kex-strict-s-v00@openssh.com
03:22:51 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
03:22:51 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
03:22:51 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
03:22:51 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
03:22:51 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
03:22:51 debug2: compression ctos: none,zlib@openssh.com
03:22:51 debug2: compression stoc: none,zlib@openssh.com
03:22:51 debug2: languages ctos: 
03:22:51 debug2: languages stoc: 
03:22:51 debug2: first_kex_follows 0 
03:22:51 debug2: reserved 0 
03:22:51 debug1: kex: algorithm: curve25519-sha256
03:22:51 debug1: kex: host key algorithm: ssh-ed25519
03:22:51 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
03:22:51 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
03:22:51 debug3: send packet: type 30
03:22:51 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
03:22:51 debug3: receive packet: type 31
03:22:51 debug1: SSH2_MSG_KEX_ECDH_REPLY received
03:22:51 debug1: Server host key: ssh-ed25519 SHA256:wNVGzlFtM7LlDCIm1akQ/8r6bG7BJjWYq0DAcYSSjLs
03:22:51 debug3: record_hostkey: found key type ED25519 in file /Users/yang/.ssh/known_hosts:24
03:22:51 debug3: record_hostkey: found key type RSA in file /Users/yang/.ssh/known_hosts:25
03:22:51 debug3: record_hostkey: found key type ECDSA in file /Users/yang/.ssh/known_hosts:26
03:22:51 debug3: load_hostkeys_file: loaded 3 keys from 127.0.0.1
03:22:51 debug1: load_hostkeys: fopen /Users/yang/.ssh/known_hosts2: No such file or directory
03:22:51 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
03:22:51 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
03:22:51 debug1: Host '127.0.0.1' is known and matches the ED25519 host key.
03:22:51 debug1: Found key in /Users/yang/.ssh/known_hosts:24
03:22:51 debug1: pkcs11_k11_free: parent 0x600003850580 ptr 0x0 idx 1
03:22:51 debug3: send packet: type 21
03:22:51 debug2: set_newkeys: mode 1
03:22:51 debug1: rekey out after 134217728 blocks
03:22:51 debug1: SSH2_MSG_NEWKEYS sent
03:22:51 debug1: expecting SSH2_MSG_NEWKEYS
03:22:51 debug3: receive packet: type 21
03:22:51 debug1: SSH2_MSG_NEWKEYS received
03:22:51 debug2: set_newkeys: mode 0
03:22:51 debug1: rekey in after 134217728 blocks
03:22:51 debug1: Will attempt key: PIV AUTH pubkey RSA SHA256:PdQXEDk9CVLzDOvOq1RuLFQ4FRCqe+CcpyKMCXLE4KE token
03:22:51 debug1: Will attempt key: /Users/yang/.ssh/id_rsa RSA SHA256:jT8FTKIyuj8CNzGCD6w2Bw+e172mxdTznx19SRQsNSg
03:22:51 debug1: Will attempt key: /Users/yang/.ssh/id_dsa 
03:22:51 debug1: Will attempt key: /Users/yang/.ssh/id_ecdsa 
03:22:51 debug1: Will attempt key: /Users/yang/.ssh/id_ecdsa_sk 
03:22:51 debug1: Will attempt key: /Users/yang/.ssh/id_ed25519 ED25519 SHA256:brLSA5srpw6fDYS3YFG52aYQuZTLh4EW4FcfCAw8Npk
03:22:51 debug1: Will attempt key: /Users/yang/.ssh/id_ed25519_sk 
03:22:51 debug1: Will attempt key: /Users/yang/.ssh/id_xmss 
03:22:51 debug2: pubkey_prepare: done
03:22:51 debug3: send packet: type 5
03:22:51 debug3: receive packet: type 7
03:22:51 debug1: SSH2_MSG_EXT_INFO received
03:22:51 debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256>
03:22:51 debug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)
03:22:51 debug1: kex_input_ext_info: ping@openssh.com (unrecognised)
03:22:51 debug3: receive packet: type 6
03:22:51 debug2: service_accept: ssh-userauth
03:22:51 debug1: SSH2_MSG_SERVICE_ACCEPT received
03:22:51 debug3: send packet: type 50
03:22:51 debug3: receive packet: type 51
03:22:51 debug1: Authentications that can continue: publickey,password,keyboard-interactive
03:22:51 debug3: start over, passed a different list publickey,password,keyboard-interactive
03:22:51 debug3: preferred publickey,keyboard-interactive,password
03:22:51 debug3: authmethod_lookup publickey
03:22:51 debug3: remaining preferred: keyboard-interactive,password
03:22:51 debug3: authmethod_is_enabled publickey
03:22:51 debug1: Next authentication method: publickey
03:22:51 debug1: Offering public key: PIV AUTH pubkey RSA SHA256:PdQXEDk9CVLzDOvOq1RuLFQ4FRCqe+CcpyKMCXLE4KE token
03:22:51 debug3: send packet: type 50
03:22:51 debug2: we sent a publickey packet, wait for reply
03:22:51 debug3: receive packet: type 60
03:22:51 debug1: Server accepts key: PIV AUTH pubkey RSA SHA256:PdQXEDk9CVLzDOvOq1RuLFQ4FRCqe+CcpyKMCXLE4KE token
03:22:51 debug3: sign_and_send_pubkey: RSA SHA256:PdQXEDk9CVLzDOvOq1RuLFQ4FRCqe+CcpyKMCXLE4KE
03:22:51 debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:PdQXEDk9CVLzDOvOq1RuLFQ4FRCqe+CcpyKMCXLE4KE
03:22:51 debug1: read_passphrase: requested to askpass
03:23:06 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:06 debug1: pkcs11_k11_free: parent 0x600003840000 ptr 0x0 idx 1
03:23:06 debug1: pkcs11_check_obj_bool_attrib: provider "/Library/OpenSC/lib/opensc-pkcs11.so" slot 0 object 105553161716192: attrib 514 = 0
03:23:10 debug3: send packet: type 50
03:23:10 debug1: pkcs11_k11_free: parent 0x6000038540b0 ptr 0x0 idx 1
03:23:10 debug3: receive packet: type 52
03:23:10 debug1: pkcs11_k11_free: parent 0x6000038582c0 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003858000 ptr 0x600000f4b180 idx 1
03:23:10 debug1: pkcs11_provider_unref: provider "/Library/OpenSC/lib/opensc-pkcs11.so" refcount 2
03:23:10 Authenticated to 127.0.0.1 ([127.0.0.1]:22) using "publickey".
03:23:10 debug1: check provider "/Library/OpenSC/lib/opensc-pkcs11.so"
03:23:10 debug1: pkcs11_provider_finalize: provider "/Library/OpenSC/lib/opensc-pkcs11.so" refcount 1 valid 1
03:23:10 debug1: pkcs11_provider_unref: provider "/Library/OpenSC/lib/opensc-pkcs11.so" refcount 1
03:23:10 debug1: Local connections to LOCALHOST:33333 forwarded to remote address codinn.com:22
03:23:10 debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
03:23:10 debug3: sock_set_v6only: set socket 5 IPV6_V6ONLY
03:23:10 debug1: Local forwarding listening on ::1 port 33333.
03:23:10 debug2: fd 5 setting O_NONBLOCK
03:23:10 debug3: fd 5 is O_NONBLOCK
03:23:10 debug1: channel 0: new [port listener]
03:23:10 debug1: Local forwarding listening on 127.0.0.1 port 33333.
03:23:10 debug2: fd 6 setting O_NONBLOCK
03:23:10 debug3: fd 6 is O_NONBLOCK
03:23:10 debug1: channel 1: new [port listener]
03:23:10 debug1: Local connections to LOCALHOST:33334 forwarded to remote address localhost:22
03:23:10 debug3: channel_setup_fwd_listener_tcpip: type 2 wildcard 0 addr NULL
03:23:10 debug3: sock_set_v6only: set socket 7 IPV6_V6ONLY
03:23:10 debug1: Local forwarding listening on ::1 port 33334.
03:23:10 debug2: fd 7 setting O_NONBLOCK
03:23:10 debug3: fd 7 is O_NONBLOCK
03:23:10 debug1: channel 2: new [port listener]
03:23:10 debug1: Local forwarding listening on 127.0.0.1 port 33334.
03:23:10 debug2: fd 8 setting O_NONBLOCK
03:23:10 debug3: fd 8 is O_NONBLOCK
03:23:10 debug1: channel 3: new [port listener]
03:23:10 debug1: Remote connections from localhost:3435 forwarded to local address localhost:22
03:23:10 debug3: send packet: type 80
03:23:10 debug1: ssh_init_forwarding: expecting replies for 1 forwards
03:23:10 debug2: fd 3 setting TCP_NODELAY
03:23:10 debug3: set_sock_tos: set socket 3 IP_TOS 0x48
03:23:10 debug1: Requesting no-more-sessions@openssh.com
03:23:10 debug3: send packet: type 80
03:23:10 debug1: Entering interactive session.
03:23:10 debug1: pledge: filesystem full
03:23:10 debug2: fd 9 setting O_NONBLOCK
03:23:10 debug2: fd 10 setting O_NONBLOCK
03:23:10 Connected
03:23:10 debug3: receive packet: type 80
03:23:10 debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
03:23:10 debug3: client_input_hostkeys: received RSA key SHA256:ua5jIGNjqD3rt8sXZlPn0T0pSB7BOwuoSwlgCqQl7R8
03:23:10 debug3: client_input_hostkeys: received ECDSA key SHA256:dGairz3UvCI2Dl9G/i4Um+wUwaHQQxRV0xCgqpasaeM
03:23:10 debug3: client_input_hostkeys: received ED25519 key SHA256:wNVGzlFtM7LlDCIm1akQ/8r6bG7BJjWYq0DAcYSSjLs
03:23:10 debug1: client_input_hostkeys: searching /Users/yang/.ssh/known_hosts for 127.0.0.1 / (none)
03:23:10 debug3: hostkeys_foreach: reading file "/Users/yang/.ssh/known_hosts"
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850370 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850630 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850370 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850630 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850370 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850630 ptr 0x0 idx 1
03:23:10 debug3: hostkeys_find: found ssh-ed25519 key at /Users/yang/.ssh/known_hosts:24
03:23:10 debug3: hostkeys_find: found ssh-rsa key at /Users/yang/.ssh/known_hosts:25
03:23:10 debug1: pkcs11_k11_free: parent 0x6000038504d0 ptr 0x0 idx 1
03:23:10 debug3: hostkeys_find: found ecdsa-sha2-nistp256 key at /Users/yang/.ssh/known_hosts:26
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850370 ptr 0x0 idx 1
03:23:10 debug3: hostkeys_find: found ssh-ed25519 key under different name/addr at /Users/yang/.ssh/known_hosts:32
03:23:10 debug1: pkcs11_k11_free: parent 0x6000038504d0 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850370 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850370 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850630 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850630 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850630 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850630 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x6000038504d0 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x6000038504d0 ptr 0x0 idx 1
03:23:10 debug1: pkcs11_k11_free: parent 0x600003850790 ptr 0x0 idx 1
03:23:10 debug1: client_input_hostkeys: searching /Users/yang/.ssh/known_hosts2 for 127.0.0.1 / (none)
03:23:10 debug1: client_input_hostkeys: hostkeys file /Users/yang/.ssh/known_hosts2 does not exist
03:23:10 debug3: client_input_hostkeys: 3 server keys: 0 new, 3 retained, 0 incomplete match. 0 to remove
03:23:10 debug1: client_input_hostkeys: no new or deprecated keys from server
03:23:10 debug1: pkcs11_k11_free: parent 0x6000038440b0 ptr 0x0 idx 1
03:23:10 debug3: receive packet: type 4
03:23:10 debug1: Remote: /Users/yang/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
03:23:10 debug3: receive packet: type 4
03:23:10 debug1: Remote: /Users/yang/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
03:23:10 debug3: receive packet: type 81
03:23:10 debug1: remote forward success for: listen localhost:3435, connect localhost:22
03:23:10 debug1: forwarding_success: all expected forwarding replies received
03:23:25 debug3: send packet: type 80
03:23:25 debug3: receive packet: type 82
03:23:40 debug3: send packet: type 80
03:23:40 debug3: receive packet: type 82

Could you please send me desensitized verbose connection logs? Both success and failure logs, these logs could help me identified problem.

Kindly Regards,

Yang

Thank you for testing.

Yes, I will work on getting logs today. Please stand by.

No port forwarding or multiplexing. A standard SSH connection for CLI.

Equivalent Command: ssh -v -o PKCS11Provider=/Library/OpenSC/lib/opensc-pkcs11.so -o ServerAliveCountMax=3 -o PreferredAuthentications=pubkey -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes -p 38590 uthyr@REMOTEHOST
12:36:00 Connecting...
12:36:00 OpenSSH_8.8p1, OpenSSL 1.1.1l  24 Aug 2021
12:36:00 debug1: Reading configuration data /Users/uthyr/.ssh/config
12:36:00 debug1: Reading configuration data /Users/uthyr/.ssh/config.d/10.conf
12:36:00 debug1: Reading configuration data /Users/uthyr/.ssh/config.d/20.conf
12:36:00 debug1: Reading configuration data /Users/uthyr/.ssh/config.d/30.conf
12:36:00 debug1: Reading configuration data /Users/uthyr/.ssh/config.d/40.conf
12:36:00 debug1: /Users/uthyr/.ssh/config line 3: Applying options for *
12:36:00 debug1: Reading configuration data /etc/ssh/ssh_config
12:36:00 debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
12:36:00 debug1: /etc/ssh/ssh_config line 54: Applying options for *
12:36:01 debug1: Connecting to REMOTEHOST [192.168.1.230] port 38590.
12:36:01 debug1: Connection established.
12:36:02 debug1: provider /Library/OpenSC/lib/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.25
12:36:03 debug1: pkcs11_register_provider: provider /Library/OpenSC/lib/opensc-pkcs11.so returned no slots
12:36:03 debug1: identity file /Users/uthyr/.ssh/WorkStation.lan_ed25519 type 3
12:36:03 debug1: identity file /Users/uthyr/.ssh/WorkStation.lan_ed25519-cert type -1
12:36:03 debug1: Local version string SSH-2.0-OpenSSH_8.8
12:36:03 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
12:36:03 debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
12:36:03 debug1: Authenticating to REMOTEHOST:38590 as 'uthyr'
12:36:03 debug1: SSH2_MSG_KEXINIT sent
12:36:03 Authenticating...
12:36:03 debug1: SSH2_MSG_KEXINIT received
12:36:03 debug1: kex: algorithm: curve25519-sha256
12:36:03 debug1: kex: host key algorithm: ssh-ed25519
12:36:03 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
12:36:03 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
12:36:03 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
12:36:03 debug1: SSH2_MSG_KEX_ECDH_REPLY received
12:36:03 debug1: Server host key: ssh-ed25519 SHA256:HASH
12:36:03 debug1: load_hostkeys: fopen /Users/uthyr/.ssh/known_hosts2: No such file or directory
12:36:03 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
12:36:03 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
12:36:03 debug1: load_hostkeys: fopen /Users/uthyr/.ssh/known_hosts2: No such file or directory
12:36:03 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
12:36:03 debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
12:36:03 debug1: Host '[REMOTEHOST]:38590' is known and matches the ED25519 host key.
12:36:03 debug1: Found key in /Users/uthyr/.ssh/known_hosts:9
12:36:03 Host key fingerprint is SHA256:HASH
+--[ED25519 256]--+
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
+----[SHA256]-----+
12:36:03 debug1: rekey out after 134217728 blocks
12:36:03 debug1: SSH2_MSG_NEWKEYS sent
12:36:03 debug1: expecting SSH2_MSG_NEWKEYS
12:36:03 debug1: SSH2_MSG_NEWKEYS received
12:36:03 debug1: rekey in after 134217728 blocks
12:36:03 debug1: Will attempt key: /Users/uthyr/.ssh/WorkStation.lan_ed25519 ED25519 SHA256:HASH explicit
12:36:03 debug1: SSH2_MSG_EXT_INFO received
12:36:03 debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
12:36:03 debug1: SSH2_MSG_SERVICE_ACCEPT received
12:36:03 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
12:36:03 debug1: No more authentication methods to try.
12:36:03 uthyr@REMOTEHOST: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
12:36:03 Abnormal Disconnect

Did you ever customized OpenSC config files? If you did, could you please send me a copy of your configuration?

Kindly Regards,

Yang

I have not modified any OpenSC config files. I have set up options for the hardware key to communicate with gpg-agent.

Workstation [~/.gnupg]$ cat gpg.conf                                                                         
no-emit-version
default-key <KEYID>
Workstation [~/.gnupg]$ cat gpg-agent.conf                                                                    
pinentry-program /opt/homebrew/bin/pinentry-mac
enable-ssh-support
default-cache-ttl 600
max-cache-ttl 7200
default-cache-ttl-ssh 600
max-cache-ttl-ssh 7200

OpenSC warns no slots found, so I guess there are permission issues, gpg-agent should not the cause. Could you please run following commands, and see what print:

cat /Library/OpenSC/etc/opensc.conf
pkcs11-tool -L
opensc-tool -l
pkcs15-tool -k
opensc-explorer

Output on my MacBook (M1 Pro, Sonoma 14.5):

❯ cat /Library/OpenSC/etc/opensc.conf
app default {
        # debug = 3;
        # debug_file = opensc-debug.txt;
}
❯ pkcs11-tool -L
Available slots:
Slot 0 (0x0): Yubico Yubikey 4 OTP+U2F+CCID
  token label        : ssh
  token manufacturer : piv_II
  token model        : PKCS#15 emulated
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 0.0
  firmware version   : 0.0
  serial num         : 00000000
  pin min/max        : 4/8
❯ opensc-tool -l
# Detected readers (pcsc)
Nr.  Card  Features  Name
0    Yes             Yubico Yubikey 4 OTP+U2F+CCID
❯ pkcs15-tool -k

Using reader with a card: Yubico Yubikey 4 OTP+U2F+CCID
Private RSA Key [PIV AUTH key]
        Object Flags   : [0x01], private
        Usage          : [0x2E], decrypt, sign, signRecover, unwrap
        Access Flags   : [0x1D], sensitive, alwaysSensitive, neverExtract, local
        Algo_refs      : 0
        ModLength      : 2048
        Key ref        : 154 (0x9A)
        Native         : yes
        Auth ID        : 01
        ID             : 01
❯ opensc-explorer
OpenSC Explorer version 0.25.1
Using reader with a card: Yubico Yubikey 4 OTP+U2F+CCID
unable to select MF: File not found

I realized that my defined gpg-agent.sock had been removed from the preferences when I upgraded (and was not re-populated by iCloud sync). When I added it back in back into the Core Tunnel/Shell Auth Agent preference, the hardware key correctly returned slots.

Thank you for all of your help