The ssh command line shipped with macOS actually also integrates Keychain with private keys, so if you didn't invoke ssh-add command, but ssh connects without prompt private key passphrase, then instead of fetch keys from ssh-agent daemon, ssh might actually decrypts key from Keychain silently.