This is a problem which bugs the Internet. Many solutions were suggested. All of them work sometimes (like making sure that the permissions on the private/public SSH key files are correct).
One solution which never gets mentioned is about DNS. When attempting login to a remote server, the server attempts to resolve the connecting address (yours) using the DNS server defined to the server. This is the default for SSHD, and I seldom see any reason to change it. However, if the DNS servers are inaccessible, then the login takes very long (actually – waiting for the DNS query attempt to timeout). If this thing happens when using SSH-agent (I use KeePass2 with KeeAgent, as a very secure means of protecting my private keys. I suggest you check it out), the timeout might be just long enough to timeout on the agent’s side, and thus, produce the dreaded ‘sign_and_send_pubkey: signing failed for RSA from agent: agent refused operation’ error message.
The easy two possible solutions (pick one)L
- Disable SSH DNS query (Remove the comment before ‘UseDNS no’ directive in /etc/ssh/sshd_config)
- Disable DNS servers – comment out the defined servers in /etc/resolv.conf
This solves the timeout, and with it – this message, and your connection problems.