I have a number of servers, including a few on the home office network, that accept SSH connections. Even though they are serving on different (non-standard) SSH ports, there are regular attempts made to break it via brute-force – I can see how some random IP addresses start trying to log in using different standard user names. It's therefore never too late to use additional software for protecting SSH service, something like fail2ban.[Read more…] about How To: Use fail2ban to Protect SSH
Super quick advice today, but one of them pearls of experience that now and then saves your day. Learn how to check and confirm your recent changes to SSH daemon config file (/etc/ssh/sshd_config) won't break your remote SSH access.
Why Double-Checking Configs Is A Good Idea
I should probably start a special section of Unix Tutorial someday, just to talk about how and when things can go wrong. These things below would certainly belong to that section.
Why it's a good idea to check that your new config file is error free:
- avoid getting service outage (syntax error means service won't restart)
- prevent service malfunction (if you end up with only partial service functionality)
- don't get yourself locked out of service (or server, in case of broken SSH)
How To Check SSHd Config
I have shown you before how to test new SSHd config on a different port, but there's also a way to check primary config.
Here's how you do it:
greys@s2:~ $ sudo sshd -t
It will either return nothing, or complain about errors or highlight deprecated options, like this:
greys@s2:~ $ sudo sshd -t
/etc/ssh/sshd_config line 56: Deprecated option RSAAuthentication
That's all there is to it, enjoy!
One of the greatest improvements introduced by the SSH protocol is key-based authentication – meaning your client and SSH server establish validity of your SSH keypair and let you gain remote SSH access without asking for your password.[Read more…] about Deploy Your SSH key To Remote Server
This is not the most obvious functionality, hence I decided to share it as a separate post. It's quite easy and perfectly acceptable to specify more than one SSH port for your sshd daemon – useful for debugging or added security (when bound to separate IP addresses).
Adding Extra SSH ports
Simply edit the /etc/ssh/sshd_config file and add more port numbers under the existing default port (it's commented out because 22 is used by default):
greys@server:~$ sudo vi /etc/ssh/sshd_config
IMPORTANT: you must uncomment Port 22, otherwise new ports will be the only SSH ports listened on (so SSH port 22 will stop working).
Now restart ssh:
greys@server:~$ sudo systemctl restart ssh
Confirm each new SSH port
netstat command with grep confirms that all 3 ports are being listened on now:
greys@server:~$ netstat -nal | grep 22
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:221 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:222 0.0.0.0:* LISTEN
If we want to, we can even try connecting to a non-standard ssh port like 221 or 222 as per our changes.
Don't be alarmed about warning:
root@server:~# ssh greys@localhost -p 222
The authenticity of host '[localhost]:222 ([127.0.0.1]:222)' can't be established.
ECDSA key fingerprint is SHA256:12efZx1MOEmlxQOWKhM5eaxDwJr4vUlLhcpElkGHTow.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:222' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 19.04 (GNU/Linux 4.18.0-20-generic x86_64)
Hope you enjoy this advice, stay tuned for more!
Here I was trying to ssh from my XPS laptop to MacBook Pro for some quick command, when SSH started giving me the too many authentication failures error. I decided to capture findings here as a blog post.
Too Many Authentication Failures
Here's how the error looked from my Ubuntu 19.04 command line:
greys@xps:~ $ ssh greys@maverick Received disconnect from 192.168.1.200 port 22:2: Too many authentication failures Disconnected from 192.168.1.200 port 22
The weird thing is that this was happening without any passwords asked, so at first it seemed really strange: you get authentication failures but you actually haven't tried authenticating at all.
Why Too Many Authentication Failures Occur
So yes, these errors happen when you attempt to log in using some credentials and you are denied access for a few times in a row due to incorrect credentials.
Something as fundamental as SSH client and server are rarely wrong in such basic things. So thinking about the error a bit more (and Googling around, of course) I realised that authentication attempts were made using SSH keys I have configured on my Ubuntu laptop. There's quite a few and SSH client was offering them one after another to the MacBook's SSH daemon in attempts to log me in.
So I never got asked for a password because my SSH client already offered a few SSH keys and remote SSH server counted each offering as an authentication attempt. So when this maxed out the SSH server limit, I got the error.
Related to the error above is this MaxAuthTries setting in /etc/ssh/sshd_config file.
This option is set to something fairly reasonable usually, in MacOS Mojave it's set to 6 by default. But because I changed it to 3 in the past for better security, it limited my access when my SSH client was offering more than 3 SSH keys to log in.
Working around the Too Many Authentication Attemps Problem
There's a number of approaches, all of them to do with SSH identities used for remote access. They are managed by SSH agent, a special software usually starting automatically with your laptop login that tracks all the usernames and SSH keys you have to try them when accessing things remotely.
Disable SSH agent temporarily
So the easiest fix is to disable SSH agent temporarily and try login again (for password logins it does the trick).
I'll quickly show the steps but will need to write a separate proper post on using SSH agent usage soon.
Step 1: we check user variables for SSH_AUTH_SOCK
This variable will usually confirm if you have SSH Agent. If this variable exists and points to the valid file, that's the Unix socket used by your SSH agent:
greys@xps:~ $ env | grep SSH SSH_AUTH_SOCK=/home/greys/.ssh/ssh-auth-sock.xps SSH_AGENT_PID=1661
Step 2: we reset the SSH_AUTH_SOCK variable
Let's set this variable to empty value and check it:
greys@xps:~ $ SSH_AUTH_SOCK= greys@xps:~ $ env | grep SSH SSH_AUTH_SOCK= SSH_AGENT_PID=1661
That's it, now logins to Macbook laptop should work again:
greys@xps:~ $ ssh greys@maverick Password: Last login: Wed Jun 12 12:31:33 2019 from 192.168.1.60 greys@maverick:~ $
That's it for today! Quite a few advanced topics in just one post, so I'll be sure to revisit it and expand with further posts on the concepts of SSH ports, SSH agent, passwordless SSH and generating SSH keys.
Sometimes you need to tweak your SSH daemon on an important system and you just don't know if particular settings will break connectivity to the server or not. In such cases it's best to test new SSHd config using separate SSH daemon instance and separate SSH port – debug it there and only then apply new configs into your primary SSHd configuration.
Creating New SSHd Config
The easiest is to start by copying /etc/ssh/sshd_config file – you will need sudo/root privileges for that:
greys@s2:~ $ sudo cp /etc/ssh/sshd_config /home/greys
I then just remove everything I don't need from it, leaving bare minimum. These are the parameters I kept (I ended up renaming my config to /home/greys/sshd_config.minimal after edits)
greys@s2:~ $ grep -v ^# /home/greys/sshd_config.minimal | uniq -u Port 2222 HostKey /etc/ssh/ssh_host_rsa_key RSAAuthentication yes PubkeyAuthentication yes AuthorizedKeysFile /var/ssh/%u/authorized_keys PasswordAuthentication no UsePAM yes
I only updated the SSH Port parameter – you can pick any other number instead of 2222.
Starting SSH daemon with custom config file
There's a few rules for testing SSH configuration using separate file:
- you need to have sudo/root privilege (mostly to avoid mess with host SSH keys)
- it's better to increase verbosity level to see what's going on
- it's best to run SSHd in foreground (non-daemon) mode
With these principles in mind, here's the command line to test the config shown above:
greys@s2:~ $ sudo /usr/sbin/sshd -f /home/greys/sshd_config.minimal -ddd -D
debug2: load_server_config: filename /home/greys/sshd_config.minimal
debug2: load_server_config: done config len = 194
debug2: parse_server_config: config /home/greys/sshd_config.minimal len 194
debug3: /home/greys/sshd_config.minimal:1 setting Port 2222
debug3: /home/greys/sshd_config.minimal:10 setting HostKey /home/greys/ssh_host_rsa_key
debug3: /home/greys/sshd_config.minimal:12 setting RSAAuthentication yes
/home/greys/sshd_config.minimal line 12: Deprecated option RSAAuthentication
debug3: /home/greys/sshd_config.minimal:13 setting PubkeyAuthentication yes
debug3: /home/greys/sshd_config.minimal:18 setting AuthorizedKeysFile /var/ssh/%u/authorized_keys
debug3: /home/greys/sshd_config.minimal:20 setting PasswordAuthentication no
debug3: /home/greys/sshd_config.minimal:22 setting UsePAM yes
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:g7xhev6zJefXRFc0ClAG4rzpFI1Ts8H7PhQ/h3PTmLM
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
That's it, the configuration is ready to be tested (assuming your firewall on server doesn't block port 2222).
Testing SSH connectivity using Different SSH Port
Here's my login session in a separate window, connecting from my MacBook Pro to the s2 server on SSH port 2222 (I have masked my static IP with aaa.bbb.ccc.ddd and my s2 server's IP with eee.fff.ggg.hhh):
greys@MacBook-Pro:~ $ ssh s2 -p 2222 Warning: untrusted X11 forwarding setup failed: xauth key data not generated Last login: Fri May 24 15:53:59 2019 from aaa.bbb.ccc.ddd debug3: Copy environment: XDG_SESSION_ID=14813 debug3: Copy environment: XDG_RUNTIME_DIR=/run/user/1000 Environment: USER=greys LOGNAME=greys HOME=/home/greys PATH=/usr/local/bin:/usr/bin MAIL=/var/mail/greys SHELL=/bin/bash SSH_CLIENT=aaa.bbb.ccc.ddd 64168 2222 SSH_CONNECTION=aaa.bbb.ccc.ddd 64168 eee.fff.ggg.hhh 2222 SSH_TTY=/dev/pts/14 TERM=xterm-256color XDG_SESSION_ID=14813 XDG_RUNTIME_DIR=/run/user/1000 SSH_AUTH_SOCK=/tmp/ssh-ajOUyvbR6i/agent.20996 greys@s2:~ $ uptime 16:18:08 up 86 days, 17:32, 2 users, load average: 1.00, 1.02, 1.05
Pretty cool, huh?
I needed to forward X11 output from one of my Linux servers recently to run virt-manager (manager for virtual machines in KVM), and because it's been a while I had to download and install X11 server again.
As some of you know, Xorg server is no longer shipped/installed with macOS by default. So you have to download it from XQuartz page: https://www.xquartz.org/releases/index.html. Usually you do it, install it and that's it – no additional steps are needed.
But things are slightly different for the macOS High Sierra, apparently.
The latest release hasn't been updated since 2016 which I believe is before High Sierra – which explains why things don't "just work" anymore. Fear not though – I tracked the issue down and it's explained below.
UPDATE 03/2019: MacOS Mojave works just great, you may skip Step 3 in the procedure below.
Steps to get X11 Forwarding in macOS High Sierra
- Download and install the latest release from xquartz.org website
- Start XQuartz
- IMPORTANT: verify xauth location
SSH configuration file /etc/ssh/ssh_config might contain path to xauth tool, which may be incorrect depending on your OSX/MacOS version. Here's how to check:
greys@maverick:~ $ grep xauth /etc/ssh/sshd_config
if this returns nothing, you can skip to Step 4 below. If this gives you an output, compare it to the path from the next command:
greys@maverick:~ $ which xauthIf the locations differ, update the /etc/ssh/ssh_config file:
greys@maverick:~ $ sudo vi /etc/ssh/ssh_config
- Connect to remote server using -X option which does X11 forwarding for SSH:
greys@maverick:~ $ ssh -X centos.unixtutorial.or
- Check the DISPLAY variable, it should now be set correctly:
greys@centos:~ $ echo $DISPLAY
That's it for today!
As you can imagine, SSH keypairs – combinations of private and public keys – are vital elements of your digital identity as a sysadmin or a developer. And since they can be used for accessing source code repositories and for deploying changes to production environments, you usually have more than one SSH key. That's why it's important to know how to inspect SSH key fingerprints.
SSH is such an integral part of everyday Linux/Unix life now, that it makes sense to use it for as many remote access and automation tasks as you can. As you probably know, you shouldn't be using password SSH authentication unless you have a pretty good reason to do so. By default, always use SSH keys. Today I'll show you how to generate SSH keys.
Generate SSH key with ssh-keygen
ssh-keygen is a standard utility supplied with SSH package. If you have ssh command on your system, you probably have the ssh-keygen command as well.
Without any command line options, ssh-keygen will ask you a few questions and create the key with default settings:
[greys@rhel8 ~]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/greys/.ssh/id_rsa): Created directory '/home/greys/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/greys/.ssh/id_rsa. Your public key has been saved in /home/greys/.ssh/id_rsa.pub. The key fingerprint is: SHA256:Seu7UBogeX+g9+iv01CDJqiXAby740JKZGrZtu1T3oQ greys@rhel8 The key's randomart image is: +---[RSA 2048]----+ |. | |.. . | | .+.o ... | | +oo.+oooo | |+.+o.o+.S. | |o*oo ..E . | |=.o o *o= | |oo . +.o.o | |o.. ..+++. | +----[SHA256]-----+ [greys@rhel8 ~]$
Specify SSH key size for ssh-keygen
Most likely you'll have your preferences for SSH keys and it is much easier to just specify them when running the ssh-keygen command.
This is how one can generate 4096-bit key, for example:
[greys@rhel8 ~]$ ssh-keygen -b 4096 Generating public/private rsa key pair. Enter file in which to save the key (/home/greys/.ssh/id_rsa): /home/greys/.ssh/rsa-4k Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/greys/.ssh/rsa-4k. Your public key has been saved in /home/greys/.ssh/rsa-4k.pub. The key fingerprint is: SHA256:4rf1AGIc99L57/xC1PWu7pJpwhkn5YCmZQqua/XdmGA greys@rhel8 The key's randomart image is: +---[RSA 4096]----+ | | | .| | . .. .o| | .. o=o... . o| | . .=*S ++ . . | | ooEo oo.o . .| | o o.o.=o=.+ . | | o ..+o=o=oo | | ... . o.=*o.| +----[SHA256]-----+