Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.
Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.
Rolled back to the backup before I made it public and now I have a security checklist.
One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.
“Hey, why is this Valheim server running like shit?”
“Wtf is
xrx?”“Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”
So anyway, now I use NixOS.
I’ve gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn’t know better, instead of pure salt like another site I won’t mention.
I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.
We all make mistakes!
Interesting. Do you know how it got compromised?
I published it to the internet and the next day, I couldn’t ssh into the server anymore with my user account and something was off.
Tried root + password, also failed.
Immediately facepalmed because the password was the generic 8 characters and there was no fail2ban to stop guessing.
deleted by creator
More importantly, don’t open up SSH to public access. Use a VPN connection to the server. This is really easy to do with Netbird, Tailscale, etc. You should only ever be able to connect to SSH privately, never over the public net.
It’s perfectly safe to run SSH on port 22 towards the open Internet with public key authentication only.
https://nvd.nist.gov/vuln/detail/cve-2024-6409 RCE as root without authentication via Open SSH. If they’ve got a connection, that’s more than nothing and sometimes it’s enough.
That attack vector is exactly the same towards a VPN.
A VPN like Wireguard can run over UDP on a random port which is nearly impossible to discover for an attacker. Unlike sshd, it won’t even show up in a portscan.
This was a specific design goal of Wireguard by the way (see “5.1 Silence is a virtue” here https://www.wireguard.com/papers/wireguard.pdf)
It also acts as a catch-all for all your services, so instead of worrying about the security of all the different sshds or other services you may have exposed, you just have to keep your vpn up to date.
Are you talking a VPN running on the same box as the service? UDP VPN would help as another mentioned, but doesn’t really add isolation.
If your vpn box is standalone, then getting root is bad but just step one. They have to own the VPN to be able to even do more recon then try SSH.
Defense in depth. They didn’t immediately get server root and application access in one step. Now they have to connect to a patched, cert only, etc SSH server. Just looking for it could trip into some honeypot. They had to find the VPN host as well which wasn’t the same as the box they were targeting. That would shut down 99% of the automated/script kiddie shit finding the main service then scanning that IP.
You can’t argue that one step to own the system is more secure than two separate pieces of updated software on separate boxes.
what’s netbird
https://netbird.io/. Wireguard based software defined networking, very similar to Tailscale.
Tailscale? Netbird? I have been using hamachi like a fucking neanderthal. I love this posts, I learn so much
Ah, timeless classic.
Which distro allows root to login via SSH?
All of them if you configure it?
Not very many. None of the enterprise ones, at least.
Lol ssh has no reason to be port exposed in 99% of home server setups.
VPNs are extremely easy, free, and wireguard is very performant with openvpn also fine for ssh. I have yet to see any usecase for simply port forwarding ssh in a home setup. Even a public git server can be tunneled through https.
Yeah I’m honest with myself that I’m a security newb and don’t know how to even know what I’m vulnerable to yet. So I didn’t bother opening anything at all on my router. That sounded way too scary.
Tailscale really is magic. I just use Cloudflare to forward a domain I own, and I can get to my services, my NextCloud, everything, from anywhere, and I’m reasonably confident I’m not exposing any doors to the innumerable botnet swarms.
It might be a tiny bit inconvenient if I wanted to serve anything to anyone not in my Tailnet or already on my home LAN (like sending al someone a link to a NextCloud folder for instance.), but at this point, that’s quite the edge case.
I learned to set up NGINX proxy manager for a reverse proxy though, and that’s pretty great! I still harden stuff where I can as I learn, even though I’m confident nobody’s even seeing it.
Honestly, crowdsec with the nginx bouncer is all you need security-wise to start experimenting. It isn’t perfect security, but it is way more comprehensive than fail2ban for just getting started and figuring more out later.
Here is my traefik-based crowdsec docker composer:
services: crowdsec: image: crowdsecurity/crowdsec:latest container_name: crowdsec environment: GID: $PGID volumes: - $USERDIR/dockerconfig/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml - $USERDIR/data/Volumes/crowdsec:/var/lib/crowdsec/data/ - $USERDIR/dockerconfig/crowdsec:/etc/crowdsec/ - $DOCKERDIR/traefik2/traefik.log:/var/log/traefik/traefik.log:ro networks: - web restart: unless-stopped bouncer-traefik: image: docker.io/fbonalair/traefik-crowdsec-bouncer:latest container_name: bouncer-traefik environment: CROWDSEC_BOUNCER_API_KEY: $CROWDSEC_API CROWDSEC_AGENT_HOST: crowdsec:8080 networks: - web # same network as traefik + crowdsec depends_on: - crowdsec restart: unless-stopped networks: web: external: truehttps://github.com/imthenachoman/How-To-Secure-A-Linux-Server this is a more in-depth crash course for system-level security but hasn’t been updated in a while.
That’s rad! Thanks so much for sharing that! Definitely gonna give this a read. Very much appreciated. :)
How are people’s servers getting compromised? I’m no security expert (I’ve never worked in tech at all) and have a public VPS, never been compromised. Mainly just use SSH keys not passwords, I don’t do anything too crazy. Like if you have open SSH on port 22 with root login enabled and your root password is
password123then maybe but I’m surprised I’ve never been pwned if it’s so easy to get got…By allowing password login and using weak passwords or by reusing passwords that have been involved in a data breach somewhere.
That makes sense. It feels a bit mad that the difference between getting pwned super easy vs not is something simple like that. But also reassuring to know, cause I was wondering how I heard about so many hobbyist home labs etc getting compromised when it’d be pretty hard to obtain a reasonably secured private key (ie not uploaded onto the cloud or anything, not stored on an unencrypted drive that other people can easily access, etc). But if it’s just password logins that makes more sense.
glad my root pass is
toorand not something as obvious aspassword123toor, like Tor, the leet hacker software. So it must be super secure.
That’s incredible, I’ve got the same combination on my luggage.
Do not allow username/password login for ssh. Force certificate authentication only!
Do not allow username/password login for ssh
This is disabled by default for the root user.
$ man sshd_config ... PermitRootLogin Specifies whether root can log in using ssh(1). The argument must be yes, prohibit-password, forced-commands-only, or no. The default is prohibit-password. ...Why though? If u have a strong password, it will take eternity to brute force
I’m confused. I never disable root user and never got hacked.
Is the issue that the app is coded in a shitty way maybe ?
You can’t really disable the root user. You can make it so they can’t login remotely, which is highly suggested.
As a linux n00b who just recently took the plunge and set up a public site (tho really just for my own / selfhosting),
Can anyone recommend a good guide or starting place for how to harden the setup? Im running mint on my former gaming rig, site is set up LAMP
The other poster gave you a lot. If that’s too much at once, the really low hanging fruit you want to start with is:
-
Choose an active, secure distro. There’s a lot of flavors of Linux out there and they can be fun to try but if you’re putting something up publicly it should be running on one that’s well maintained and known for security. CentOS and Debian are excellent easy choices for example.
-
Similarly, pick well maintained software with a track record. Nginx and Apache have been around forever and have excellent track records, for example, both for being secure and fixing flaws quickly.
-
If you use Docker, once again keep an eye out for things that are actively maintained. If you decide to use Nginx, there will be five million containers to choose from. DockerHub gives you the tools to make this determination: Download number is a decent proxy for “how many people are using this” and the list of updates tells you how often and how recently it’s being updated.
-
Finally, definitely do look at the other poster’s notes about SSH. 5 seconds after you put up an SSH server, you’ll be getting hit with rogue login attempts.
-
Definitely get a password manager, and it’s not just one password per server but one password per service. Your login password to the computer is different from your login to any other things your server is running.
The rest requires research, but these steps will protect you from the most common threats pretty effectively. The world is full of bots poking at every service they can find, so keeping them out is crucial. You won’t be protected from a dedicated, knowledgeable attacker until you do the rest of what the other poster said, and then some, so try not to make too many enemies.
The TLDR is here : https://www.digitalocean.com/community/tutorials/recommended-security-measures-to-protect-your-servers
You won’t be protected from a dedicated, knowledgeable attacker until you do the rest of what the other poster said, and then some,
You’re right I didn’t even get to ACME and PKI or TOTP
https://letsencrypt.org/getting-started/
https://openbao.org/docs/secrets/pki/
https://openbao.org/docs/secrets/totp/
And for bonus points build your own certificate authority to sign it all.
https://smallstep.com/blog/build-a-tiny-ca-with-raspberry-pi-yubikey/
Thank you for this! I’ve got some homework to do!
Good luck on your journey.
I would suggest having two servers one to test and one to expose to the Internet. That way if you make a mistake hopefully you’ll find it before you expose it to the Internet.
Thank you!
-
Paranoid external security. I’m assuming you already have a domain name. I’m also assuming you have some ICANN anonymization setup.
This is your local reverse Proxy. You can manage all this with a container called nginx proxy manager, but it could benefit you to know it’s inner workings first. https://www.howtogeek.com/devops/what-is-a-reverse-proxy-and-how-does-it-work/
https://cloud9sc.com/nginx-proxy-manager-hardening/
https://github.com/NginxProxyManager/nginx-proxy-manager
Next you’ll want to proxy your IP address as you don’t want that pointing to your home address
https://developers.cloudflare.com/learning-paths/get-started-free/onboarding/proxy-dns-records/
Remote access is next. I would suggest setting up wireguard on a machine that’s not your webserver, but you can also set that up in a container as well. Either way you’ll need to punch another hole in your router to point to your wire guard bastion host on your local network. It has many clients for windows and linux and android and IOS
https://github.com/angristan/wireguard-install
https://www.wireguard.com/quickstart/
https://github.com/linuxserver/docker-wireguard
Now internally, I’m assuming you’re using Linux. In that case I’d suggest securing your ssh on all machines that you log into. On the machines you’re running you should also install fail2ban, UFW, git, and some monitoring if you have the overhead but the monitoring part is outside of the purview of this comment. If you’re using UFW your very first command should be
sudo ufw allow sshhttps://www.howtogeek.com/443156/the-best-ways-to-secure-your-ssh-server/
https://github.com/fail2ban/fail2ban
https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands
Now for securing internal linux harden the kernel and remove root user. If you do this you should have a password manager setup. keepassx or bitwarden are ones I like. If those suck I’m sure someone will suggest something better. The password manager will have the root password for all of your Linux machines and they should be different passwords.
https://www.makeuseof.com/ways-improve-linux-user-account-security/
https://bitwarden.com/help/self-host-an-organization/
Finally you can harden the kernel
https://codezup.com/the-ultimate-guide-to-hardening-your-linux-system-with-kernel-parameters/
TLDR: it takes research but a good place to start is here
Correct, horse battery staple.
I couldn’t justify putting correct in my username on Lemmy. But I loved the reference too much not to use it, so here I am, a less secure truncated version of a better password.
Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.
When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.
At least you had a backup
On a new linux install or image I will always:
- Make new users(s)
- Setup new user to sudo
- Change ssh port
- Change new user to authenticate ssh via key+password
- Disable root ssh login
Although disabling the root user is a good part of security, leaving it enabled should not alone cause you to get compromised. If it did, you were either running a very old version of OpenSSH with a known flaw, or, your chosen root password was very simple.
The latter. It was autogenerated by the VPS hosting service and I didn’t think about it.
It should be a serious red flag that your VPS host is generating root passwords simple enough to get quickly hacked.
I’m pretty sure they assumed if you bought their service, you have the competency to properly set it up.
And I proved them wrong.
I’m having the opposite problem right now. Tightend a VM down so hard that now I can’t get into it.
I’ve been quite stupid with this but never really had issues. Ever since I changed the open ssh port from 22 to something else, my server is basically ignored by botnets. These days I obviously also have some other tricks like fail2ban, but it was funny how effective that was.
Almost the same here. I also change some ssh settings: disable root login, disable password, allow only public key login. That’s about it. I never had any problems.
Hmmm those are some good extra tips. I’ll take a look, thanks!
We’re not really supposed to expose the ssh port to the internet at all. Better to hide it behind a vpn.
But it’s too damn convenient for so many use cases. Fuck it. Fail2Ban works fine.
You can also set up an ssh tarpit on port 22, which will tie up the bot’s resources and get them stuck in a loop for a while. But I didn’t think it was worth attracting extra attention from the bot admins to satisfy my pettiness.
Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.
The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.
Endlessh and fail2ban are great to setup a ssh honeypot. There even is a Prometheus exporter version for some nice stats
Just expose endlessh on your public port 22 and if needed, configure your actual ssh on a different port. But generally: avoid exposing ssh if you don’t actually need it or at least disable root login and disable password authentication completely.
https://github.com/skeeto/endlessh https://github.com/shizunge/endlessh-go https://github.com/itskenny0/fail2ban-endlessh
If it is your single purpose to create a blocklist of suspect IP addresses, I guess this could be a honeypot strategy.
If it’s to secure your own servers, you’re only playing whack-a-mole using this method. For every IP you block, ten more will pop up.
Instead of blacklisting, it’s better to whitelist the IP addresses or ranges that have a legitimate reason to connect to your server, or alternatively use someting like geoip firewall rules to limit the scope of your exposure.
Since I’ve switched to using SSH keys for all auth Ive had no problems I’m aware of. Plus I don’t need to remember a bunch of passwords.
But then I’ve had no training in this area. What do I know
I’ve recently seen login attempts using keys, found it curious…
Probably still looking for hosts that have weak Debian SSH keys that users forgot to replace. https://www.hezmatt.org/~mpalmer/blog/2024/04/09/how-i-tripped-over-the-debian-weak-keys-vuln.html
Weird. My last setup had a NAT with a few VMs hosting a few different services. For example, Jellyfin, a web server, and novnc/vm. That turned out perfectly fine and it was exposed to the web. You must have had a vulnerable version of whatever web host you were using, or maybe if you had SSH open without rate limits.
