Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.

Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.

Rolled back to the backup before I made it public and now I have a security checklist.

  • gerryflap@feddit.nl
    link
    fedilink
    arrow-up
    4
    ·
    2 hours ago

    I’ve been quite stupid with this but never really had issues. Ever since I changed the open ssh port from 22 to something else, my server is basically ignored by botnets. These days I obviously also have some other tricks like fail2ban, but it was funny how effective that was.

  • potentiallynotfelix@lemmy.fish
    link
    fedilink
    arrow-up
    5
    ·
    2 hours ago

    Weird. My last setup had a NAT with a few VMs hosting a few different services. For example, Jellyfin, a web server, and novnc/vm. That turned out perfectly fine and it was exposed to the web. You must have had a vulnerable version of whatever web host you were using, or maybe if you had SSH open without rate limits.

  • Fair Fairy@thelemmy.club
    link
    fedilink
    arrow-up
    23
    ·
    8 hours ago

    I’m confused. I never disable root user and never got hacked.

    Is the issue that the app is coded in a shitty way maybe ?

    • Xanza@lemm.ee
      link
      fedilink
      English
      arrow-up
      15
      ·
      7 hours ago

      You can’t really disable the root user. You can make it so they can’t login remotely, which is highly suggested.

        • Xanza@lemm.ee
          link
          fedilink
          English
          arrow-up
          2
          ·
          3 hours ago

          There’s no real advantage to disable the root user, and I really don’t recommend it. You can disable SSH root login, and as long as you ensure root has a secure password that’s different than your own account your system is just as safe with the added advantage of having the root account incase something happens.

          • Possibly linux@lemmy.zip
            link
            fedilink
            English
            arrow-up
            2
            ·
            3 hours ago

            That wouldn’t be defense in depth. You want to limit anything that’s not necessary as it can become a source of attack. There is no reason root should be enabled.

            • Faresh@lemmy.ml
              link
              fedilink
              English
              arrow-up
              1
              ·
              6 minutes ago

              I don’t understand. You will still need to do administrative tasks once in a while so it isn’t really unnecessary, and if root can’t be logged in, that will mean you will have to use sudo instead, which could be an attack vector just as su.

            • Xanza@lemm.ee
              link
              fedilink
              English
              arrow-up
              2
              arrow-down
              1
              ·
              2 hours ago

              Why do like, houses have doors man. You gotta eliminate all points of egress for security, maaaan. /s

              There’s no particular reason to disable root, and with a hardened system, it’s not even a problem you need to worry about…

      • MehBlah@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        6 hours ago

        Another thing you can do under certain circumstances which I’m sure someone on here will point out is depreciated is use TCP Wrappers. If you are only connecting to ssh from known IP addresses or IP address ranges then you can effectively block the rest of the world from accessing you. I used a combination of ipset list, fail2ban and tcp wrappers along with my firewall which like is also something old called iptables-persistent. I’ve also moved my ssh port up high and created several other fake ports that keep anyone port scanning my IP guessing.

        These days I have all ports closed except for my wireguard port and access all of my hosted services through it.

    • cley_faye@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      8 hours ago

      You can’t really disable it anyway.

      Hardening is mostly prevent root login from outside in case every other layer of authentication and access control broke, do not allow regular user to su/sudo into it for free, and have a tight grip on anything that’s executable and have a setuid bit set. I did not install a system from scratch in a long time but I believe this would be the default on most things that are not geared toward end-user devices, too.

  • MonkderVierte@lemmy.ml
    link
    fedilink
    arrow-up
    4
    ·
    9 hours ago

    Yeah, about this; any ssh server that can be run as user and doesn’t do shenanigans like switching user?

  • recklessengagement@lemmy.world
    link
    fedilink
    arrow-up
    24
    ·
    15 hours ago

    This sounds like something everyone should go through at least once, to underscore the importance of hardening that can be easily taken for granted

  • ohshit604@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    22
    ·
    edit-2
    15 hours ago

    I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.

    • Valmond@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      11 hours ago

      I’m interested, how do you expose your services (on your PC I assume) to the internet through wireguard? Is it theough some VPN?

      • Zanathos@lemmy.world
        link
        fedilink
        arrow-up
        7
        ·
        9 hours ago

        Wireguard IS a VPN. He has somehow through his challenges of exposing services to the internet, exposed wireguard from his home to the internet for him to connect to. Then he can connect to his internal services from there.

        It’s honestly the best option and how I operate as well. I only have a handful of items exposed and even those flow through a DMZ proxy before hitting their destination servers.

      • ohshit604@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        3
        ·
        9 hours ago

        VPN’s are neat, besides the fact they’re capable of masking your IP & DNS they’re also capable of providing resources to devices outside a network.

        A good example is the server at my work is only accessible on my works network, to access the server remotely without exposing it directly to the internet would be to use a VPN tunnel.

  • nonentity@sh.itjust.works
    link
    fedilink
    arrow-up
    21
    ·
    16 hours ago

    Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.

    The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.

  • DaCrazyJamez@sh.itjust.works
    link
    fedilink
    arrow-up
    30
    ·
    18 hours ago

    As a linux n00b who just recently took the plunge and set up a public site (tho really just for my own / selfhosting),

    Can anyone recommend a good guide or starting place for how to harden the setup? Im running mint on my former gaming rig, site is set up LAMP

  • Hozerkiller@lemmy.ca
    link
    fedilink
    English
    arrow-up
    39
    ·
    20 hours ago

    I’ve gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn’t know better, instead of pure salt like another site I won’t mention.

    • Tablaste@linux.communityOP
      link
      fedilink
      English
      arrow-up
      28
      ·
      20 hours ago

      I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.

      We all make mistakes!

      • LordCrom@lemmy.world
        link
        fedilink
        arrow-up
        13
        ·
        18 hours ago

        There’s a 40 year I.T. veteran here that still suffers imposter syndrome. It’s a real thing I’ve never been able to shake off

        • pulsewidth@lemmy.world
          link
          fedilink
          arrow-up
          8
          ·
          16 hours ago

          Just look at who is in the White House, mate - and not just the president, but basically you can pick anyone he’s hand-picked for his staff.

          Surely that’s an instant cure for any qualified person feeling imposter syndrome in their job.

  • phx@lemmy.ca
    link
    fedilink
    arrow-up
    20
    ·
    19 hours ago

    Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.

    When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.

  • Punkie@lemmy.world
    link
    fedilink
    arrow-up
    165
    ·
    1 day ago

    Basic setup for me is scripted on a new system. In regards to ssh, I make sure:

    • Root account is disabled, sudo only
    • ssh only by keys
    • sshd blocks all users but a few, via AllowUsers
    • All ‘default usernames’ are removed, like ec2-user or ubuntu for AWS ec2 systems
    • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.
    • Services are only allowed connections by an allow list of IPs or subnets. Internal, when possible.

    My systems are not “unhackable” but not low-hanging fruit, either. I assume everything I have out there can be hacked by someone SUPER determined, and have a vector of protection to mitigate backwash in case they gain full access.

    • feddylemmy@lemmy.world
      link
      fedilink
      arrow-up
      70
      ·
      1 day ago
      • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.

      Gosh I get unreasonably frustrated when someone says yeah but that’s just security through obscurity. Like yeah, we all know what nmap is, a persistent threat will just look at all 65535 and figure out where ssh is listening… But if you change your threat model and talk about bots? Logs are much cleaner and moving ports gets rid of a lot of traffic. Obviously so does enabling keys only.

      Also does anyone still port knock these days?

      • josefo@leminal.space
        link
        fedilink
        arrow-up
        4
        ·
        18 hours ago

        Literally the only time I got somewhat hacked was when I left the default port of the service. Obscurity is reasonable, combined with other things like the ones mentioned here make you pretty much invulnerable to casuals. Somebody needs to target you to get anything.

      • kernelle@0d.gs
        link
        fedilink
        arrow-up
        19
        ·
        1 day ago

        Also does anyone still port knock these days?

        Enter Masscan, probably a net negative for the internet, so use with care.

        • davidgro@lemmy.world
          link
          fedilink
          arrow-up
          7
          ·
          24 hours ago

          I didn’t see anything about port knocking there, it rather looks like it has the opposite focus - a quote from that page is “features that support widespread scanning of many machines are supported, while in-depth scanning of single machines aren’t.”

          • kernelle@0d.gs
            link
            fedilink
            arrow-up
            4
            ·
            23 hours ago

            Sure yeah it’s a discovery tool OOTB, but I’ve used it to perform specific packet sequences as well.

    • DefederateLemmyMl@feddit.nl
      link
      fedilink
      arrow-up
      1
      ·
      8 hours ago

      Do not allow username/password login for ssh

      This is disabled by default for the root user.

      $ man sshd_config
      
      ...
             PermitRootLogin
                     Specifies whether root  can  log  in  using  ssh(1).   The  argument  must  be  yes,  prohibit-password,
                     forced-commands-only, or no.  The default is prohibit-password.
      ...
      
      
    • LordCrom@lemmy.world
      link
      fedilink
      arrow-up
      10
      ·
      18 hours ago

      If it’s public facing, how about dont turn on ssh to the public, open it to select ips or ranges. Use a non standard port, use a cert or even a radius with TOTP like privacyIdea. How about a port knocker to open the non standard port as well. Autoban to lock out source ips.

      That’s just off the top of my head.

      There’s a lot you can do to harden a host.

  • communism@lemmy.ml
    link
    fedilink
    arrow-up
    39
    ·
    24 hours ago

    How are people’s servers getting compromised? I’m no security expert (I’ve never worked in tech at all) and have a public VPS, never been compromised. Mainly just use SSH keys not passwords, I don’t do anything too crazy. Like if you have open SSH on port 22 with root login enabled and your root password is password123 then maybe but I’m surprised I’ve never been pwned if it’s so easy to get got…

    • cmnybo@discuss.tchncs.de
      link
      fedilink
      English
      arrow-up
      27
      ·
      24 hours ago

      By allowing password login and using weak passwords or by reusing passwords that have been involved in a data breach somewhere.

      • communism@lemmy.ml
        link
        fedilink
        arrow-up
        7
        ·
        23 hours ago

        That makes sense. It feels a bit mad that the difference between getting pwned super easy vs not is something simple like that. But also reassuring to know, cause I was wondering how I heard about so many hobbyist home labs etc getting compromised when it’d be pretty hard to obtain a reasonably secured private key (ie not uploaded onto the cloud or anything, not stored on an unencrypted drive that other people can easily access, etc). But if it’s just password logins that makes more sense.

    • pageflight@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      ·
      23 hours ago

      The one db I saw compromised at a previous employer was an AWS RDS with public Internet access open and default admin username/password. Luckily it was just full of test data, so when we noticed its contents had been replaced with a ransom message we just deleted the instance.