I know, I know, clickbaity title but in a way it did. It also brought in the situation in the first place but I’m just going to deliberately ignore that. Quick recap:

  1. I came home at 3pm from the city, my internet at home didnt work.
  2. checked multiple devices, phones worked out of wifi, I figured I need to restart the router
  3. I login to the router and it responds totally normal but my local network doesnt. (Its always dns, I know)
  4. I check the router log and see 100s of login attempts over the past couple of days.
  5. I panic and pull the plug, try to get into my server by installing an old monitor, works, many errors about dns
  6. Wife googles with her phone, seems I had https login from outside on and someone found the correct port, its disabled now
  7. Obviously, local network still down, I replug everything and ssh into the server which runs pihole as dns
  8. pihole wont start dns, whatever I do
  9. I use history and find I "chmod 700"ed the dns mask directory instead of putting it in a docker volume…
  10. I check the pihole.log, nothing
  11. I check the FTL log, there is the issue
  12. I return it to 777, everything is hunky dory again.

Now I feel very stupid but I found a very dangerous mistake by having my lan fail due to a less dangerous mistake so I’ll take this as a win.

Thanks for reading and have a good day! I hope this helps someone at some day.

  • carzian@lemmy.mlEnglish
    5·
    8 months ago

    Did you expose your router login page to the open internet? How’d they get access? Why are you chmoding anything to be 777?

    • haui@lemmy.giftedmc.comOPEnglish
      2·
      8 months ago

      There was an option that I had enabled years before and forgotten so yes, I didnt know but it was, on some obscure port.

      And yes, pihole in docker makes its files be 777 which is pretty disgusting, I know. Thats why I tried to make it 700 and broke my whole network.

      • 1984@lemmy.todayEnglish
        1·
        8 months ago

        I think you are still learning… What you say doesn’t make sense, so I think you may have misunderstood what happened.

        • haui@lemmy.giftedmc.comOPEnglish
          01·
          8 months ago

          Imo we are all constantly learning. Otherwise we stagnate. What I say makes perfect sense, you just dont get it. So let me explain it again, in more detail:

          I was going through my docker compose files to sanitize them and upload them to my private forgejo instance.

          While doing that I found a directory in my filesystem, a remnant of the early days of my server where my knowledge was severely more limited, that was a docker volume mapped to a regular directory, something I wouldnt do today for something like this.

          It was owned by root:root and had 777 permissions which is a bad idea imo. So I changed it to 700 since I dont think I had any other users in group root and others, well.

          Nothing bad happened, until today when my unattended backups triggered a restart at noon and the tragedy started. I put it back for now to 777 but I‘ll try and integrate it in a real docker volume which resides in the docker folders.

          • Auli@lemmy.caEnglish
            1·
            8 months ago

            Well I’m running Pihole in docker and don’t have 777 on anything.

              • Domi@lemmy.secnd.meEnglish
                3·
                8 months ago

                I don’t run Pi-hole but quickly peeking into the container (docker run -it --rm --entrypoint /bin/sh pihole/pihole:latest) the folder and files belong to root with the permissions being 755 for the folder and 644 for the files.

                chmod 700 most likely killed Pi-hole because a service that is not running as root will be accessing those config files and you removed their read access.

                Also, I’m with the guys above. Never chmod 777 anything, period. In 99.9% of cases there’s a better way.

                • haui@lemmy.giftedmc.comOPEnglish
                  1·
                  8 months ago

                  Thanks for checking that. I will change the permissions accordingly and restart pihole to check if it works. Probably later today.

      • Possibly linux@lemmy.zipEnglish
        3·
        8 months ago

        Because you don’t have a way to know what’s been compromised. Take your data only and make sure to verify nothings been tampered with.

        Trust me it will be better in the long run.

        • haui@lemmy.giftedmc.comOPEnglish
          01·
          8 months ago

          Yeah, I dont feel like setting up a whole cloud infrastructure on a hunch. I‘m running like 15 different services and they are all compartmentalized. It would take weeks to reset all this. So far nobody got anywhere from what I can see.

          • khorak@lemmy.dbzer0.comEnglish
            1·
            8 months ago

            One word of advice. Document the steps you do to deploy things. If your hardware fails or you make a simple mistake, it will cost you weeks of work to recover. This is a bit extreme, but I take my time when setting things up and automate as good as possible using ansible. You don’t have to do this, but the ability to just scrap things and redeploy gives great peace of mind.

            And right now you are reluctant to do this because it’s gonna cost you too much time. This should not be the case. I mean, just imagine things going wrong in a year or two and you can’t remember most things you know now. Document your setup and write a few scripts. It’s a good start.

            • haui@lemmy.giftedmc.comOPEnglish
              11·
              8 months ago

              I get your point. Ansible is quite interesting too. I do document most of the things I do but I have to admit I have been slacking a bit, recently. There is just so much stuff that needs doing and a lot of interesting projects to learn about that sometimes stuff gets forgotten.

              My personal impression of the linux space is still that folks get dumped on by the community for not being immersed in the nitty gritty though.

              Thats neither fun nor will it work to get more people interested in linux. People make mistakes, learn to help without judging.

              Have a good one.

            • haui@lemmy.giftedmc.comOPEnglish
              01·
              8 months ago

              Wow, a lot of people would set up a new server because of intrusion attempts in a log i guess. If I did that in a job I‘d get fired for doing nothing else but resetting everything every week.

              As an admin, you have to keep the CTO from using „master“ or „admin“ as the ssh password on a production server. Just so you know what level of stupidity makes the big bucks out there.

              • prettybunnys@sh.itjust.worksEnglish
                1·
                8 months ago

                As an admin I’d question why the CTO has a login on a production server.

                You would do well listening more when you ask for advice.

          • teawrecks@sopuli.xyzEnglish
            02·
            8 months ago

            You’re saying you see a bunch of login attempts on your router, but you don’t think they actually got into it?

  • Decronym@lemmy.decronym.xyzBEnglish
    2·
    8 months ago

    Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

    Fewer Letters More Letters
    DNS Domain Name Service/System
    Git Popular version control system, primarily for code
    VPN Virtual Private Network
    VPS Virtual Private Server (opposed to shared hosting)

    4 acronyms in this thread; the most compressed thread commented on today has 11 acronyms.

    [Thread #616 for this sub, first seen 20th Mar 2024, 00:15] [FAQ] [Full list] [Contact] [Source code]