This morning I woke up to my phone using mobile data and my home assistant automations not working. Initially I thought it the power was out, but I could turn on the lights just fine. I checked my UniFi app and saw that the server was not connected to the network at all. This meant that the cable got unplugged, the switch isn’t working, or the server isn’t working. It said the switch was connected and another device was connected to the switch so that narrows it down to just 2 cases. So I opened my server closet in the basement and immediately noticed something was wrong. I couldn’t figure out what was wrong but I just felt like something was wrong. Everything was plugged in, the network switch lights were blinking like normal, my raspberry pi was running just fine, even the server indicator lights were on. My main server is an old gaming PC so it has a glass side panel so I looked inside and I could see the fan spinning, but I could not hear it. Usually I have it set to full speed and I can hear full speed very well. I tried rebooting the server with the power button and the fans didn’t go to full speed. As a last resort, I brought down a keyboard and monitor. As soon as I plugged in the monitor, I saw that there was a prompt to set the time on the BIOS! Picture of the prompt In my opinion, this was the stupidest reason for an outage.

Further investigations

I dug a little deeper and discovered that the BIOS had been reset during a power outage right before all of this happened. So far I have consulted the motherboard manual and found absolutely nothing about this. After a bit of research, I think it could have been that the CMOS battery has died. This is a really simple fix but I don’t have the replacement battery right now. This means that I will have the same exact issue after the next power outage unless I replace the battery.

Preventing this in the future

From what I can see, I just need to replace the CMOS battery. But this computer has been running for over 4 years, so what is stopping this from happening again around 2028? The most effective solution is going to be preventing power outages in the first place. This can be done using a battery backup or a standby generator. Standby generators will last longer during a power outage but are typically more expensive and harder to setup than a simple battery backup.

  • Dave.@aussie.zone
    link
    fedilink
    English
    arrow-up
    118
    arrow-down
    1
    ·
    edit-2
    2 months ago
    1. Replace CMOS battery.
    2. Get small UPS.
    3. Discover that small UPS’s fail regularly, usually with cooked batteries.
    4. Add maintenance routine for UPS battery.
    5. Begin to wonder if this is really worth it when the rest of the house has no power during an outage.
    6. Get small generator.
    7. Discover that small generators also need maintenance and exercise.
    8. Decide to get a whole house battery backup a-la Tesla Powerwall topped off by solar and a dedicated generator.
    9. Spend 15 years paying this off while wondering if the payback was really worth it, because you can count on one hand the number of extended power outages in that time.
    10. In the end times a roving band of thugs comes around and kills you and strips your house of valuable technology, leaving your homelab setup behind and - sadly - without power. Your dream of unlimited availability has all been for nought.

    Conclusion: just replace the CMOS battery on a yearly basis during planned system downtime.

    • catloaf@lemm.ee
      link
      fedilink
      English
      arrow-up
      18
      arrow-down
      1
      ·
      2 months ago

      CMOS batteries last a lot longer than a year. Unless the system has been unplugged for a long time, they should be good for several years. I’m sure there’s actual data out there somewhere.

      But yeah, a lot of people think “oh I’ll just put a UPS on it”. They don’t consider that unless you get a really big UPS, they’re only good for very short outages, seconds or minutes, to bridge the gap between the outage and your generator coming on (or the mains power coming back if it was just a flicker).

      Also, the batteries in them need to be replaced every 3-5 years.

      • Dave.@aussie.zone
        link
        fedilink
        English
        arrow-up
        7
        ·
        2 months ago

        Yeah , it’s really a little strange in OPs case, I can’t really recall changing a CMOS battery in ages, like decades of computer use.

      • yonder@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        4
        ·
        2 months ago

        I bought a small APC UPS about a year ago and am glad I did. In my area, very brief outages are somewhat common so a small UPS will work for the majority of outages.

    • rambos@lemm.ee
      link
      fedilink
      English
      arrow-up
      10
      ·
      2 months ago

      What about UPS just for CMOS battery? And a tiny diesel generator with 60 ml tank

      • poVoq@slrpnk.net
        cake
        link
        fedilink
        English
        arrow-up
        4
        arrow-down
        2
        ·
        2 months ago

        Currently at step 9. Waiting for the roving bands of thugs to arrive 😅

    • umbrella@lemmy.ml
      link
      fedilink
      English
      arrow-up
      6
      ·
      edit-2
      2 months ago

      hey, AI defense turrets sound like a cool and totally not dangerous project. hope you have a decent GPU in your setup, good luck.

  • hendrik@palaver.p3x.de
    link
    fedilink
    English
    arrow-up
    5
    ·
    2 months ago

    If you’re having frequent power outages, you might consider buying an UPS. Other than that, I’d just buy a new CMOS battery every few years. Mine seem to last way longer than 4 years. Maybe 8y or so.

  • superkret@feddit.org
    link
    fedilink
    English
    arrow-up
    5
    ·
    2 months ago

    From what I can see, I just need to replace the CMOS battery. But this computer has been running for over 4 years, so what is stopping this from happening again around 2028?

    Replace the CMOS battery again.

  • Sparky@lemmy.blahaj.zone
    link
    fedilink
    English
    arrow-up
    4
    ·
    2 months ago

    If you’re the owner of the website, I suggest you look into those php errors

    
    Warning: simplexml_load_file(https://feeds.thefossrant.com/rss_feed.xml): failed to open stream: php_network_getaddresses: getaddrinfo failed: No address associated with hostname in /var/www/html/article.php on line 2
    
    Warning: simplexml_load_file(): I/O warning : failed to load external entity "https://feeds.thefossrant.com/rss_feed.xml" in /var/www/html/article.php on line 2
    
    Warning: Invalid argument supplied for foreach() in /var/www/html/article.php on line 5
    
    Warning: Invalid argument supplied for foreach() in /var/www/html/article.php on line 19
    
    Published on: 1970-01-01 00:00:00 UTC
    
    Tags:
    Image unavailable
    
    
  • EarMaster@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    ·
    2 months ago

    Some BIOS manufacturers allow you to disable all halts on errors. As soon as you connect to the network your system should be able to sync its internal clock.

    • MangoPenguin@lemmy.blahaj.zone
      link
      fedilink
      English
      arrow-up
      7
      ·
      2 months ago

      Some BIOS manufacturers allow you to disable all halts on errors.

      That will be reset to default if the CMOS battery is dead and power is removed though.

  • Possibly linux@lemmy.zip
    link
    fedilink
    English
    arrow-up
    3
    ·
    2 months ago

    Why you are at it you might want to check the battery in your smoke alarm. I think your are support to change the smoke alarm battery every 5 years.

    • crusty@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      2 months ago

      The battery in smoke detectors should be changed every year and you should ideally test the detectors using the little button monthly.

      In Norway we have an official smoke detector day which is on December 1. where we’re supposed to change the batteries. Its because Christmas is the high season for house fires.

      • Bronzie@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        1
        ·
        2 months ago

        Just took over a house last year with smart detectors connected to a homebase, all Li-Ion. Can see the battery percentage in the app. No more December 1st checks for a long time!

      • Possibly linux@lemmy.zip
        link
        fedilink
        English
        arrow-up
        1
        ·
        2 months ago

        It is 5 years in the US as it says it on the box. The battery can last 10 years but that is way to long from a safety perspective.

        • crusty@lemmy.dbzer0.com
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          2 months ago

          Lithium batteries last 5-10 years, regular 9v alkaline batteries last about 12-18 months. Some sources recommend changing the batteries every 6th month but i personally think that’s a bit excessive.

          I only have a small 2 room apartment so i just got myself a cheap-ish optical smoke detector with a built in lithium battery. The battery should last 10 years which is how long smoke detectors last before they should be replaced.

  • MangoPenguin@lemmy.blahaj.zone
    link
    fedilink
    English
    arrow-up
    2
    ·
    2 months ago

    Honestly just replace the CMOS battery on a schedule if it’s a big deal, a UPS is nice to have but it doesn’t really solve that specific problem.

  • runesmite@lemmy.runesmite.com
    link
    fedilink
    English
    arrow-up
    2
    ·
    2 months ago

    I agree with you about an uninterruptible power supply. While a computer’s Power supply can normally handle about half a second of the wrong voltage, a UPS should provide you with minutes to hours of run time. While that won’t solve the CMOS going bad, it would probably reduce the amount of times a power outage would last long enough to shut down the computer - though that absolutely depends on the reliability of power in your area.

    Another option would be to look at different motherboard options. AsRock Rack (and I believe Supermicro) offer boards with BMCs, so you’d be able to change your BIOS settings from anywhere with a virtual KVM. They’re certainly pricier options, but a lovely convenience to have.

  • WasPentalive@lemmy.one
    link
    fedilink
    English
    arrow-up
    2
    ·
    2 months ago

    Can you replace the CMOS battery with a supercapacitor that is kept charged? This should not need to be replaced every 4 years, I think.

    • Bronzie@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      1
      ·
      2 months ago

      We use these in our products at work and I’ve never heard about them failing before the equipment is replaced anyways. They are soldered.

      I won’t quote life expectancy (MTBF) but we have many customers with 15-20 year old PLC’s with the original capacitor.

    • tal@lemmy.today
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      2 months ago

      I once worked on a product that you really did not want to have not coming back up. I was on it several years after the original engineers had designed an early model. Said engineers had not tested what happened when the CMOS battery died and triggered a reset of BIOS settings, brought it back to the hardware platform’s default state. When it did, the thing entered a non-bootable state. You could, with a serial port, access the BIOS and fiddle the settings back for one good boot…but the CMOS battery was non-removable, soldered to the motherboard. Our manufacturing process had not involved changing the default BIOS settings, just what was stored in CMOS. Oops.

      IIRC our customer care guys just sent out new models for free to affected customers – the original hardware model wasn’t sold in large volume, and the cost of the actual hardware components wasn’t especially large relative to the cost of the product.

      I had one sitting around on my desk, as it was sometimes handy to have a physically-accessible device to do work on. I rolled down to Radio Shack – yes, this was a few years back – got a removable CMOS battery case, stripped the non-removable battery out, soldered the battery case to the motherboard, and had the only instance of the device out there that could take a fresh CMOS battery.

  • knobpolisher@feddit.nl
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    I wonder… How about a hot swap cmos battery? Thinking you wire the cmos terminals to a few batteries in parallel Or 3 And add a voltage indicator for each of them