After a very long break in updates, I’ve made a few changes to my lab and servers. (More on that later…)
In the meantime, I’ve updated our pets page.
- Duane
After a very long break in updates, I’ve made a few changes to my lab and servers. (More on that later…)
In the meantime, I’ve updated our pets page.
- Duane
Last Wednesday my AC didn’t work either because of the load buffer module (SMM) that delays turning the AC on if the generator has been activated. Normally, it keeps the AC from switching on immediately and drawing too much juice when the generator hasn’t fully started. In this case, it failed and defaulted to “open” meaning no power to the AC. Hopefully, Generac will extend warranty work because of the failed units and replaced it with a default “closed” SMM (so the AC won’t be unavailable if it fails again). I called the generator service company and they had a repairman schedule to visit the next day.
That repair was supposed to take “5 – 10 minutes.” The UPS on my homelab server rack is good for 15 minutes or so. Around 12 minutes into the repair, it started beeping like crazy and my whole lab shut off without powerdown. (ESXi VM server which includes media server, automation server, web server and DNS/Pihole VMs plus the SAS array of storage.) Servers don’t like when that happens. It wouldn’t post when I tried to boot. I really, really should have shut everything down in advance… but I wanted to save my over 600 days of uptime on the VM host!
After I got home from a weekend of racing, I spent several hours getting it back up and running by pulling hardware, re-installing VMware to new SD cards (it boots from internal SD cards), running diagnostics on the RAID arrays, etc. Luckily I had the DNS and Web Server VMs backed up to another server and powered that on to cover me for the weekend… but I thought I was screwed. (None of that effort contributed to the solution.)
The fix was to run the Lifecycle Manager… which is a Dell EMC server feature that walks you through server setup and maintenance. I don’t think I used it since I set up the server! It walked through the automated firmware upgrade and it reinstalled BIOS, etc. That unexpected process allowed me to post. I just had to mark all of the disks in my RAID arrays as “good, offline” and re-scan the foreign configuration and I was back online.
Lessons learned:
Like any other hobby, running a homelab can be a lot of work. I’ve become VERY dependent on the main VMs that run on my host. Most of the experimental hosts could be lost and I could easily start over with them to play with Docker and K8s, etc. I’ll have to make the routine maintenance part of the “fun” of my homelab, too.
- Duane
I’ll be honest with you, this post is going to be challenging for me to write and for some to read. I’ll do my best if you do the same. Let’s get started:
When I last updated, I explained that we had a lot going on. As it turns out, I didn’t know the half of it. In October of 2017, both of my parents were celebrating Halloween, handing out candy and commenting on everyone’s costumes with our family in my brother’s house. They still lived in their home that my grandfather built. 15 months later… my mom has passed away and dad is alone in assisted living.
Within a month, my parents’ health began to worsen in alternating increments. The first big scare was Christmas morning (2017) when my mother was too weak to get out of bed and ended up in the hospital with a UTI. A few months later and we were taking her to urgent care because she “fell and hurt her knee”… in retrospect, she probably had a small stroke. We took her to the ER that night (another UTI, too) and decided in the ER with the whole family that it was time to find an independent living community.
We briefly pursued this direction a few years earlier, but my father was very resistant. They had too much “stuff” and weren’t ready to part with it. In hindsight, the additional care might have improved their quality of life significantly. This time, everyone was very serious about the endeavor. We kept my mom apprised of our progress while she get better in the hospital. We established a budget and timeline, then researched a few communities. After a handful of visits, we arranged a company to move them and their 2 pets. In just 3 weeks, they were living in a moderately-sized 1-bedroom apartment on the ground floor about 15 minutes from their house. At first, it seemed promising. The meal plan and care from the staff helped, but my parents continued to “take turns” in the hospital each needing care a few times over the next 3 months. The peak of this situation occurred mid-summer, 2018. My father was hospitalized (and later in long-term physical rehab) for a serious infection for almost 2 months while my mom was at the apartment alone. We maxed out the care program offered by the community, but the staff wasn’t trained to the level of care she truly needed. We didn’t realize that her dementia/Alzheimer’s had progressed to moderate. She wasn’t drinking enough water, not eating enough food and just not taking care of herself. When we visited, she put on a great show. We didn’t know how much help she really needed.
In late August, the day before my father returned to their apartment, my mom went to the ER by ambulance. She was having a dementia-related panic attack exacerbated by confusion from yet another infection. This was the first time mom looked really bad in the hospital… but she appeared to get better quickly, even joking in the hospital. Her care was extended by a stay in a rehab facility near their apartment. Eventually, she was even walking (with help) and making plans for Halloween in a few months. She was in the hospital and rehab for 5 weeks.
During my mom’s recovery, our family decided to upgrade my parents’ care to assisted living with guidance from the hospital staff. We identified a handful of assisted living locations and decided to move to a community very close to their house (which were just finishing cleaning out to prepare for sale). The cost was significantly greater, but the level of care increased accordingly. The staff is amazing, their apartment is very nice (2 bedroom to accommodate all of their stuff) and their pets were welcome.
Things seemed to calm down for a few weeks, but it was obvious that mom’s dementia was worsening exponentially. In the evenings she was more confused and prone to repeatedly calling out. For years, she could use an iPad for crossword puzzles, video calls, email and web browsing… now she had trouble turning it on or figuring out how to charge it. The staff tried to monitor her fluid intake, but could only do so accurately during meals. Mom, always the social perfectionist, continued to put on a good show in the dining hall.
Mom’s plans for Halloween would never happen: on October 30th mom was back in the hospital. Their new apartment was closer to their “old neighborhood” and allowed for mom to be transported to a familiar hospital nearly around the corner from my brother’s house. We didn’t know then how much time we would be spending at that hospital, but were very glad to have the care, guidance and support of the staff at Beaumont in Grosse Pointe. This stay was different: mom was having trouble swallowing and the speech therapist wasn’t able to get her to follow instructions consistently. We discussed moving mom to the memory care area of their community for closer monitoring and therapy. She was in the hospital/rehab through November, but we were able to drive her to their community for Thanksgiving dinner. This time, mom went back to the hospital (followed by rehab) briefly before returning home.
We wanted to celebrate mom’s 80th birthday a week-or-so before Christmas, but realized that her health was very complicated and unstable… and had to call it off when she returned to the hospital 3 days before her birthday. Again, this stay in the hospital took its toll on mom. We didn’t know it, but the repeated infections weakened her to the point where she couldn’t fight off pneumonia. It just wasn’t detected with tests or examinations and she went home (against protest) instead of rehab.
A week later (Jan 2) and Mom was in the ER with respiratory failure. We all rushed to the hospital and the staff explained that she needed a breathing tube. Her kidneys weren’t working well and her (existing) heart condition was the root cause. I can still remember the minute-by-minute details, but will spare you. After a few days, her condition improved slightly and the breathing tube was removed, but her respiration never fully recovered. Her other organs were not working correctly. At the same time, my father was admitted to the hospital. With all of the stress, he wasn’t taking his medications and had to be admitted. This turned out to be a blessing: he was able to spend time with my mom.
While on the topic of blessings, climate change provided us with oddly warm weather so a smooth 50+ mile commute each morning for updates from the ICU staff was possible. The construction on ALL of the expressways between our house and the hospital wrapped up, further shortening travel time. Mom was able to recognize all of us until she became unresponsive and that time meant a lot to us.
Gina and I went home to have dinner and rest for a night after a few days at the hospital, but we got a call that mom had taken a turn. I went to the hospital and joined by brother and dad. The doctors told us that we only had a few days (at most)…
Mom’s pastor was out of the country, so our pastor and his wife visited, talked to the kids and spent time with us. We moved to a peaceful room and my brother and I stayed with mom until she passed peacefully. We cried a bit, but were grateful for the time we had with her. It was hard telling my dad, but he was able see her one more time, too. We started to make arrangements and my dad’s health improved enough to go to a nearby rehab for a few weeks. He’s home now and we’re preparing for mom’s memorial this weekend.
I never prepared for my parents to be “old” or get sick. I knew one day they would be gone and tried to brace myself, but caring for them and putting together a plan was more than I expected. The past 15 months have been a roller coaster of work, fear, worry, stress, sadness and (sometimes) relief. We should have planned better but luckily my mom had clear instructions for us to follow as her health declined. We’ve learned a lot and still have a lot to take care of, but I want to share some of the basics in case they help others now that I’ve had some time to reflect and gather my thoughts:
Our family grieves in waves. Happy memories outnumber the difficult ones from recent months… but those memories are most fresh and both bring tears. In time, I know the hard times will fade and happy thoughts will once again take over. I crafted a pair of photo boards for my mom’s memorial service and this helped me more than I expected. We trying our best to keep dad happy; his grand-kids are a constant source of support and new, happy memories. We are working with him to improve his health so that he can more actively participate in our family events and, hopefully, outings. We’ve been dreading the future for so long. It’s nice to look forward to things again and hope for continued improvement and family time.
- Duane
Well, no racing for us this year. The combination of a 1-year old and new engine/new racing class didn’t really work out. We’ve been able to spend some quality time with Kara at the state and metro parks, plus a few great trips to Northport, MI. The weather has been nice, but my work schedule (especially having my home office in the Mountain time zone) has severely limited my time outside during the week. Hopefully this will ease up soon in my new-ish role (promoted back into Sr. Management again: Sr. Director).
I’ve done a lot of work cleaning up our workshop this year. I’ve meant to clean it up and prepare to insulate/heat the shop so I can install the new engine in my racecar… but it’s turned into a lot of cleanup and corrections for lack of ventilation from our new roof (circa 2009). I also spent a few hours replacing the suspension and lifting my Jeep due to a need for new tires and shocks.
I’ve been keeping up with my retro computer collection: everything is still working and I find an few hours per week to enjoy my Amigas, PCs, Macs and Commodore machines. I’m almost through my Leisure Suit Larry bucket list: wrap up the series. I’ve owned about 3 or 4 versions of the game series (or parts of it) over the years and it’s time to finally wrap it up! (I took a short break for Space Quest and Police Quest in there, too.) I added a SX-64 Executive "e;portable" to my collection earlier this year and have had my eye on few Apple IIc computers, but the price or condition haven’t been up to snuff.
Most recently, I’ve been working on capturing/digitizing a few boxes of 8mm/Hi8/DV that literally fell out of my closet a week ago. I’ve found some amazing stuff on those tapes: lots of high school wrestling, some hanging out with friends, and an emotional collection of holiday footage with many family members that have since passed. The process is relatively slow as I must capture each tape in real-time. However, my setup is surprisingly effective: a Sony Hi8 camera connected via analog S-Video and Audio (when capturing 8mm and Hi8) to my Sharp DV Camera, connected via Firewire 800 to an Apple Thunderbolt Lightning Cinema Display, connected to my 2011 Macbook Air via Thunderbolt with an external Thunderbolt drive for storage. Both cameras are time base corrected resulting in decent quality considering the source media. The capturing process is taking a long time, but it’s mostly hands-off… the real work will be sorting, editing and finalizing for DVD (all source is 480p at best). Some of the content might make it to YouTube, but I’ll likely keep it private for friends.
That sums up a lot from the past few months. Seems like each season I hope to make more regular updates to my blog, but I end up doing this instead. More to come?
- Duane