How did the year get by me?

Wow, I'm surprised that a year has passed since my last post. Largely, it has been due to my inability to get the editing tools to play well with the iPhone and now the iPad. Normally, I don't update my stuff on employer time, or employer equipment. Couple this with a VERY busy year, and me putting updates lower on the priority list, and here we are on Memorial Day weekend.

A lot has happened in the past 12 months. Too much for a single post, but let me sum up.

I'm working with a variety of Service Processors this year, making the work, and documenting what I've learned. Besides working at the BIOS level, I'm trying to determine if/how the settings can be modified from the OS's involved, and that is more difficult than I supposed. The main problem seems like the websites holding the IPMI drivers or libraries have disappeared after 3-4 years. (The biggest lesson here is to capture, collect, and archive these files for any hardware you acquire. Mergers and acquisitions contribute to the loss of the files, even as the instructions make it into a knowledge base.)

An old lesson re-learned, is that older Sun gear will ignore their serial console settings as soon as you plug a keyboard in! Your only clue, if you are logging the console output with Conserver or some other application, is a small line about detecting a keyboard. Removing the keyboard does NOT revert back to serial console output, you need to reboot the server with no keyboard attached. I knew this long ago, but then was thwarted when folks in a remote office attached a keyboard, and we didn't notice that console I/O had gone away.

I've also been doing a LOT of work with SNMP in the past 6 months, and I found an oddity with an MGE (which sold the line to APC, who then sold the line to Schneider Electric) room-sized UPS. The brains of the UPS uses a serial line to pass a data stream to the Network Interface shelf. If this streaming interface is removed, the SNMP cards do not get an update... But the cards simply report the LAST value for any queried OID. As a result, your only clue of a disconnect is seeing all results "flat-line". BUT, if you are also set up forSNMP traps, you will get also see a string that communications was lost with the UPS. This is a case where you really need to be able to parse Syslog and SNMP traps for interesting strings and send an alert.

Other UPS problems led to the integration of a UPS of Last Resort, so that core devices could track and log events with the big UPS. This included watching the serial output of the APC Smart-UPS 2200XL, plus SNMP integration into our monitoring system.

Another Big Project in the past year was due to a planned network outage. This was due to a planned partial power outage in the building. Unfortunately, some key devices also lost power, causing trouble, which could not send alerts via email because the network was down. We decided that we needed to make sure there was a second path out of the data center for critical alerts and messages. This led to integration with a FoxBox, a Network-connected cell-phone device, allowing SMS alerting AND control!

I've also added many more hosts to the host-to adapter database, but I only have a basic Cyclades console connection guide posted, and I still need to make a Digi page. (Cyclades was another fine example of collecting information from the web as soon as you hear about an acquisition!)

I'll make stronger efforts to post more frequently through the rest of 2010. thanks for reading.

-Zonker-