Assiging DHCP addresses to APC Power Strips

This is certainly a post related to serial consoles. But, you are limited in the amount of useful monitoring data that you can pull via the CLI, so folks often want to use SNMP or SYSLOG to monitor these devices. Usually, that's where they want CLI access.

In my case, we are re-numbering the management subnet, and we've decided to try to make this job easier for the next time a re-number is required. We're telling the devices to use DHCP addressing, and using MAC Address reservations for all of the devices that we care about. Two complications arose; How to connect to the serial CLI, and how to set the device to use DHCP.

The complication with using DHCP addressing on the AP-7868 PDU (and likely many other APC devices), is that their default configuration REQUIRES that your DHCP server set option 43 to a specific vendor code. If you don't do this, the PDU will not accept the DHCP Offer from your server.

APC AP-7868 Smart PDU Serial Pinouts

APC RJ-13 serial console, 4-wire, data-only connection
APC Console     Lantronix TS (Cisco, Opengear with Cisco wiring, etc.)
(RJ-13)         (RJ-45)
1  ---(???) nc            default settings:  9600-8-N-1
2  ---(GND)---  5
3  ---(TxD)-->  6         default username:  apc
4  <--(RxD)---  3         default password:  apc
5  ---(GND)---  
6  ---(???) nc 
( nc = No Connection )

NOTE: You must configure a default gateway for these devices (APC-7868), even if you are only talking on your own subnet!

On your Conserver host

This section includes what you need to type on the CLI console, to configure the PDU to ignore Option 43 and accept the DHCP Offer. I use the spare port on one of my console servers to do this, and I always name my unused ports in my Conserver deployments, so that I can use them at a moments notice. In this case, I'm using port 30, on Console Server 5...

NOTE: the CLI is not good about sanitizing user input, or detecting errors! You can type commas (or TEXT!) in an IP address field, and non-standard TLDs in the domain field! Be very careful about your typing, and visually check your data before accepting the changes!

console unused-cs5-30     (this is the port I picked...)
[Enter]
[Enter]     (twice, to wake-up the CLI interface)
apc [Enter]     (default login name: apc  case-sensitive!)
apc [Enter]     (default password: apc  case-sensitive!)
2 [Enter]     (Network)
1 [Enter]     (TCP/IP)
(If you need the MAC address, you'll find it here)
4 [Enter]     (Boot Mode   only if currently set for Manual...)
2 [Enter]     (DHCP Only)
2 [Enter]     (Advanced)
1 [Enter]     (Device Name)
ps-6.12i [Enter]
2 [Enter>     (Domain Name
garage.com [Enter]
8 [Enter]     (DHCP Cookie Is: ...)
1 [Enter]     (Not Required to accept offer)
9 [Enter]     (Accept the pending changes)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
4 [Enter]     (logout)

When you log out of the PDU, the device will reload the TCP stack, and in a few minutes, your PDU should accept the new IP address via DHCP.

Configuring your DHCP Server for Option 43

You need to define you option 43. How (or IF) you can do this will depend on your DHCP server software. But, this section includes the particular HEX characters that the APC PDU expects to see (or it will refuse the DHCP Offer).

These clues are for the Internet Software Consortium (ISC) DHCP server. See the dhcp-options man page - this describes all the standard DHCP options. Look for the section titled "VENDOR ENCAPSULATED OPTIONS", which is option 43. If you already have a class for APC units, then that would be a good place to define option 43 as it will be defined for all devices that match the class. The simplest way to define the value is something like this:

     option vendor-encapsulated-options 01:04:31:41:50:43;

DHCP option names and numbers are listed in RFC1533, and in the ISC DHCPd source file common/tables.c

(Explanation: I forget the first character (01), but I believe that it designates to expect HEX characters. The second says there are (4) characters in the field. The last three characters are ASCII for "APC". I don't know why the "31" is there...)

NOTE: All APC PDU's should be protected by UPS power.

In an unprotected environment, if the utility power flickers off and on quickly, the onboard network card may not power back on. In these instances, the PDU may power on but the LED display will be blank and any type of access (web, telnet, console) to the PDU will be disabled. In this case, you must power cycle the PDU again, by pulling the power cord, waiting at least 5 seconds, and re-applying power.

The on-board network card was not designed for quick off/on scenarios. Customers protecting their PDU with a UPS, as designed, will not have this problem.

There are also some APC UPS clues on my console site, http://www.conserver.com/consoles/Clues/cons-apc.html
 
Regards,

-Zonker-

How did the year get by me?

Wow, I'm surprised that a year has passed since my last post. Largely, it has been due to my inability to get the editing tools to play well with the iPhone and now the iPad. Normally, I don't update my stuff on employer time, or employer equipment. Couple this with a VERY busy year, and me putting updates lower on the priority list, and here we are on Memorial Day weekend.

A lot has happened in the past 12 months. Too much for a single post, but let me sum up.

I'm working with a variety of Service Processors this year, making the work, and documenting what I've learned. Besides working at the BIOS level, I'm trying to determine if/how the settings can be modified from the OS's involved, and that is more difficult than I supposed. The main problem seems like the websites holding the IPMI drivers or libraries have disappeared after 3-4 years. (The biggest lesson here is to capture, collect, and archive these files for any hardware you acquire. Mergers and acquisitions contribute to the loss of the files, even as the instructions make it into a knowledge base.)

An old lesson re-learned, is that older Sun gear will ignore their serial console settings as soon as you plug a keyboard in! Your only clue, if you are logging the console output with Conserver or some other application, is a small line about detecting a keyboard. Removing the keyboard does NOT revert back to serial console output, you need to reboot the server with no keyboard attached. I knew this long ago, but then was thwarted when folks in a remote office attached a keyboard, and we didn't notice that console I/O had gone away.

I've also been doing a LOT of work with SNMP in the past 6 months, and I found an oddity with an MGE (which sold the line to APC, who then sold the line to Schneider Electric) room-sized UPS. The brains of the UPS uses a serial line to pass a data stream to the Network Interface shelf. If this streaming interface is removed, the SNMP cards do not get an update... But the cards simply report the LAST value for any queried OID. As a result, your only clue of a disconnect is seeing all results "flat-line". BUT, if you are also set up forSNMP traps, you will get also see a string that communications was lost with the UPS. This is a case where you really need to be able to parse Syslog and SNMP traps for interesting strings and send an alert.

Other UPS problems led to the integration of a UPS of Last Resort, so that core devices could track and log events with the big UPS. This included watching the serial output of the APC Smart-UPS 2200XL, plus SNMP integration into our monitoring system.

Another Big Project in the past year was due to a planned network outage. This was due to a planned partial power outage in the building. Unfortunately, some key devices also lost power, causing trouble, which could not send alerts via email because the network was down. We decided that we needed to make sure there was a second path out of the data center for critical alerts and messages. This led to integration with a FoxBox, a Network-connected cell-phone device, allowing SMS alerting AND control!

I've also added many more hosts to the host-to adapter database, but I only have a basic Cyclades console connection guide posted, and I still need to make a Digi page. (Cyclades was another fine example of collecting information from the web as soon as you hear about an acquisition!)

I'll make stronger efforts to post more frequently through the rest of 2010. thanks for reading.

-Zonker-

Look at all the pretty lights

This week, I needed to reconfigure a previously-used Cyclades ACS console server. Finding the right adapters was easy enough, but getting the USB serial dongle to play was the tricky bit. Without a signal tracer, I was stuck with the evidence... "it's not working"...and I was trying different hardware combinations, and a few software changes. I wasted 2.5 hours going through different combinations, and I put a reminder in my calendar to bring my signal tracers and my Serial Doctors Bag the next day.

Once I had my trusty signal tracers, it was a matter of 10 minutes to find the proper USB settings, COM port configuration in my terminal emulator, and start the reconfiguration process. Blinky lights will save you time, every time. Good quality signal tracers are worth a high cost, because they will save you time often, and serve you well for many years. One day, blinky lights will probably save my life...

So, having solved Problem #1, and with kit in hand, I tackled Problem #2;

Years ago, a couple of hundred APC smart power strips (OK, "0-RU, metered Power Distribution Units") were installed in a data center. Fast forward to last week, when one unit is failing to respond to SNMP after a high-temperature 'event' in the room. Since it was in critical service, we wanted to try using the serial console to see if the brains were still alive. Unfortunately, nobody could find a serial cable for the job, and APC doesn't want folks making their own cables so APC doesn't post any signaling clues. I did find one website with clues about the wiring for the 940-0144 cable.

The clues indicated that the 6-pin RJ-14 interface was, indeed, RS-232 (and not TTL, for example), I could do some experimenting. A new toy in my Doctors Bag (not in the pictures (yet), is a MODAPT from Siemon. With this, and my trusty Digital Volt Meter (DVM), I was able to confirm some signalling that wasn't mentioned in the LuxNET web page.

Pin1 -2v (in reference to Pin2)
Pin2 GND (0v potential to Pin5)
Pin3 TxD output (-5.5v in reference to Pin2)
Pin4 RxD input (0.2V in reference to Pin2)
Pin5 GND (0v potential to Pin2)
Pin6 -2v (in reference to Pin2)

With this information, I could then make an adapter cable, and then use the RJ45 signal tracer to confirm that the signals are on the right pins. (I found that I could use just 3 wires, Ground - RxD - TxD) for communications to the APC AP7868 rPDU. I did not need to loop RJ-14 pins 1 and 6, and I didn't need to connect RJ-14 pins 1 or 6 to ground to establish communication.)

I'll update my APC console page and my Signals page soon with some of this information.

APC Serial Ports, and websites, wikis, and backups, oh my!

How many of you use APC brand UPS gear? Raise your hands! Ok, you can put your hands down.

Now, how many of you have connected the UPS serial port to your computer, to get power information, or to have the UPS tell your computer to shut down? Not so many? How many tried, but had trouble? 

Question 1: Were you using the APC-supplied cable (with the correct end plugged into the UPS)?

I ask, because the APC serial ports are unusual! If you use one of the cables you have lying around your shop, there is a good chance that the UPS shut down, and would not turn on again until you disconnected the cable. 

When I first tried this, the cable was missing, and I tried various cables in the shop (because I didn't have passive signal tracers at the time). Nothing worked, and I sometimes lost power. 

When I had another UPS, with the APC cable, I found that the cable wasn't long enough, and some of the cables I tried using to extend the run caused troubles. That's when I dug deeper into the problem. Here's what I've found;

• The APC port puts data and ground on unusual pins!
  
• The UPS uses the two outputs and inputs for other purposes!

You can actually make a "Big Red Switch" for your small APC, using a couple wires and the DE-9 connection. Because of the signals being on unusual pins, when you plug in a PC with a non-APC cable, you are actually applying a "shut-down" hard-wired signal to the UPS.

I made an APC Clues web page, documenting the findings, in late 2007. But, for some reason, I guess I didn't post it. Only when I needed the information again (and couldn't find it) did the panic set in, because some of the legacy information is no longer on the web.

That brings us to....

Question 2: How many of you are making backups of your critical files?

You can replace the word "critical" with Photos, Homework, Research, Family Tree, Tax Info, you get the idea. The data that you do not want to lose in a hard disk crash, or on a lost Flash Memory Drive. Where do you keep it, and how often do you update it?

I use an automated system, backing up critical files to a disk "in the cloud", for $99 a year. This runs every time I start up my computer, and I put a lot of faith in it. But, I need to pay money to keep it alive.

I also make manual copies to a 1 GB external hard disk (but I have to remember to move new files). I should really set this up to run automatically.

For some of my hobbies, and console stuff, I put copies of the web. (Technically, I made them for display on the web, so I guess the copies on my home machine are my backups.)

For some data sets (photos for some of my hobbies or family, or my console-related files, for example) will be occasionally copied to a CD or a DVD, and distributed to friends, family, or colleagues. While it isn't complete, this may be one of my best backup methods!

My point here is that disks die, memory sticks get lost, laptops get stolen, and even servers or hosting companies die without prior notice. Data stored on websites may still be transient. If it is your data, you should make multiple, diverse copies, and make more copies over time as new media becomes popular. (And, if the data is sensitive, ENCRYPT IT!)

When I couldn't find my APC web page (I was certain that I had made one), I started checking the Wacky Wayback Machine to try to find the information. Sometimes this comes up empty, if the domain/pages you seek were not deemed interesting enough.

In closing, if you have any APC UPS gear, please check out my APC Clues page. Save a printed copy, because you may want that information again someday.  ;)

    Regards,

       -Z-