Assiging DHCP addresses to APC Power Strips

This is certainly a post related to serial consoles. But, you are limited in the amount of useful monitoring data that you can pull via the CLI, so folks often want to use SNMP or SYSLOG to monitor these devices. Usually, that's where they want CLI access.

In my case, we are re-numbering the management subnet, and we've decided to try to make this job easier for the next time a re-number is required. We're telling the devices to use DHCP addressing, and using MAC Address reservations for all of the devices that we care about. Two complications arose; How to connect to the serial CLI, and how to set the device to use DHCP.

The complication with using DHCP addressing on the AP-7868 PDU (and likely many other APC devices), is that their default configuration REQUIRES that your DHCP server set option 43 to a specific vendor code. If you don't do this, the PDU will not accept the DHCP Offer from your server.

APC AP-7868 Smart PDU Serial Pinouts

APC RJ-13 serial console, 4-wire, data-only connection
APC Console     Lantronix TS (Cisco, Opengear with Cisco wiring, etc.)
(RJ-13)         (RJ-45)
1  ---(???) nc            default settings:  9600-8-N-1
2  ---(GND)---  5
3  ---(TxD)-->  6         default username:  apc
4  <--(RxD)---  3         default password:  apc
5  ---(GND)---  
6  ---(???) nc 
( nc = No Connection )

NOTE: You must configure a default gateway for these devices (APC-7868), even if you are only talking on your own subnet!

On your Conserver host

This section includes what you need to type on the CLI console, to configure the PDU to ignore Option 43 and accept the DHCP Offer. I use the spare port on one of my console servers to do this, and I always name my unused ports in my Conserver deployments, so that I can use them at a moments notice. In this case, I'm using port 30, on Console Server 5...

NOTE: the CLI is not good about sanitizing user input, or detecting errors! You can type commas (or TEXT!) in an IP address field, and non-standard TLDs in the domain field! Be very careful about your typing, and visually check your data before accepting the changes!

console unused-cs5-30     (this is the port I picked...)
[Enter]
[Enter]     (twice, to wake-up the CLI interface)
apc [Enter]     (default login name: apc  case-sensitive!)
apc [Enter]     (default password: apc  case-sensitive!)
2 [Enter]     (Network)
1 [Enter]     (TCP/IP)
(If you need the MAC address, you'll find it here)
4 [Enter]     (Boot Mode   only if currently set for Manual...)
2 [Enter]     (DHCP Only)
2 [Enter]     (Advanced)
1 [Enter]     (Device Name)
ps-6.12i [Enter]
2 [Enter>     (Domain Name
garage.com [Enter]
8 [Enter]     (DHCP Cookie Is: ...)
1 [Enter]     (Not Required to accept offer)
9 [Enter]     (Accept the pending changes)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
4 [Enter]     (logout)

When you log out of the PDU, the device will reload the TCP stack, and in a few minutes, your PDU should accept the new IP address via DHCP.

Configuring your DHCP Server for Option 43

You need to define you option 43. How (or IF) you can do this will depend on your DHCP server software. But, this section includes the particular HEX characters that the APC PDU expects to see (or it will refuse the DHCP Offer).

These clues are for the Internet Software Consortium (ISC) DHCP server. See the dhcp-options man page - this describes all the standard DHCP options. Look for the section titled "VENDOR ENCAPSULATED OPTIONS", which is option 43. If you already have a class for APC units, then that would be a good place to define option 43 as it will be defined for all devices that match the class. The simplest way to define the value is something like this:

     option vendor-encapsulated-options 01:04:31:41:50:43;

DHCP option names and numbers are listed in RFC1533, and in the ISC DHCPd source file common/tables.c

(Explanation: I forget the first character (01), but I believe that it designates to expect HEX characters. The second says there are (4) characters in the field. The last three characters are ASCII for "APC". I don't know why the "31" is there...)

NOTE: All APC PDU's should be protected by UPS power.

In an unprotected environment, if the utility power flickers off and on quickly, the onboard network card may not power back on. In these instances, the PDU may power on but the LED display will be blank and any type of access (web, telnet, console) to the PDU will be disabled. In this case, you must power cycle the PDU again, by pulling the power cord, waiting at least 5 seconds, and re-applying power.

The on-board network card was not designed for quick off/on scenarios. Customers protecting their PDU with a UPS, as designed, will not have this problem.

There are also some APC UPS clues on my console site, http://www.conserver.com/consoles/Clues/cons-apc.html
 
Regards,

-Zonker-

How did the year get by me?

Wow, I'm surprised that a year has passed since my last post. Largely, it has been due to my inability to get the editing tools to play well with the iPhone and now the iPad. Normally, I don't update my stuff on employer time, or employer equipment. Couple this with a VERY busy year, and me putting updates lower on the priority list, and here we are on Memorial Day weekend.

A lot has happened in the past 12 months. Too much for a single post, but let me sum up.

I'm working with a variety of Service Processors this year, making the work, and documenting what I've learned. Besides working at the BIOS level, I'm trying to determine if/how the settings can be modified from the OS's involved, and that is more difficult than I supposed. The main problem seems like the websites holding the IPMI drivers or libraries have disappeared after 3-4 years. (The biggest lesson here is to capture, collect, and archive these files for any hardware you acquire. Mergers and acquisitions contribute to the loss of the files, even as the instructions make it into a knowledge base.)

An old lesson re-learned, is that older Sun gear will ignore their serial console settings as soon as you plug a keyboard in! Your only clue, if you are logging the console output with Conserver or some other application, is a small line about detecting a keyboard. Removing the keyboard does NOT revert back to serial console output, you need to reboot the server with no keyboard attached. I knew this long ago, but then was thwarted when folks in a remote office attached a keyboard, and we didn't notice that console I/O had gone away.

I've also been doing a LOT of work with SNMP in the past 6 months, and I found an oddity with an MGE (which sold the line to APC, who then sold the line to Schneider Electric) room-sized UPS. The brains of the UPS uses a serial line to pass a data stream to the Network Interface shelf. If this streaming interface is removed, the SNMP cards do not get an update... But the cards simply report the LAST value for any queried OID. As a result, your only clue of a disconnect is seeing all results "flat-line". BUT, if you are also set up forSNMP traps, you will get also see a string that communications was lost with the UPS. This is a case where you really need to be able to parse Syslog and SNMP traps for interesting strings and send an alert.

Other UPS problems led to the integration of a UPS of Last Resort, so that core devices could track and log events with the big UPS. This included watching the serial output of the APC Smart-UPS 2200XL, plus SNMP integration into our monitoring system.

Another Big Project in the past year was due to a planned network outage. This was due to a planned partial power outage in the building. Unfortunately, some key devices also lost power, causing trouble, which could not send alerts via email because the network was down. We decided that we needed to make sure there was a second path out of the data center for critical alerts and messages. This led to integration with a FoxBox, a Network-connected cell-phone device, allowing SMS alerting AND control!

I've also added many more hosts to the host-to adapter database, but I only have a basic Cyclades console connection guide posted, and I still need to make a Digi page. (Cyclades was another fine example of collecting information from the web as soon as you hear about an acquisition!)

I'll make stronger efforts to post more frequently through the rest of 2010. thanks for reading.

-Zonker-

Look at all the pretty lights

This week, I needed to reconfigure a previously-used Cyclades ACS console server. Finding the right adapters was easy enough, but getting the USB serial dongle to play was the tricky bit. Without a signal tracer, I was stuck with the evidence... "it's not working"...and I was trying different hardware combinations, and a few software changes. I wasted 2.5 hours going through different combinations, and I put a reminder in my calendar to bring my signal tracers and my Serial Doctors Bag the next day.

Once I had my trusty signal tracers, it was a matter of 10 minutes to find the proper USB settings, COM port configuration in my terminal emulator, and start the reconfiguration process. Blinky lights will save you time, every time. Good quality signal tracers are worth a high cost, because they will save you time often, and serve you well for many years. One day, blinky lights will probably save my life...

So, having solved Problem #1, and with kit in hand, I tackled Problem #2;

Years ago, a couple of hundred APC smart power strips (OK, "0-RU, metered Power Distribution Units") were installed in a data center. Fast forward to last week, when one unit is failing to respond to SNMP after a high-temperature 'event' in the room. Since it was in critical service, we wanted to try using the serial console to see if the brains were still alive. Unfortunately, nobody could find a serial cable for the job, and APC doesn't want folks making their own cables so APC doesn't post any signaling clues. I did find one website with clues about the wiring for the 940-0144 cable.

The clues indicated that the 6-pin RJ-14 interface was, indeed, RS-232 (and not TTL, for example), I could do some experimenting. A new toy in my Doctors Bag (not in the pictures (yet), is a MODAPT from Siemon. With this, and my trusty Digital Volt Meter (DVM), I was able to confirm some signalling that wasn't mentioned in the LuxNET web page.

Pin1 -2v (in reference to Pin2)
Pin2 GND (0v potential to Pin5)
Pin3 TxD output (-5.5v in reference to Pin2)
Pin4 RxD input (0.2V in reference to Pin2)
Pin5 GND (0v potential to Pin2)
Pin6 -2v (in reference to Pin2)

With this information, I could then make an adapter cable, and then use the RJ45 signal tracer to confirm that the signals are on the right pins. (I found that I could use just 3 wires, Ground - RxD - TxD) for communications to the APC AP7868 rPDU. I did not need to loop RJ-14 pins 1 and 6, and I didn't need to connect RJ-14 pins 1 or 6 to ground to establish communication.)

I'll update my APC console page and my Signals page soon with some of this information.

APC Serial Ports, and websites, wikis, and backups, oh my!

How many of you use APC brand UPS gear? Raise your hands! Ok, you can put your hands down.

Now, how many of you have connected the UPS serial port to your computer, to get power information, or to have the UPS tell your computer to shut down? Not so many? How many tried, but had trouble? 

Question 1: Were you using the APC-supplied cable (with the correct end plugged into the UPS)?

I ask, because the APC serial ports are unusual! If you use one of the cables you have lying around your shop, there is a good chance that the UPS shut down, and would not turn on again until you disconnected the cable. 

When I first tried this, the cable was missing, and I tried various cables in the shop (because I didn't have passive signal tracers at the time). Nothing worked, and I sometimes lost power. 

When I had another UPS, with the APC cable, I found that the cable wasn't long enough, and some of the cables I tried using to extend the run caused troubles. That's when I dug deeper into the problem. Here's what I've found;

• The APC port puts data and ground on unusual pins!
  
• The UPS uses the two outputs and inputs for other purposes!

You can actually make a "Big Red Switch" for your small APC, using a couple wires and the DE-9 connection. Because of the signals being on unusual pins, when you plug in a PC with a non-APC cable, you are actually applying a "shut-down" hard-wired signal to the UPS.

I made an APC Clues web page, documenting the findings, in late 2007. But, for some reason, I guess I didn't post it. Only when I needed the information again (and couldn't find it) did the panic set in, because some of the legacy information is no longer on the web.

That brings us to....

Question 2: How many of you are making backups of your critical files?

You can replace the word "critical" with Photos, Homework, Research, Family Tree, Tax Info, you get the idea. The data that you do not want to lose in a hard disk crash, or on a lost Flash Memory Drive. Where do you keep it, and how often do you update it?

I use an automated system, backing up critical files to a disk "in the cloud", for $99 a year. This runs every time I start up my computer, and I put a lot of faith in it. But, I need to pay money to keep it alive.

I also make manual copies to a 1 GB external hard disk (but I have to remember to move new files). I should really set this up to run automatically.

For some of my hobbies, and console stuff, I put copies of the web. (Technically, I made them for display on the web, so I guess the copies on my home machine are my backups.)

For some data sets (photos for some of my hobbies or family, or my console-related files, for example) will be occasionally copied to a CD or a DVD, and distributed to friends, family, or colleagues. While it isn't complete, this may be one of my best backup methods!

My point here is that disks die, memory sticks get lost, laptops get stolen, and even servers or hosting companies die without prior notice. Data stored on websites may still be transient. If it is your data, you should make multiple, diverse copies, and make more copies over time as new media becomes popular. (And, if the data is sensitive, ENCRYPT IT!)

When I couldn't find my APC web page (I was certain that I had made one), I started checking the Wacky Wayback Machine to try to find the information. Sometimes this comes up empty, if the domain/pages you seek were not deemed interesting enough.

In closing, if you have any APC UPS gear, please check out my APC Clues page. Save a printed copy, because you may want that information again someday.  ;)

    Regards,

       -Z-

How about Ubiquitous USB Serial Consoles

Happy New Year! Sorry I've missed posting in a couple months.

It's odd to me, when sysadmins tell me that "Serial Consoles are Dead. USB killed them." But, when I ask them what they mean, I get different answers.

The first meaning is: "RS-232 is dead, long live USB!" So, then I ask them what they do when they need to get to the serial console, and they tell me about needing to find a USB-to-DE9 serial adapter, get that to work, and point the serial console getty to that device. ("I bet that's tough to do when you are having trouble with a server" I offer. They usually agree that it is...)

The second meaning is: "You don't need DE-9 connectors anymore, you can do it all with USB". This has more insidious meanings...

2a) Most new servers don't have a DE-9 connector on the chassis, just USB ports.

2b) If I need a serial port, I can add a USB-to-DE9 dongle, and use minicom.

Have you ever tried plugging in a USB-connected external modem, and then tried to dial into it, to access the serial console (getty) on your server? The adventure is fraught with problems, which must be solved with a Crash Cart or other form of KVM or remote desktop (which means you need to think about it BEFORE you have a problem, so that it's a useful avenue when trouble presents itself).

Remember, the serial console has been your friend, when the network or the network-facing configuration goes awry. Do you think USB is ready to replace that role? Can you go to Fry's, and buy a USB-to-USB cable which you can simply plug in between two servers, and automatically establish a "null-modem"-type data path? I haven't seen one. Why Not?

Someone needs to develop a little, microprocessor pod, with two USB cables. Power the unit from the 5-volt signal from either USB device. Maybe even add some TX and RX data status LEDs, to show which USB cable is talking. You don't even need to do the conversion from TTL to RS-232 and back...just keep it all at TTL levels! But the microprocessor needs to be able to talk to each USB port when the computer inquires "Who are you, what are you?"...this is the key!

USB relies on drivers, residing on your computer(s), to recognize the ID sent from a USB-attached device when you first plug it in. To make things easy, many basic drivers are bundled into Operating System packages, which lets you plug in many devices without needing to install the drivers first. (This is the difference between Play and Pray in "Plug-n-Play" architecture.)

Looking at the basic USB-to-DE9 dongles, most are derivative works from two basic hardware types. For example, since Brands A, G, M and Q all use "Chip X", and Brands D, I and S all use "Chip Z", it's easy to see that just having two basic (generic) drivers will let you plug in most of the devices from these 7 brands of dongles. Getting Microsoft, and the various Linux distributions is an important step in making things easier for the folks who use these operating systems with serial devices.

But, if you make this USB cable I'm describing, and you try to use either of these USB-to-Serial chips as a base, you will find that most systems will think that you are attaching a plain serial device, and they'll use the generic driver. So, you'll need a bit more engineering...you need a driver that will determine if it needs to find (or create) a getty process, and connect it to the I/O address for the cable. Since you are going to need to make your own driver, you just gave yourself the flexibility to define how this type of cable communicates with the computers.

In a basic "Crash Cart" scenario, I want to plug the "Server" cable into my server chassis. If there is a console getty process running, I want it to be directed to this new device (even if it thinks it was connected to some other previous USB-to-Serial dongle, since that may have failed - causing me to need the crash cart).

When I plug in the "Client" cable to my crash cart, or a laptop, I want them to think that it's just another serial port. (For real Ease Of Use, let's default the serial speed to 9600 8N1.) If it's a Windows box, I'd like the option to choose (in the driver configuration) which COM port this device is assigned to (so it's going to ALWAYS be there...even if it needs to steal the assignment from a previous device). I'd also like to be able to associate this interface with an application, so that the OS will try to launch that application when the cable is plugged in.

I want to walk up to a server, just out of the box, with my laptop, and plug this cable into the server and into the laptop...and have the laptop launch my copy of ProCom, or TerraTerm, or HyperTerm...and have the server configure it's getty to talk to the new port, if the OS is healthy.

If we can loop in a few of the big BIOS makers, the BIOS might notice this device on startup, if BIOS redirection has been turned on, and use the cable (as a preference above the usual COM A assignment).

Of course, until we can get to this point, USB hasn't replaced the venerable RS-232 serial port, at least in my data center.

Good luck with this winter!

-Z-

Why aren't service processors "easy to use"?

You could try to tell me that it's because serial consoles are sensitive access points to vital hardware, and that Security is inversely proportional to Ease Of Use, therefore Service Processors cannot be Easy, because they need to be Secure. And, doing my best Jeremy Clarkson (Top Gear) impression, I'd tell you that "you'd be wrong".

I'll tell you that it's because most the consumers are not asking for ease of use, or for interoperability. Because it seems (to me) to be on the minds (and mouths, and blogs) of a relatively small subset of vocal customers, and the 'collective voice' of this subset is further diminished because we are asking different vendors. (That is, I ask Dell, you ask HP, she asks Sun...but darn few of us are asking more than one or two vendors.)

Security features are only just now beginning to be built-in, either already enabled, or very easy to configure, largely because the US government has tied such requirements to their purchasing rules. If you want to sell the US government a desktop computer, there is now a whole checklist of security items to tick on the list before your kit might be considered. The US government, across many departments, now speaks with a large, unified voice (through the purchasing rules and policies)

Similarly, in the early days of the Point-to-Point Protocol (the follow-on to S.L.I.P. for IP-over-modem communications), PPP would allow more computer network protocols to connect and exchange data. As a customer, you'd have thought that vendor A's PPP (the choice inYOUR shop) would talk to Vendor B's gear at some Strategic Partner's network. But you'd have been wrong. Coupled with the birth of the Commercial Internet (it wasn't just for DARPA anymore), many companies were trying to get onto the Internet, or to connect to other companies to exchange information.

There was a large dissatisfacton with a lack up interoperability. And many companies spoke with the vendors to complain. BUT, besides complaining to the vendor who supplied the kit to YOU, you also would have complained to the vendors of the other guys gear as well. You might complain to 5 or 10 vendors, between routers and modems.

As a result of the dissatisfaction of LOTS of users, 6 companies started the PPP Consortium, with a week-long meeting at Telebit Corp in Sunnyvale, CA, with the express purpose of trying to test all of their gear with each other, and test across the matrix of then-new (and still evolving) PPP protocol options. At then end of it, the scorecard wasn't too bad, and all of the participants had a list of things to fix. They also hashed out some of the ambiguities in the draft Request For Comments (RFCs) describing the options. (Many of the vendors were part of the IEEE, and were helping define the RFCs, but the meeting helped settle misunderstandings about the options.)

Because the group of six could now claim a large amount of interoperability with the others in the group, it helped their marketing. Because the customers saw progress, they harped on the vendors who were NOT part of the original group, and they wanted to get in on the consortium. Six months after the first group, I believe there were 13 members, and 18 at the next. By the third meeting, PPP was Stable. PPP 'just worked', most of the time. PPP had become "easy", and SLIP was largely relagated to antique links on old equipment.

When I look at the various implementations of BIOS redirection to the serial port, they almost exclusively presume (or require) a modem connection, which then requires hardware level wiring issues if you are using a Console Server connection. The mapping of Function Keys, command sequences, and the timing issues to invoke the commands make using them tricky on a Console Server. It makes them impractical on any link slower than 19.2 Kbps. Yet the few BIOS makers don't see a need to try to do better, let alone to work with the other vendors to provide some consistancy of user interface to their customers. Do they know whether you care?

Now that Service Processors are becoming prevalent on new servers, it's hard to get a server without one. Yet they suffer from problems very similar to BIOS. The user interfaces are different from vendor to vendor. But what about processors from the same vendor. What about within the same vendor, across a hardware platform? You'd think that a vendor might want to keep their command sequences consistent, for the sake of their customers (who might be scripting support sequences for their machines). You'd probably like to think that...but you'd be wrong. A couple vendors seem to change at will, but I'm guessing that they are just buying server designs from other companies...and those same vendors probably haven't taken the time to design that Human User Interface, they probably simply rely on the hardware maker to fulfill the tasks (power cycle the server, set the IP address of the Ethernet interface for service processor, implement a web interface with certain features), leaving the details of keystrokes to the hardware maker...and when the vendor chooses another hardware maker, they get the same latitude with the design, and make different choices. (I wonder what Donald Norman thinks about tis problem.) Do your vendors know if you care?

When you tell a sales person that you want some new feature on their next year's model, the usual response is "OK, if we can get that done, how many will you buy?". It's going to cost them labor hours, engineering and QA time, technical documentation, maybe changing part of a training class. While you may only want 6, or 12, or 40, your voice will be tallied. When they from enough of you, it's going to be worth it to make a change. But we need to tip that scale by asking, and probably tying that request to some money. (My dad would sometimes ask me to "write your request on a $5 bill, and I'll consider it", but I have seen this over and over again, from different seats in different companies. For the vendor, it's all about money.)

My point here, is that you are going to spend money on new servers at some point. Make your money count! Tie the money to your requests for easier-to-use service processors. Decide what you want, and tell your possible vendors about your requirements. They all want your money...so vote with your wallet! Reward the responsive vendors! If the unresponsive vendors don't get the order, they'll at least know why, and they'll learn that you were serious. As more of us tie our requirements to purchasing, we can make a difference.

As you get into your busy end-of-year, some of you are already going through your budget cycle for 2009. Use any slowdown you get to think about requirements to go along with those purchases. Get them on paper, and in email, so they are ready to send to your vendors with your Request for Quote. And good luck in the new year.

-Z-

A little perl of wisdom

When it comes to finding some quick help (after the Helpdesk has closed), or those tidbits of knowledge and experience of someone who has tread the path before me, Google has been my friend for many years (taking up the mantle after AltaVista!).

But, there is a downside to learning all about a topic from Google. It is that you, the reader, are guiding the learning. You are getting the answers to the questions you ask...but it doesn't always offer things that you should know (or at least consider), based on your searches.

I taught myself perl, reading books, and using Google for answers, and examples...but I sometimes found myself confused when reading someone else's code. This year, I'm taking formal classes, from a few sources, hoping to patch a few holes in my foundation of knowledge.

While I'm glad to have taken this approach, to have someone guiding me through the learning, systematically, and answering my questions...I'm finding that the Teacher in this equation is even more important than the content. Even if someone knows the content well, that doesn't mean that they will be a good teacher. (Answering the occasional question does not equal teaching.)

After 5 weeks in a semester-long class at a local community college, I too a full-day Perl 101 course from Bayview Training...and then I coded for 15 hours straight, because that class had made a number of concepts more concrete for me (several "light-bulb" moments), and I was excited about the possibilities for my future projects.

I heartily recommend Bill Ward as an instructor. While he teaches primarily in Silicon Valley, he does travel for some classes and consulting as well. The cost of the course was money well spent! If you really need to learn some perl, check out Bayview Training and Bill Ward!