Why aren't service processors "easy to use"?

You could try to tell me that it's because serial consoles are sensitive access points to vital hardware, and that Security is inversely proportional to Ease Of Use, therefore Service Processors cannot be Easy, because they need to be Secure. And, doing my best Jeremy Clarkson (Top Gear) impression, I'd tell you that "you'd be wrong".

I'll tell you that it's because most the consumers are not asking for ease of use, or for interoperability. Because it seems (to me) to be on the minds (and mouths, and blogs) of a relatively small subset of vocal customers, and the 'collective voice' of this subset is further diminished because we are asking different vendors. (That is, I ask Dell, you ask HP, she asks Sun...but darn few of us are asking more than one or two vendors.)

Security features are only just now beginning to be built-in, either already enabled, or very easy to configure, largely because the US government has tied such requirements to their purchasing rules. If you want to sell the US government a desktop computer, there is now a whole checklist of security items to tick on the list before your kit might be considered. The US government, across many departments, now speaks with a large, unified voice (through the purchasing rules and policies)

Similarly, in the early days of the Point-to-Point Protocol (the follow-on to S.L.I.P. for IP-over-modem communications), PPP would allow more computer network protocols to connect and exchange data. As a customer, you'd have thought that vendor A's PPP (the choice inYOUR shop) would talk to Vendor B's gear at some Strategic Partner's network. But you'd have been wrong. Coupled with the birth of the Commercial Internet (it wasn't just for DARPA anymore), many companies were trying to get onto the Internet, or to connect to other companies to exchange information.

There was a large dissatisfacton with a lack up interoperability. And many companies spoke with the vendors to complain. BUT, besides complaining to the vendor who supplied the kit to YOU, you also would have complained to the vendors of the other guys gear as well. You might complain to 5 or 10 vendors, between routers and modems.

As a result of the dissatisfaction of LOTS of users, 6 companies started the PPP Consortium, with a week-long meeting at Telebit Corp in Sunnyvale, CA, with the express purpose of trying to test all of their gear with each other, and test across the matrix of then-new (and still evolving) PPP protocol options. At then end of it, the scorecard wasn't too bad, and all of the participants had a list of things to fix. They also hashed out some of the ambiguities in the draft Request For Comments (RFCs) describing the options. (Many of the vendors were part of the IEEE, and were helping define the RFCs, but the meeting helped settle misunderstandings about the options.)

Because the group of six could now claim a large amount of interoperability with the others in the group, it helped their marketing. Because the customers saw progress, they harped on the vendors who were NOT part of the original group, and they wanted to get in on the consortium. Six months after the first group, I believe there were 13 members, and 18 at the next. By the third meeting, PPP was Stable. PPP 'just worked', most of the time. PPP had become "easy", and SLIP was largely relagated to antique links on old equipment.

When I look at the various implementations of BIOS redirection to the serial port, they almost exclusively presume (or require) a modem connection, which then requires hardware level wiring issues if you are using a Console Server connection. The mapping of Function Keys, command sequences, and the timing issues to invoke the commands make using them tricky on a Console Server. It makes them impractical on any link slower than 19.2 Kbps. Yet the few BIOS makers don't see a need to try to do better, let alone to work with the other vendors to provide some consistancy of user interface to their customers. Do they know whether you care?

Now that Service Processors are becoming prevalent on new servers, it's hard to get a server without one. Yet they suffer from problems very similar to BIOS. The user interfaces are different from vendor to vendor. But what about processors from the same vendor. What about within the same vendor, across a hardware platform? You'd think that a vendor might want to keep their command sequences consistent, for the sake of their customers (who might be scripting support sequences for their machines). You'd probably like to think that...but you'd be wrong. A couple vendors seem to change at will, but I'm guessing that they are just buying server designs from other companies...and those same vendors probably haven't taken the time to design that Human User Interface, they probably simply rely on the hardware maker to fulfill the tasks (power cycle the server, set the IP address of the Ethernet interface for service processor, implement a web interface with certain features), leaving the details of keystrokes to the hardware maker...and when the vendor chooses another hardware maker, they get the same latitude with the design, and make different choices. (I wonder what Donald Norman thinks about tis problem.) Do your vendors know if you care?

When you tell a sales person that you want some new feature on their next year's model, the usual response is "OK, if we can get that done, how many will you buy?". It's going to cost them labor hours, engineering and QA time, technical documentation, maybe changing part of a training class. While you may only want 6, or 12, or 40, your voice will be tallied. When they from enough of you, it's going to be worth it to make a change. But we need to tip that scale by asking, and probably tying that request to some money. (My dad would sometimes ask me to "write your request on a $5 bill, and I'll consider it", but I have seen this over and over again, from different seats in different companies. For the vendor, it's all about money.)

My point here, is that you are going to spend money on new servers at some point. Make your money count! Tie the money to your requests for easier-to-use service processors. Decide what you want, and tell your possible vendors about your requirements. They all want your money...so vote with your wallet! Reward the responsive vendors! If the unresponsive vendors don't get the order, they'll at least know why, and they'll learn that you were serious. As more of us tie our requirements to purchasing, we can make a difference.

As you get into your busy end-of-year, some of you are already going through your budget cycle for 2009. Use any slowdown you get to think about requirements to go along with those purchases. Get them on paper, and in email, so they are ready to send to your vendors with your Request for Quote. And good luck in the new year.

-Z-

What is 'Normal', anyway?

Yow! It's already well into May, and April got by without a post. It was a busy month, with a LOT of discovery tasks happening. I'll try to get a few more posts in this month to summarize. But first, I wanted to get this thought online...

Those who know me are not surprised if the come into a dark wiring closet, and find me sitting on a milk crate, staring at the 'blinkly lights'. (Frankly, I've startled plenty of people who don't know me this way!) Most just write it of as a fixation on blinky lights, but it is not. This is how I can 'visually spot trends' of activity on devices and interfaces. It's also a chance to remember which link status indicators are on, off, in an alarm condition. And, I'm also listening... to fans, air conditioner inlets, hard drive spindle bearings, modem speakers, relays clicking.

This is how I get to know what "Normal" looks like, sounds like, and smells like in the data center and wiring closets. (Yes, smell... does it 'usually' smell damp in this room? Is the 'ozone' smell something that's always here, or does it indicate a component failure. Scent is a strong trigger for memory!)

Knowing what is 'Normal' helps us spot what is unusual. When I have a network failure, I can go to the associated wiring closet and look, listen, and smell...I don't need to ponder "has that always been like this?", because I'll remember. "That light is usually blinking...so there isn't traffic on that interface!" Knowing what is normal is a key to fast troubleshooting.

The RRD tool has been a great resource for graphing monitoring data, allowing you to visualize 'normal', to see 'now', and to identify trends. You find this under MANY open source tools, such as MRTG, Cricket, Cacti, and many more.

But, can you tell what's 'normal' with a serial console? Yes, you can! The key is, you need to LOOK when things are operating normally. Look when the system/device/network is idle some night or weekend. Look again when backups are running. Look again when the network is busy, but not failing. Then, compare your notes, or, your LOGS!

You can do a LOT interactively, using a simple terminal emulator and a cable. You have a lot more flexibility with a Console Server and multiple Telnet sessions (for example, you can monitor many consoles simultaneously, and cut-and-paste between them). The real benefits are had when you combine the console servers with a Serial Console Management Application such as Conserver, or ConsoleWorks, and you can compare historic data with today's results.

I have a handful of devices which have "diagnostics' ports, that only the field engineers will use. They are not for normal use by customers. However, when you connect to these ports, you can find some of the devices are 'beaconing' about events, or reporting regular status messages. Even if the port doesn't say much, if you hit a carriage return, you'll probably get a prompt...doing this occasionally will tell you it's still alive. And, sometimes, you can also leverage the simple diagnostics that are built in. It's good to have a baseline log of what the devices are reporting to these diagnostics when they are healthy, so you can compare them if the device starts misbehaving.

But, to do any of these things, you need to start by looking when things are working as expected, doing their job the way you want them to. That's when you want to get your first look, and save that data for later comparison. (Speaking of comparing log data, try SPLUNK! Check out splunkbase, and the SPLUNK Forums! Splunk is worth a few blog articles by itself...later.)

The bottom line: Do you KNOW what you are missing? Do you WANT to know? Then LOOK!

-Z-