Look at all the pretty lights

This week, I needed to reconfigure a previously-used Cyclades ACS console server. Finding the right adapters was easy enough, but getting the USB serial dongle to play was the tricky bit. Without a signal tracer, I was stuck with the evidence... "it's not working"...and I was trying different hardware combinations, and a few software changes. I wasted 2.5 hours going through different combinations, and I put a reminder in my calendar to bring my signal tracers and my Serial Doctors Bag the next day.

Once I had my trusty signal tracers, it was a matter of 10 minutes to find the proper USB settings, COM port configuration in my terminal emulator, and start the reconfiguration process. Blinky lights will save you time, every time. Good quality signal tracers are worth a high cost, because they will save you time often, and serve you well for many years. One day, blinky lights will probably save my life...

So, having solved Problem #1, and with kit in hand, I tackled Problem #2;

Years ago, a couple of hundred APC smart power strips (OK, "0-RU, metered Power Distribution Units") were installed in a data center. Fast forward to last week, when one unit is failing to respond to SNMP after a high-temperature 'event' in the room. Since it was in critical service, we wanted to try using the serial console to see if the brains were still alive. Unfortunately, nobody could find a serial cable for the job, and APC doesn't want folks making their own cables so APC doesn't post any signaling clues. I did find one website with clues about the wiring for the 940-0144 cable.

The clues indicated that the 6-pin RJ-14 interface was, indeed, RS-232 (and not TTL, for example), I could do some experimenting. A new toy in my Doctors Bag (not in the pictures (yet), is a MODAPT from Siemon. With this, and my trusty Digital Volt Meter (DVM), I was able to confirm some signalling that wasn't mentioned in the LuxNET web page.

Pin1 -2v (in reference to Pin2)
Pin2 GND (0v potential to Pin5)
Pin3 TxD output (-5.5v in reference to Pin2)
Pin4 RxD input (0.2V in reference to Pin2)
Pin5 GND (0v potential to Pin2)
Pin6 -2v (in reference to Pin2)

With this information, I could then make an adapter cable, and then use the RJ45 signal tracer to confirm that the signals are on the right pins. (I found that I could use just 3 wires, Ground - RxD - TxD) for communications to the APC AP7868 rPDU. I did not need to loop RJ-14 pins 1 and 6, and I didn't need to connect RJ-14 pins 1 or 6 to ground to establish communication.)

I'll update my APC console page and my Signals page soon with some of this information.

APC Serial Ports, and websites, wikis, and backups, oh my!

How many of you use APC brand UPS gear? Raise your hands! Ok, you can put your hands down.

Now, how many of you have connected the UPS serial port to your computer, to get power information, or to have the UPS tell your computer to shut down? Not so many? How many tried, but had trouble? 

Question 1: Were you using the APC-supplied cable (with the correct end plugged into the UPS)?

I ask, because the APC serial ports are unusual! If you use one of the cables you have lying around your shop, there is a good chance that the UPS shut down, and would not turn on again until you disconnected the cable. 

When I first tried this, the cable was missing, and I tried various cables in the shop (because I didn't have passive signal tracers at the time). Nothing worked, and I sometimes lost power. 

When I had another UPS, with the APC cable, I found that the cable wasn't long enough, and some of the cables I tried using to extend the run caused troubles. That's when I dug deeper into the problem. Here's what I've found;

• The APC port puts data and ground on unusual pins!
  
• The UPS uses the two outputs and inputs for other purposes!

You can actually make a "Big Red Switch" for your small APC, using a couple wires and the DE-9 connection. Because of the signals being on unusual pins, when you plug in a PC with a non-APC cable, you are actually applying a "shut-down" hard-wired signal to the UPS.

I made an APC Clues web page, documenting the findings, in late 2007. But, for some reason, I guess I didn't post it. Only when I needed the information again (and couldn't find it) did the panic set in, because some of the legacy information is no longer on the web.

That brings us to....

Question 2: How many of you are making backups of your critical files?

You can replace the word "critical" with Photos, Homework, Research, Family Tree, Tax Info, you get the idea. The data that you do not want to lose in a hard disk crash, or on a lost Flash Memory Drive. Where do you keep it, and how often do you update it?

I use an automated system, backing up critical files to a disk "in the cloud", for $99 a year. This runs every time I start up my computer, and I put a lot of faith in it. But, I need to pay money to keep it alive.

I also make manual copies to a 1 GB external hard disk (but I have to remember to move new files). I should really set this up to run automatically.

For some of my hobbies, and console stuff, I put copies of the web. (Technically, I made them for display on the web, so I guess the copies on my home machine are my backups.)

For some data sets (photos for some of my hobbies or family, or my console-related files, for example) will be occasionally copied to a CD or a DVD, and distributed to friends, family, or colleagues. While it isn't complete, this may be one of my best backup methods!

My point here is that disks die, memory sticks get lost, laptops get stolen, and even servers or hosting companies die without prior notice. Data stored on websites may still be transient. If it is your data, you should make multiple, diverse copies, and make more copies over time as new media becomes popular. (And, if the data is sensitive, ENCRYPT IT!)

When I couldn't find my APC web page (I was certain that I had made one), I started checking the Wacky Wayback Machine to try to find the information. Sometimes this comes up empty, if the domain/pages you seek were not deemed interesting enough.

In closing, if you have any APC UPS gear, please check out my APC Clues page. Save a printed copy, because you may want that information again someday.  ;)

    Regards,

       -Z-

How about Ubiquitous USB Serial Consoles

Happy New Year! Sorry I've missed posting in a couple months.

It's odd to me, when sysadmins tell me that "Serial Consoles are Dead. USB killed them." But, when I ask them what they mean, I get different answers.

The first meaning is: "RS-232 is dead, long live USB!" So, then I ask them what they do when they need to get to the serial console, and they tell me about needing to find a USB-to-DE9 serial adapter, get that to work, and point the serial console getty to that device. ("I bet that's tough to do when you are having trouble with a server" I offer. They usually agree that it is...)

The second meaning is: "You don't need DE-9 connectors anymore, you can do it all with USB". This has more insidious meanings...

2a) Most new servers don't have a DE-9 connector on the chassis, just USB ports.

2b) If I need a serial port, I can add a USB-to-DE9 dongle, and use minicom.

Have you ever tried plugging in a USB-connected external modem, and then tried to dial into it, to access the serial console (getty) on your server? The adventure is fraught with problems, which must be solved with a Crash Cart or other form of KVM or remote desktop (which means you need to think about it BEFORE you have a problem, so that it's a useful avenue when trouble presents itself).

Remember, the serial console has been your friend, when the network or the network-facing configuration goes awry. Do you think USB is ready to replace that role? Can you go to Fry's, and buy a USB-to-USB cable which you can simply plug in between two servers, and automatically establish a "null-modem"-type data path? I haven't seen one. Why Not?

Someone needs to develop a little, microprocessor pod, with two USB cables. Power the unit from the 5-volt signal from either USB device. Maybe even add some TX and RX data status LEDs, to show which USB cable is talking. You don't even need to do the conversion from TTL to RS-232 and back...just keep it all at TTL levels! But the microprocessor needs to be able to talk to each USB port when the computer inquires "Who are you, what are you?"...this is the key!

USB relies on drivers, residing on your computer(s), to recognize the ID sent from a USB-attached device when you first plug it in. To make things easy, many basic drivers are bundled into Operating System packages, which lets you plug in many devices without needing to install the drivers first. (This is the difference between Play and Pray in "Plug-n-Play" architecture.)

Looking at the basic USB-to-DE9 dongles, most are derivative works from two basic hardware types. For example, since Brands A, G, M and Q all use "Chip X", and Brands D, I and S all use "Chip Z", it's easy to see that just having two basic (generic) drivers will let you plug in most of the devices from these 7 brands of dongles. Getting Microsoft, and the various Linux distributions is an important step in making things easier for the folks who use these operating systems with serial devices.

But, if you make this USB cable I'm describing, and you try to use either of these USB-to-Serial chips as a base, you will find that most systems will think that you are attaching a plain serial device, and they'll use the generic driver. So, you'll need a bit more engineering...you need a driver that will determine if it needs to find (or create) a getty process, and connect it to the I/O address for the cable. Since you are going to need to make your own driver, you just gave yourself the flexibility to define how this type of cable communicates with the computers.

In a basic "Crash Cart" scenario, I want to plug the "Server" cable into my server chassis. If there is a console getty process running, I want it to be directed to this new device (even if it thinks it was connected to some other previous USB-to-Serial dongle, since that may have failed - causing me to need the crash cart).

When I plug in the "Client" cable to my crash cart, or a laptop, I want them to think that it's just another serial port. (For real Ease Of Use, let's default the serial speed to 9600 8N1.) If it's a Windows box, I'd like the option to choose (in the driver configuration) which COM port this device is assigned to (so it's going to ALWAYS be there...even if it needs to steal the assignment from a previous device). I'd also like to be able to associate this interface with an application, so that the OS will try to launch that application when the cable is plugged in.

I want to walk up to a server, just out of the box, with my laptop, and plug this cable into the server and into the laptop...and have the laptop launch my copy of ProCom, or TerraTerm, or HyperTerm...and have the server configure it's getty to talk to the new port, if the OS is healthy.

If we can loop in a few of the big BIOS makers, the BIOS might notice this device on startup, if BIOS redirection has been turned on, and use the cable (as a preference above the usual COM A assignment).

Of course, until we can get to this point, USB hasn't replaced the venerable RS-232 serial port, at least in my data center.

Good luck with this winter!

-Z-

Why aren't service processors "easy to use"?

You could try to tell me that it's because serial consoles are sensitive access points to vital hardware, and that Security is inversely proportional to Ease Of Use, therefore Service Processors cannot be Easy, because they need to be Secure. And, doing my best Jeremy Clarkson (Top Gear) impression, I'd tell you that "you'd be wrong".

I'll tell you that it's because most the consumers are not asking for ease of use, or for interoperability. Because it seems (to me) to be on the minds (and mouths, and blogs) of a relatively small subset of vocal customers, and the 'collective voice' of this subset is further diminished because we are asking different vendors. (That is, I ask Dell, you ask HP, she asks Sun...but darn few of us are asking more than one or two vendors.)

Security features are only just now beginning to be built-in, either already enabled, or very easy to configure, largely because the US government has tied such requirements to their purchasing rules. If you want to sell the US government a desktop computer, there is now a whole checklist of security items to tick on the list before your kit might be considered. The US government, across many departments, now speaks with a large, unified voice (through the purchasing rules and policies)

Similarly, in the early days of the Point-to-Point Protocol (the follow-on to S.L.I.P. for IP-over-modem communications), PPP would allow more computer network protocols to connect and exchange data. As a customer, you'd have thought that vendor A's PPP (the choice inYOUR shop) would talk to Vendor B's gear at some Strategic Partner's network. But you'd have been wrong. Coupled with the birth of the Commercial Internet (it wasn't just for DARPA anymore), many companies were trying to get onto the Internet, or to connect to other companies to exchange information.

There was a large dissatisfacton with a lack up interoperability. And many companies spoke with the vendors to complain. BUT, besides complaining to the vendor who supplied the kit to YOU, you also would have complained to the vendors of the other guys gear as well. You might complain to 5 or 10 vendors, between routers and modems.

As a result of the dissatisfaction of LOTS of users, 6 companies started the PPP Consortium, with a week-long meeting at Telebit Corp in Sunnyvale, CA, with the express purpose of trying to test all of their gear with each other, and test across the matrix of then-new (and still evolving) PPP protocol options. At then end of it, the scorecard wasn't too bad, and all of the participants had a list of things to fix. They also hashed out some of the ambiguities in the draft Request For Comments (RFCs) describing the options. (Many of the vendors were part of the IEEE, and were helping define the RFCs, but the meeting helped settle misunderstandings about the options.)

Because the group of six could now claim a large amount of interoperability with the others in the group, it helped their marketing. Because the customers saw progress, they harped on the vendors who were NOT part of the original group, and they wanted to get in on the consortium. Six months after the first group, I believe there were 13 members, and 18 at the next. By the third meeting, PPP was Stable. PPP 'just worked', most of the time. PPP had become "easy", and SLIP was largely relagated to antique links on old equipment.

When I look at the various implementations of BIOS redirection to the serial port, they almost exclusively presume (or require) a modem connection, which then requires hardware level wiring issues if you are using a Console Server connection. The mapping of Function Keys, command sequences, and the timing issues to invoke the commands make using them tricky on a Console Server. It makes them impractical on any link slower than 19.2 Kbps. Yet the few BIOS makers don't see a need to try to do better, let alone to work with the other vendors to provide some consistancy of user interface to their customers. Do they know whether you care?

Now that Service Processors are becoming prevalent on new servers, it's hard to get a server without one. Yet they suffer from problems very similar to BIOS. The user interfaces are different from vendor to vendor. But what about processors from the same vendor. What about within the same vendor, across a hardware platform? You'd think that a vendor might want to keep their command sequences consistent, for the sake of their customers (who might be scripting support sequences for their machines). You'd probably like to think that...but you'd be wrong. A couple vendors seem to change at will, but I'm guessing that they are just buying server designs from other companies...and those same vendors probably haven't taken the time to design that Human User Interface, they probably simply rely on the hardware maker to fulfill the tasks (power cycle the server, set the IP address of the Ethernet interface for service processor, implement a web interface with certain features), leaving the details of keystrokes to the hardware maker...and when the vendor chooses another hardware maker, they get the same latitude with the design, and make different choices. (I wonder what Donald Norman thinks about tis problem.) Do your vendors know if you care?

When you tell a sales person that you want some new feature on their next year's model, the usual response is "OK, if we can get that done, how many will you buy?". It's going to cost them labor hours, engineering and QA time, technical documentation, maybe changing part of a training class. While you may only want 6, or 12, or 40, your voice will be tallied. When they from enough of you, it's going to be worth it to make a change. But we need to tip that scale by asking, and probably tying that request to some money. (My dad would sometimes ask me to "write your request on a $5 bill, and I'll consider it", but I have seen this over and over again, from different seats in different companies. For the vendor, it's all about money.)

My point here, is that you are going to spend money on new servers at some point. Make your money count! Tie the money to your requests for easier-to-use service processors. Decide what you want, and tell your possible vendors about your requirements. They all want your money...so vote with your wallet! Reward the responsive vendors! If the unresponsive vendors don't get the order, they'll at least know why, and they'll learn that you were serious. As more of us tie our requirements to purchasing, we can make a difference.

As you get into your busy end-of-year, some of you are already going through your budget cycle for 2009. Use any slowdown you get to think about requirements to go along with those purchases. Get them on paper, and in email, so they are ready to send to your vendors with your Request for Quote. And good luck in the new year.

-Z-

A little perl of wisdom

When it comes to finding some quick help (after the Helpdesk has closed), or those tidbits of knowledge and experience of someone who has tread the path before me, Google has been my friend for many years (taking up the mantle after AltaVista!).

But, there is a downside to learning all about a topic from Google. It is that you, the reader, are guiding the learning. You are getting the answers to the questions you ask...but it doesn't always offer things that you should know (or at least consider), based on your searches.

I taught myself perl, reading books, and using Google for answers, and examples...but I sometimes found myself confused when reading someone else's code. This year, I'm taking formal classes, from a few sources, hoping to patch a few holes in my foundation of knowledge.

While I'm glad to have taken this approach, to have someone guiding me through the learning, systematically, and answering my questions...I'm finding that the Teacher in this equation is even more important than the content. Even if someone knows the content well, that doesn't mean that they will be a good teacher. (Answering the occasional question does not equal teaching.)

After 5 weeks in a semester-long class at a local community college, I too a full-day Perl 101 course from Bayview Training...and then I coded for 15 hours straight, because that class had made a number of concepts more concrete for me (several "light-bulb" moments), and I was excited about the possibilities for my future projects.

I heartily recommend Bill Ward as an instructor. While he teaches primarily in Silicon Valley, he does travel for some classes and consulting as well. The cost of the course was money well spent! If you really need to learn some perl, check out Bayview Training and Bill Ward!

Big News! SSH clients for iPhone!

I haven't had any 'big news' on the console front for a couple months now. I have been missing my BlueConsole adapters (which I've blogged about before), and SSH access from my iPhone, BUT that has changed!

New in the Apple Apps Store are four SSH applications (use the search dialogue, look for "ssh"...). I've already bought two to try, and one has already met my most common needs. In a nutshell, here are the four I found, with URLs to the maker's websites, in case you want to do your own homework on the topic.

TouchTerm http://www.jbrink.net/touchterm/index.html $2.99 This one looks pretty good, from the online docs. There is only basic information on the maker's home page, BUT, check the SUPPORT page! Key Exchange, font colors, different input modes (immediate, batch, command history). The developer offers an email address for comments and suggestions. This looks like a good pick!

pTerm http://www.instantcocoa.com/products/pTerm/ $4.99 This maker also lists an email address, as well as a Google discussion group. The web page posts a short-term development schedule. They also boast that this app is based on PuTTY, and I'm a big PuTTY fan (but I understand that this app is only scratching the surface on features). I've sent email to the developer with my wants, and I already received a positive reply. He is also willing to work on the bluetooth-serial-devices interfacing as well.

iSSH http://www.zinger-soft.com/iSSH_features.html $4.99 This one also looks good. The main web page is odd... there are small, light-grey arrows under a few of the descriptions...they indicate that there are additional items at this menu level of the web page. Clicking on them will show you the other topics. This one also supports shared keys, but doesn't seem as flexible (to me, yet), as TouchTerm.

SSH http://www.throughput.biz/ $3.99 This looks simple, and it word-wraps the screen. There is little information on the maker's homepage. I'm currently not inclined to try it.

So, with this in mind, I have bought the first two (TouchTerm and pTerm), and I'm spending my efforts to put TouchTerm through it's paces first, because pTerm doesn't have RSA/DSA keys yet (though keys are on the short-range list for new features, so I'll be giving them both a good workout in September). I was able to set up a basic profile, with screen and font colors, RSA-2048 key, emailed the key to another host, so I could place the key, and use the session. It will take a bit of getting used to, since I'll need to leave the fonts a bit larger, and pan around the screen (using mutt was kinda awkward, but that won't be my primary use of SSH). I like it so far, and I hope to be able to put my BlueConsole adapters back into service sometime in the near future.

I hope you had a chance to enjoy some vacation this summer as well.

-Z-

Social Networking, professional connections

The question comes up, from time to time... Why do System Administrators seem to be better at (Social) Networking than Network Administrators. You'd think that Netadmins would have the communications thing down cold. But, I've found, we tend to have smaller social networks, but with links we can rely upon to do our jobs.

Linked in? Yep, I've been there a while...

But, why are there so many more Systems-related conferences than Network-related? OS related trade shows, versus Network trade shows.

I just find it odd, that's all.

-Z-