Dell R6xx, 7xx series servers (iDRAC and Console)

Here are two clues that will help you plan for installing the new Dell server families.

They seem to be great servers... lots of capacity for RAM, lots of CPU strength, and I love the rack mounting rail system. I also like the iDRAC service processors.

What's not to like? The back of the chassis is too thick (or the DE9 console port is recessed too far). Either way, most of the serial console adapters (DE9 to RJ45) won't fit well, and provide reliable connections, unless you grind down the cases of the adapters. And, since the adapter vendors don't offer to sell you a "trimmed" version, that job is going to cost you extra, somehow. Hey, maybe we can get the intern to do it? ;-)

I'm *really* glad that the serial consoles are still on the back of the servers, but if you are going to put them there, it seems to me that making the serial adapters fit should be part of the plan to have them on the server.

The one adapter I've found that works on the R620 series servers is from Cyclades (ADB0036). Yes, you can still get them from CDW. Fortunately, our latest batch went into racks with Cyclades TS2000. But, I did need to modify SO MANY of the Lantronix adapters that I made a custom jig for a Dremel router, to get consistent results.

Clue number two: On the R620, ttyS0 goes tot he iDRAC, and ttyS1 goes to the DE9 on the rear of the server. You will need to start an agetty for BOTH of these, if you want to be able to use both of them.

My other big projects are nearly done, and I plan to post much more often this summer. Thanks for reading.

How did the year get by me?

Wow, I'm surprised that a year has passed since my last post. Largely, it has been due to my inability to get the editing tools to play well with the iPhone and now the iPad. Normally, I don't update my stuff on employer time, or employer equipment. Couple this with a VERY busy year, and me putting updates lower on the priority list, and here we are on Memorial Day weekend.

A lot has happened in the past 12 months. Too much for a single post, but let me sum up.

I'm working with a variety of Service Processors this year, making the work, and documenting what I've learned. Besides working at the BIOS level, I'm trying to determine if/how the settings can be modified from the OS's involved, and that is more difficult than I supposed. The main problem seems like the websites holding the IPMI drivers or libraries have disappeared after 3-4 years. (The biggest lesson here is to capture, collect, and archive these files for any hardware you acquire. Mergers and acquisitions contribute to the loss of the files, even as the instructions make it into a knowledge base.)

An old lesson re-learned, is that older Sun gear will ignore their serial console settings as soon as you plug a keyboard in! Your only clue, if you are logging the console output with Conserver or some other application, is a small line about detecting a keyboard. Removing the keyboard does NOT revert back to serial console output, you need to reboot the server with no keyboard attached. I knew this long ago, but then was thwarted when folks in a remote office attached a keyboard, and we didn't notice that console I/O had gone away.

I've also been doing a LOT of work with SNMP in the past 6 months, and I found an oddity with an MGE (which sold the line to APC, who then sold the line to Schneider Electric) room-sized UPS. The brains of the UPS uses a serial line to pass a data stream to the Network Interface shelf. If this streaming interface is removed, the SNMP cards do not get an update... But the cards simply report the LAST value for any queried OID. As a result, your only clue of a disconnect is seeing all results "flat-line". BUT, if you are also set up forSNMP traps, you will get also see a string that communications was lost with the UPS. This is a case where you really need to be able to parse Syslog and SNMP traps for interesting strings and send an alert.

Other UPS problems led to the integration of a UPS of Last Resort, so that core devices could track and log events with the big UPS. This included watching the serial output of the APC Smart-UPS 2200XL, plus SNMP integration into our monitoring system.

Another Big Project in the past year was due to a planned network outage. This was due to a planned partial power outage in the building. Unfortunately, some key devices also lost power, causing trouble, which could not send alerts via email because the network was down. We decided that we needed to make sure there was a second path out of the data center for critical alerts and messages. This led to integration with a FoxBox, a Network-connected cell-phone device, allowing SMS alerting AND control!

I've also added many more hosts to the host-to adapter database, but I only have a basic Cyclades console connection guide posted, and I still need to make a Digi page. (Cyclades was another fine example of collecting information from the web as soon as you hear about an acquisition!)

I'll make stronger efforts to post more frequently through the rest of 2010. thanks for reading.

-Zonker-

It's OK to talk back...

With the new year came some new toys, but I still have the same old Doctor's Bag, so I needed to find out what needed to stay, and what could be combined or made smaller.

One of the relatively large things were a couple of DB25 Loopback modules, from Cyclades (one included with each server, so I have a bunch). To use this useful tool, I also have a collection of DB25 adapters for a few of the console servers I often work with.

I'd tried using some simple loopback plugs...just short wires into an RJ45 plug (sometimes called an 'ice cube') like the one you also get with a new Cyclades console server...but these get lost in places, sometimes even lost in console server ports if bigger gear is mounted above and below the console server, so I gave up on them.

My next attempt was a 'pigtail'...a 4"-6" section of cable into the 'ice cube'. This was a 'flag', to help me find it sticking out of the console server, but I still had the trouble getting these out of console servers which had been mounted in tight corners.

All the while, I still needed a way to check the signal at the other end of the cable. My real concern was usually "do I have good connectivity up to the back of the machine?", and this required a loopback at the device in question, so I started packing the loopback modules.

I'd thought about taking an RJ45 socket out of a patch-panel, and looping wires around in that. It would be smaller and lighter, but how do I label them. I thought about using color, like the hood colors on the console server adapters. (As baby boomers get older, color coding lets one get by without reading glasses, at least some of the time.) I'd tried a variety of discarded sockets, with some success, but they all had flaws.

I finally stumbled on a part that I like for the purpose! Panduit's Minicom line of CAT-3 patch panel parts. You can get them in the common resistor color-code colors, they are compact, rugged, you can buy them in small quantites, and they have a flat spot you can use for labeling the jack. I'll try to get a couple pictures on Flickr, so I can link them here (and update my Doctor's Bag page). The only real modification to make is to round the sharp corners with some sandpaper, so they don't cause problems in the bag.

-Z-

All Together Now...

You use Console Servers to manage the serial ports of various devices, right? So, it only makes sense that a 'Best Practice' would be to connect the console ports of your Console Servers to a port on a console server, so you can monitor the Console Servers, too, right?

BUT, to do this 'right', you need to connect it to another, different console server. After all, when do you really need to use the console? When the machine is having a problem, and WHEN THE DEVICE REBOOTS! (All Caps used for emphasis...sorry if I offended anyone. ;-) When the device reboots, your serial console is how you see what the machine is doing until it gets going sufficiently for the network to be of use to you. So, if you connect your serial port to another port on the same Console server, you can monitor it when all is going well, but when things go badly, you've just lost your console access as well.

Tonight, I hit a bit of a snag...because I had connected some Console Server console ports to other "nearby" Console Servers. (Imagine 18" console jumper cables...why not, the 'different' units were close enough.) "What could be the problem with that", I had thought.

Tonight was the night to upgrade the software on this set of Console Servers...complete with systematic reboots between revision levels. Doing the three in parallel, to take advantage of 'cut-and-paste' to minimize the chances for typographical errors. Halfway through the reboots, I stop seeing progress...because the reboot of Console Server A had taken out the console access for Console Server B...

So, I type this note as I wait for the Console Servers to reboot and for my console access to stabilize again, and I consider whether my tie would have been worth the extra ~125 feet of cable to make the console runs to diversely-disparate Console Servers, to minimize the chance of a 'cluster service' outage taking out closely-grouped Console Servers. Food for thought.

-Z-

Whether we need DCD more than symetry

I'm very much in favor of a symmetrical RJ45 wiring schema for console servers, because it could minimize how many adapters I would need to stock. The advantages of a symmetrical pinout include using a simple 'rolled' cable to make a 'null-modem' connection between devices and adapters.

Now, I'm not picking any particular vendors wiring scheme as 'the best'. I'm espousing symmetry as the 'best way'. The problem is, this means having two Signal Ground pins, which usually means that DCD won't have a pin. (On an 8-wire RJ45 interface, you would have two ground leads, two data leads, two hardware flow control leads, and two hardware handshaking leads.)

Remember that today's Console Server has its roots in yesteryears Terminal Servers. The old TS used modems or terminals, to let remote users connect to hosts across the network. The hardware handshaking was used by the terminals to indicate if they were connected. With modems, they could be connected and powered, but it was DCD signals from the modem to the TS which told the TS whether an interface was 'on-line' with a remote user.

But, these days in console service, modems are a pretty rare thing. Still, I'll be the first to recognize that modems could be used to dial into a remote site and try to connect when the main WAN links are down. However, you can configure most modems to toggle the DSR lead to reflect the status of DCD, and you can configure most console servers to recognize the DSR lead as a signal that the device isn't ready. (So, I don't think we really need DCD in console service. I'll leave Dial-up Networking for later article.)

If you look at my Signals Page, in the RJ45 section, you'll notice that there are already a couple of symmetrical pinout schemes. But you can also see the wide variety of combinations that I've found so far. While pin 4 seems to be the most common pin for signal ground, its clear that not everyone uses pin for for the signal ground. That makes it harder to make an RJ45 signal tracer that works in all cases, so there is less incentive for a vendor to take on that project.

So, for the existing symmetrical RJ45 pinouts on my signals page, you'll notice that they use pins 4 and 5 as ground pins. Data pins are on pins 3 and 6. Handshaking is on pins 2 and 7, and flow control are on pins 1 and 8.

If you're asking "why can't we use pin 5 for DCD?", then you may not have thought about the issue deeply. The first problem comes when you use a rolled cable as a null modem, and you now connect two devices signal leads, but not their signal ground pins! You'll be replying on the chassis ground connections to provide the signal reference. Using these console servers in large data centers means that you may be connecting devices on different phases of the AC circuit, or even across two different Power Distrubtion Units (PDUs) or Uninterruptible Power Supplies (UPSs). The result could be burned out serial ports on your console server and the attached devices, or more sever damage to the units.

So, to summarize the vendors I know about today;

Symmetrical, with 2 grounds (9+ of 26):
Cisco, Logical Solutions SCS, Lantronics SCS-1600, Digi PortServer II, Xyplex 1600/1800, iTouch/MRV in-Reach lines, Juniper Networks (J-series devices, and many NetScreen devices), Server Technology CDU consoles, and Sun Microsystems console/TTY ports

Symmetrical, except for 1 ground and DCD (3 of 26):
Digi CM, Digi STS-1610, Opengear CM

Of the 27 console connections I've documented, 15 use pin 4 for ground, and 9 use pin 6 for ground. (Keep that in mind if you are making your own RJ45 signal tracers.) And of the 9 symmetrical pinouts, eight of them use pins 4 and 5 for ground, and one uses pin6 3 and 6.

It seems to me that other vendors (who are not making console servers) have been choosing the Cisco wiring scheme (Juniper, Sun, Server Technology). Lantronix picked the null-modem compliment to the Cisco pinout. Those choices have been good for me, since it simplifies some of the deployments that I work on. It's interesting to note that the Xyplex/iTouch/MRV pinouts are almost identical, except that pins 1 and 8 (the flow control leads) are inverted from the Cisco schema. (So, if you don't care about flow control, you can mix-and-match these devices with the Cisco-schema cousins.)

I'm under the impression that some vendors are considering adopting a different wiring scheme for their future product series. I'd strongly urge that they consider adopting a symmetrical schema, and I'd also strongly encourage them to consider using an existing schema, rather than trying to come up with a unique schema. And I'm always interested in discussing it in person, though an email on the topic is also welcome.

Insert Wire A into Slot B (specialty cables and adapters)

The other title for this article could be "Pocket Notes (you really CAN take it with you)". Here's how I 'remember' how to wire a serial crossover cable when I'm in a data center, without easy web access.

I used to carry a small pocket notebook with notes and adapter diagrams, but those got worn and ratty and dog-eared. So I kept them in a computer, printed the notes and drawings, and carried a small (5" x 8.5") binder. This way, I could mark-up the notes when I was away from the computer, and make changes on the computer, and print more 'clean' pages. It was also easy to give a page away, since I knew I could easily replace it. Eventually, the binder got bigger, thicker, and heavier, until I finally started carrying the computer around.

Now, ten years later, I keep my console information pages on the web, but I also keep critical notes on my Treo (Palm OS device). But the OS isn't important! If you have a pocket/palmtop computer, you can use the memo functions to carry useful cable info in the note files.

ASCII art isn't easy, and it isn't pretty unless you are using a mono-spaced font. But recording the wiring for specialty cables is easy. In an earlier posting, I described how to determine the wiring for specialty cables. And most of us could easily spout off the wiring sequence for an AT&T 468B-wired CAT-5 cable, right? So, you start with that color sequence on one end...but what colors do you need on the other end? THIS is easily described with text, and proportional fonts do not get in the way.

I usually show the cable, wired from both ends, since I can use a simple RJ-45 cable continuity tester...but, I add the Cat-5 color code in between the pin numbers. I use the AT&T 468B sequence on the first end, since it's a good visual starting place. Then, I show the sequence on the 'other' end, with the resulting color sequence. It's this other sequence that it hard to translate in my mind, so I keep a 'note' that won't get dog-eared in my phone, since it's almost always going to be with me.

Here's an example of a crossover cable between Cyclades ACS Console Servers;

Cyclades crossover
one end...
1 whi-orn 5
2 orn-whi 8
3 whi-grn 6
4 blu-whi 4
5 whi-blu 1
6 grn-whi 3
7 whi-brn 7
8 brn-whi 2

other end...
1 whi-blu 5
2 brn-whi 8
3 grn-whi 6
4 blu-whi 4
5 whi-orn 1
6 whi-grn 3
7 whi-brn 7
8 orn-whi 2

Using the PalmOS memopad feature, I have note files for a variety of console servers. It's easy to combine wiring notes (crossover, loopback, signal inputs/outputs) for a single vendor into a separate file, for quick reference. You should be able to do something similar with other PalmTop devices.

I do still keep post-its handy at a few sites, with the "other end" wiring sequence, in case the memos aren't available. (The battery died, or I can't take a camera-phone into certain data centers...) When they start to get worn, I make new ones, using the records on my phone, or my web pages.

I hope this note is helpful to some of you. I know that it saves me a LOT of time.

The Emergency Kit

I've put together a small kit, which can give me quick access to the serial consoles of various machines, for those little emergency trips to the colocation facility, or a customer site. It's been so successful and handy that colleagues are borrowing it for small cluster jumpstarts, and even for a Disaster Recovery exercise. So, I'd also describe it here.

The gist is, I started with a small 8-port console server, so I could use my laptop, a pre-configured telnet client, a crossover-Ethernet cable, and a handful of adapters to connect to various devices. I found that I often needed to be connected to the machine's network as well, so I added a small hub to the kit, as well as a small power outlet strip to my laptop bag.

I picked the Cyclades TS-800 8-port console server because it was small, was BREAK-safe, and ran from a 'wall wart' (power pack). I really wanted one that could run on 12-volts DC, so I could run the kit on car battery power, but couldn't find one. My laptop can also run from 12-volts DC (which could be handy in an emergency). Sadly, the TS-800 units are no longer sold by Avocent, but there are still a variety of small console servers on the market.

The small console server fit in the large pocket of my laptop bag, but limited how many cables I could carry, along with all the adapters. I finally fell to using a small, extra bag for the console bits.

I added a small 10-Mbps hub initially, so I could also do some packet sniffing, since most small switches don't have the ability to 'span' or 'mirror' ports. But, that bottleneck also throttled network access to the attached device, and I wasn't doing so much packet sniffing, so I 'traded up' to a Allied Telesyn AT-FS708LE 8-port 10/100 Mbps switch. This is a 12-volt DC device. It's small, lightweight, sturdy, and still available cheaply.

The kit now includes the switch, the console server, two crossover-Ethernet cables (1'- and 5'-long), an assortment of lengths of regular 8-wire CAT-5 cables, and an assortment of 24 Cyclades serial adapters. I also have a few 1'-long Cyclades null-modem (DCE-to-DTE crossed RJ-45 cables, and a few CAT-5e-rated female RF45 couplers. Add in the wall warts, and the kit lives in a 6-quart Sterilite plastic container about the size of a shoebox. Easy for security folks to see what is inside, but you don't want to put this in the belly of an airplane.

The Console Server is really the key to this kit, so here's what I've done to make it work;

First, configured the device as a console server, then copied the config file so I have an on-board copy, in case someone feels the need to change settings while they borrow it.
cp /etc/portslave/pslave.conf /etc/portslave/pslave.bkup

Make sure the backup config is saved into the unit's NVRAM. On the Cyclades, this means adding the file into /etc/config_files. (This is one reason I strip most of the comments out of my console server config files, since I'm basically saving two copies of the configuration file.) Now, if someone needs to change the config file, it's quick and easy to get back on the console, and copy the config back over the pslave.conf file, then 'saveconf' and reboot.

I also add an entry for my laptop into the /etc/hosts file on the console server, to reduce delays when making your reverse-TCP connections to the console server.

I had originally assigned different port speeds (19.2k, 38.8, 57.6, and 115.2) to ports 5 through 8, to make it quick and easy to access a high-speed port. But, I found that most ports are STILL at 9.6kbps as a default, and sometimes I needed more than 4 ports. In those cases when I needed a higher speed, I often needed more than one port at the higher speed at the same time. Since I found myself making many configuration changes I decided to revert the base configuration to 9.6k all 'round, but that's just explaining my reasons. You may want to do it your own way, and that's up to you.

I also pointed all the "extra" services (Radius server, syslog, etc) that the console server might want to use to the laptop IP address. I already use a small but of RFC-1918 address space, to reduce the chances of a conflict with the network that I may attach to. Pointing just to my laptop reduces the chance of the TS tripping any Intrusion Detection System (IDS) that might be watching the network. (I don't use the Syslog much anymore, but if I want to try it, that's one less part I need to configure.)

The current laptop is a PC, using PuTTY as the client tool. This lets me save the basic settings for "ts-1" through "ts-8" (configured to connect to the IP address of the console server, at ports 7001-7008, respectively). I use a distinctive window size and text/background colors for these windows, and each connection is configured with a unique 'window name', and a unique log-file name. The distinctive window settings make it easy to tell which open windows are serial connections, versus network sessions when I'm debugging.

An advantage to using PuTTY for this, is I can also 'export' a copy the Windows Registry settings for these sessions onto my FLASH fob, and easily give the settings to someone who is going to borrow the kit, and VIOLA! They are ready to connect to the ports quickly and easily. PuTTY is small, and runs without needing to be 'installed', so I have a copy of that on the FLASH fob as well.

The final bit that makes the kit as easy to loan as it is for me to use it, is a 1-page README document (also on the FLASH fob...with a canned Conserver config snippet for this device, just in case), which outlines the default IP settings, serial speed settings, adapter and cabling clues, and basics about how to open a reverse-TCP session to a serial port. I also include clues and the password for configuring the device. (Add a few useful labels to the hub and console server, mark the wall-wart connectors with a color-code at the device-end, and things go pretty smoothly.)

I don't use it very often, but when I need it, it's ready to go! When your time is worth money, having a kit on the shelf is money well spent!

-Z-

Kudos to the Cyclades folks

Their product was developed across three continents, and sometimes you could find grammatical errors in their on-line tech notes (perhaps because the authors first language was not English?), but I have not found any techical errors in the command-line instructions of the older files.

To be honest, I occassionally took some liberties, trying to optimize the steps that they had outlined. But, if things didn't work the way I expected, I have always had success when I did it exactly the way they typed it.

As someone who owns and operates on a bunch of older Cyclades gear, I'm very thankful that Avocent has retained the old Cyclades files at http://global.cyclades.com. Access to the old information has proven to be invaluable for upgrading older units to newer code.

And, as a fan of documentation (see my Console Connection Guides...), I tip my hats to those who made sure that the original instructions were spot-on correct, every time! Thanks, folks! I hope the Avocent team will follow well in your footsteps.

-Z-