20071129

All Together Now...

You use Console Servers to manage the serial ports of various devices, right? So, it only makes sense that a 'Best Practice' would be to connect the console ports of your Console Servers to a port on a console server, so you can monitor the Console Servers, too, right?

BUT, to do this 'right', you need to connect it to another, different console server. After all, when do you really need to use the console? When the machine is having a problem, and WHEN THE DEVICE REBOOTS! (All Caps used for emphasis...sorry if I offended anyone. ;-) When the device reboots, your serial console is how you see what the machine is doing until it gets going sufficiently for the network to be of use to you. So, if you connect your serial port to another port on the same Console server, you can monitor it when all is going well, but when things go badly, you've just lost your console access as well.

Tonight, I hit a bit of a snag...because I had connected some Console Server console ports to other "nearby" Console Servers. (Imagine 18" console jumper cables...why not, the 'different' units were close enough.) "What could be the problem with that", I had thought.

Tonight was the night to upgrade the software on this set of Console Servers...complete with systematic reboots between revision levels. Doing the three in parallel, to take advantage of 'cut-and-paste' to minimize the chances for typographical errors. Halfway through the reboots, I stop seeing progress...because the reboot of Console Server A had taken out the console access for Console Server B...

So, I type this note as I wait for the Console Servers to reboot and for my console access to stabilize again, and I consider whether my tie would have been worth the extra ~125 feet of cable to make the console runs to diversely-disparate Console Servers, to minimize the chance of a 'cluster service' outage taking out closely-grouped Console Servers. Food for thought.

-Z-