Whether you wear sports shoes, cowboy boots, or birkenstocks, the life of your shoe's soles are determined partly by how much walking you do in them. I'm a big fan of recreational walking, but I'd like to minimize the running around I need to do at the shop.
In the event of an emergency, getting on a serial console saves me from pushing a cart around, waiting for elevators, carrying a laptop around (and all the extra adapters, cables and power packs), just so I can get on the console of various devices, just so I can check configurations.
Yes, if the network is working, I can check a lot of the configurations across the network. But, sometimes interfaces die. Sometimes a typographic error will change an interface or network setting. Sometimes a cable gets unplugged accidentally. That's when your normal access breaks, and leaves you scrambling. Unless you have a console management network in place, and some strategicly placed console servers. (Better still, you should have something logging all of those remote consoles, but that's a topic for another post.)
Today I watched a friend cutting over to a new network over the holidays. Progress was being made, but it was slower than planned, and the days are starting to run out. He had developers from the new network gear vendor helping to debug the lack of interaction. While they all had their laptops, I watched as they wheeled a cart around the buildings, and up and down elevators, trying to check configurations because the network wasn't yet stable.
What was missing was a stable, simple console server deployment, with 8-16 ports in each of the main and intermediate frame rooms (MDF and IDFs). There was already fiber between the rooms. The console net could have been simple, stable, and independent of the main network. And it would have allowed them to be logged into many consoles at one time. They could have been watching errors and events on many devices as they tried tuning various settings.
Sure, this costs a bit of money to set up. But the price per port is low for simple, reliable gear. Consider the time for a couple contractors, and three developers, working over the holidays, trying to debug a problem. (I guess if your doing that work as an hourly worker, it's not so bad...but if you're the person in finance trying to close out the end-of-year books, and the network is being flaky, I imagine that your perspective about how soon the network should be stable would be different.)
I've written elsewhere about my portable Emergency Kit (a small hub, console server, adapters, cables, and canned telnet configurations to the console server). Today, I'm trying to lobby for you to consider a simple configuration, to support your current devices, with a bit of room to grow. Hopefully, you may find yourself trying to add more gear in preparation for a cutover, adding extra devices in every wiring closet, and you'll save yourself time and steps if you have some extra ports ready in each location.
Remote offices/sites deserve this consideration as well. I know many places that install a modem on the core router at their remote sites. But, what happens when the router relies on a TACACS or RADIUS server at that location, and the problem is not the router? You can dial in, get a prompt, but you can't authenticate, so you can't get into the router, or the authentication server. If you had a small console server there, and the modem allowed you into the console server, you would have a better chance of getting the access you need. (Even if you only see errors, you'd understand how to resolve the problems later...maybe the authentication server is trying to log your attempt by resolving to a DNS host that it cannot reach (since the network to the main office is down)? You know you need a DNS host in that office, or at least some static entries in the local hosts table for that host.)
Trust me about this...adding console servers isn't going to make you lazy. It WILL save you time, which you can spend doing some of the other tasks on your plate. It is well worth the investment of time and money to set it up.
-Zonker-
Showing posts with label console server. Show all posts
Showing posts with label console server. Show all posts
20070919
Insert Wire A into Slot B (specialty cables and adapters)
The other title for this article could be "Pocket Notes (you really CAN take it with you)". Here's how I 'remember' how to wire a serial crossover cable when I'm in a data center, without easy web access.
I used to carry a small pocket notebook with notes and adapter diagrams, but those got worn and ratty and dog-eared. So I kept them in a computer, printed the notes and drawings, and carried a small (5" x 8.5") binder. This way, I could mark-up the notes when I was away from the computer, and make changes on the computer, and print more 'clean' pages. It was also easy to give a page away, since I knew I could easily replace it. Eventually, the binder got bigger, thicker, and heavier, until I finally started carrying the computer around.
Now, ten years later, I keep my console information pages on the web, but I also keep critical notes on my Treo (Palm OS device). But the OS isn't important! If you have a pocket/palmtop computer, you can use the memo functions to carry useful cable info in the note files.
ASCII art isn't easy, and it isn't pretty unless you are using a mono-spaced font. But recording the wiring for specialty cables is easy. In an earlier posting, I described how to determine the wiring for specialty cables. And most of us could easily spout off the wiring sequence for an AT&T 468B-wired CAT-5 cable, right? So, you start with that color sequence on one end...but what colors do you need on the other end? THIS is easily described with text, and proportional fonts do not get in the way.
I usually show the cable, wired from both ends, since I can use a simple RJ-45 cable continuity tester...but, I add the Cat-5 color code in between the pin numbers. I use the AT&T 468B sequence on the first end, since it's a good visual starting place. Then, I show the sequence on the 'other' end, with the resulting color sequence. It's this other sequence that it hard to translate in my mind, so I keep a 'note' that won't get dog-eared in my phone, since it's almost always going to be with me.
Here's an example of a crossover cable between Cyclades ACS Console Servers;
Cyclades crossover
one end...
1 whi-orn 5
2 orn-whi 8
3 whi-grn 6
4 blu-whi 4
5 whi-blu 1
6 grn-whi 3
7 whi-brn 7
8 brn-whi 2
other end...
1 whi-blu 5
2 brn-whi 8
3 grn-whi 6
4 blu-whi 4
5 whi-orn 1
6 whi-grn 3
7 whi-brn 7
8 orn-whi 2
Using the PalmOS memopad feature, I have note files for a variety of console servers. It's easy to combine wiring notes (crossover, loopback, signal inputs/outputs) for a single vendor into a separate file, for quick reference. You should be able to do something similar with other PalmTop devices.
I do still keep post-its handy at a few sites, with the "other end" wiring sequence, in case the memos aren't available. (The battery died, or I can't take a camera-phone into certain data centers...) When they start to get worn, I make new ones, using the records on my phone, or my web pages.
I hope this note is helpful to some of you. I know that it saves me a LOT of time.
I used to carry a small pocket notebook with notes and adapter diagrams, but those got worn and ratty and dog-eared. So I kept them in a computer, printed the notes and drawings, and carried a small (5" x 8.5") binder. This way, I could mark-up the notes when I was away from the computer, and make changes on the computer, and print more 'clean' pages. It was also easy to give a page away, since I knew I could easily replace it. Eventually, the binder got bigger, thicker, and heavier, until I finally started carrying the computer around.
Now, ten years later, I keep my console information pages on the web, but I also keep critical notes on my Treo (Palm OS device). But the OS isn't important! If you have a pocket/palmtop computer, you can use the memo functions to carry useful cable info in the note files.
ASCII art isn't easy, and it isn't pretty unless you are using a mono-spaced font. But recording the wiring for specialty cables is easy. In an earlier posting, I described how to determine the wiring for specialty cables. And most of us could easily spout off the wiring sequence for an AT&T 468B-wired CAT-5 cable, right? So, you start with that color sequence on one end...but what colors do you need on the other end? THIS is easily described with text, and proportional fonts do not get in the way.
I usually show the cable, wired from both ends, since I can use a simple RJ-45 cable continuity tester...but, I add the Cat-5 color code in between the pin numbers. I use the AT&T 468B sequence on the first end, since it's a good visual starting place. Then, I show the sequence on the 'other' end, with the resulting color sequence. It's this other sequence that it hard to translate in my mind, so I keep a 'note' that won't get dog-eared in my phone, since it's almost always going to be with me.
Here's an example of a crossover cable between Cyclades ACS Console Servers;
Cyclades crossover
one end...
1 whi-orn 5
2 orn-whi 8
3 whi-grn 6
4 blu-whi 4
5 whi-blu 1
6 grn-whi 3
7 whi-brn 7
8 brn-whi 2
other end...
1 whi-blu 5
2 brn-whi 8
3 grn-whi 6
4 blu-whi 4
5 whi-orn 1
6 whi-grn 3
7 whi-brn 7
8 orn-whi 2
Using the PalmOS memopad feature, I have note files for a variety of console servers. It's easy to combine wiring notes (crossover, loopback, signal inputs/outputs) for a single vendor into a separate file, for quick reference. You should be able to do something similar with other PalmTop devices.
I do still keep post-its handy at a few sites, with the "other end" wiring sequence, in case the memos aren't available. (The battery died, or I can't take a camera-phone into certain data centers...) When they start to get worn, I make new ones, using the records on my phone, or my web pages.
I hope this note is helpful to some of you. I know that it saves me a LOT of time.
20070602
The Emergency Kit
I've put together a small kit, which can give me quick access to the serial consoles of various machines, for those little emergency trips to the colocation facility, or a customer site. It's been so successful and handy that colleagues are borrowing it for small cluster jumpstarts, and even for a Disaster Recovery exercise. So, I'd also describe it here.
The gist is, I started with a small 8-port console server, so I could use my laptop, a pre-configured telnet client, a crossover-Ethernet cable, and a handful of adapters to connect to various devices. I found that I often needed to be connected to the machine's network as well, so I added a small hub to the kit, as well as a small power outlet strip to my laptop bag.
I picked the Cyclades TS-800 8-port console server because it was small, was BREAK-safe, and ran from a 'wall wart' (power pack). I really wanted one that could run on 12-volts DC, so I could run the kit on car battery power, but couldn't find one. My laptop can also run from 12-volts DC (which could be handy in an emergency). Sadly, the TS-800 units are no longer sold by Avocent, but there are still a variety of small console servers on the market.
The small console server fit in the large pocket of my laptop bag, but limited how many cables I could carry, along with all the adapters. I finally fell to using a small, extra bag for the console bits.
I added a small 10-Mbps hub initially, so I could also do some packet sniffing, since most small switches don't have the ability to 'span' or 'mirror' ports. But, that bottleneck also throttled network access to the attached device, and I wasn't doing so much packet sniffing, so I 'traded up' to a Allied Telesyn AT-FS708LE 8-port 10/100 Mbps switch. This is a 12-volt DC device. It's small, lightweight, sturdy, and still available cheaply.
The kit now includes the switch, the console server, two crossover-Ethernet cables (1'- and 5'-long), an assortment of lengths of regular 8-wire CAT-5 cables, and an assortment of 24 Cyclades serial adapters. I also have a few 1'-long Cyclades null-modem (DCE-to-DTE crossed RJ-45 cables, and a few CAT-5e-rated female RF45 couplers. Add in the wall warts, and the kit lives in a 6-quart Sterilite plastic container about the size of a shoebox. Easy for security folks to see what is inside, but you don't want to put this in the belly of an airplane.
The Console Server is really the key to this kit, so here's what I've done to make it work;
First, configured the device as a console server, then copied the config file so I have an on-board copy, in case someone feels the need to change settings while they borrow it.
cp /etc/portslave/pslave.conf /etc/portslave/pslave.bkup
Make sure the backup config is saved into the unit's NVRAM. On the Cyclades, this means adding the file into /etc/config_files. (This is one reason I strip most of the comments out of my console server config files, since I'm basically saving two copies of the configuration file.) Now, if someone needs to change the config file, it's quick and easy to get back on the console, and copy the config back over the pslave.conf file, then 'saveconf' and reboot.
I also add an entry for my laptop into the /etc/hosts file on the console server, to reduce delays when making your reverse-TCP connections to the console server.
I had originally assigned different port speeds (19.2k, 38.8, 57.6, and 115.2) to ports 5 through 8, to make it quick and easy to access a high-speed port. But, I found that most ports are STILL at 9.6kbps as a default, and sometimes I needed more than 4 ports. In those cases when I needed a higher speed, I often needed more than one port at the higher speed at the same time. Since I found myself making many configuration changes I decided to revert the base configuration to 9.6k all 'round, but that's just explaining my reasons. You may want to do it your own way, and that's up to you.
I also pointed all the "extra" services (Radius server, syslog, etc) that the console server might want to use to the laptop IP address. I already use a small but of RFC-1918 address space, to reduce the chances of a conflict with the network that I may attach to. Pointing just to my laptop reduces the chance of the TS tripping any Intrusion Detection System (IDS) that might be watching the network. (I don't use the Syslog much anymore, but if I want to try it, that's one less part I need to configure.)
The current laptop is a PC, using PuTTY as the client tool. This lets me save the basic settings for "ts-1" through "ts-8" (configured to connect to the IP address of the console server, at ports 7001-7008, respectively). I use a distinctive window size and text/background colors for these windows, and each connection is configured with a unique 'window name', and a unique log-file name. The distinctive window settings make it easy to tell which open windows are serial connections, versus network sessions when I'm debugging.
An advantage to using PuTTY for this, is I can also 'export' a copy the Windows Registry settings for these sessions onto my FLASH fob, and easily give the settings to someone who is going to borrow the kit, and VIOLA! They are ready to connect to the ports quickly and easily. PuTTY is small, and runs without needing to be 'installed', so I have a copy of that on the FLASH fob as well.
The final bit that makes the kit as easy to loan as it is for me to use it, is a 1-page README document (also on the FLASH fob...with a canned Conserver config snippet for this device, just in case), which outlines the default IP settings, serial speed settings, adapter and cabling clues, and basics about how to open a reverse-TCP session to a serial port. I also include clues and the password for configuring the device. (Add a few useful labels to the hub and console server, mark the wall-wart connectors with a color-code at the device-end, and things go pretty smoothly.)
I don't use it very often, but when I need it, it's ready to go! When your time is worth money, having a kit on the shelf is money well spent!
-Z-
The gist is, I started with a small 8-port console server, so I could use my laptop, a pre-configured telnet client, a crossover-Ethernet cable, and a handful of adapters to connect to various devices. I found that I often needed to be connected to the machine's network as well, so I added a small hub to the kit, as well as a small power outlet strip to my laptop bag.
I picked the Cyclades TS-800 8-port console server because it was small, was BREAK-safe, and ran from a 'wall wart' (power pack). I really wanted one that could run on 12-volts DC, so I could run the kit on car battery power, but couldn't find one. My laptop can also run from 12-volts DC (which could be handy in an emergency). Sadly, the TS-800 units are no longer sold by Avocent, but there are still a variety of small console servers on the market.
The small console server fit in the large pocket of my laptop bag, but limited how many cables I could carry, along with all the adapters. I finally fell to using a small, extra bag for the console bits.
I added a small 10-Mbps hub initially, so I could also do some packet sniffing, since most small switches don't have the ability to 'span' or 'mirror' ports. But, that bottleneck also throttled network access to the attached device, and I wasn't doing so much packet sniffing, so I 'traded up' to a Allied Telesyn AT-FS708LE 8-port 10/100 Mbps switch. This is a 12-volt DC device. It's small, lightweight, sturdy, and still available cheaply.
The kit now includes the switch, the console server, two crossover-Ethernet cables (1'- and 5'-long), an assortment of lengths of regular 8-wire CAT-5 cables, and an assortment of 24 Cyclades serial adapters. I also have a few 1'-long Cyclades null-modem (DCE-to-DTE crossed RJ-45 cables, and a few CAT-5e-rated female RF45 couplers. Add in the wall warts, and the kit lives in a 6-quart Sterilite plastic container about the size of a shoebox. Easy for security folks to see what is inside, but you don't want to put this in the belly of an airplane.
The Console Server is really the key to this kit, so here's what I've done to make it work;
First, configured the device as a console server, then copied the config file so I have an on-board copy, in case someone feels the need to change settings while they borrow it.
cp /etc/portslave/pslave.conf /etc/portslave/pslave.bkup
Make sure the backup config is saved into the unit's NVRAM. On the Cyclades, this means adding the file into /etc/config_files. (This is one reason I strip most of the comments out of my console server config files, since I'm basically saving two copies of the configuration file.) Now, if someone needs to change the config file, it's quick and easy to get back on the console, and copy the config back over the pslave.conf file, then 'saveconf' and reboot.
I also add an entry for my laptop into the /etc/hosts file on the console server, to reduce delays when making your reverse-TCP connections to the console server.
I had originally assigned different port speeds (19.2k, 38.8, 57.6, and 115.2) to ports 5 through 8, to make it quick and easy to access a high-speed port. But, I found that most ports are STILL at 9.6kbps as a default, and sometimes I needed more than 4 ports. In those cases when I needed a higher speed, I often needed more than one port at the higher speed at the same time. Since I found myself making many configuration changes I decided to revert the base configuration to 9.6k all 'round, but that's just explaining my reasons. You may want to do it your own way, and that's up to you.
I also pointed all the "extra" services (Radius server, syslog, etc) that the console server might want to use to the laptop IP address. I already use a small but of RFC-1918 address space, to reduce the chances of a conflict with the network that I may attach to. Pointing just to my laptop reduces the chance of the TS tripping any Intrusion Detection System (IDS) that might be watching the network. (I don't use the Syslog much anymore, but if I want to try it, that's one less part I need to configure.)
The current laptop is a PC, using PuTTY as the client tool. This lets me save the basic settings for "ts-1" through "ts-8" (configured to connect to the IP address of the console server, at ports 7001-7008, respectively). I use a distinctive window size and text/background colors for these windows, and each connection is configured with a unique 'window name', and a unique log-file name. The distinctive window settings make it easy to tell which open windows are serial connections, versus network sessions when I'm debugging.
An advantage to using PuTTY for this, is I can also 'export' a copy the Windows Registry settings for these sessions onto my FLASH fob, and easily give the settings to someone who is going to borrow the kit, and VIOLA! They are ready to connect to the ports quickly and easily. PuTTY is small, and runs without needing to be 'installed', so I have a copy of that on the FLASH fob as well.
The final bit that makes the kit as easy to loan as it is for me to use it, is a 1-page README document (also on the FLASH fob...with a canned Conserver config snippet for this device, just in case), which outlines the default IP settings, serial speed settings, adapter and cabling clues, and basics about how to open a reverse-TCP session to a serial port. I also include clues and the password for configuring the device. (Add a few useful labels to the hub and console server, mark the wall-wart connectors with a color-code at the device-end, and things go pretty smoothly.)
I don't use it very often, but when I need it, it's ready to go! When your time is worth money, having a kit on the shelf is money well spent!
-Z-
20070413
Kudos to the Cyclades folks
Their product was developed across three continents, and sometimes you could find grammatical errors in their on-line tech notes (perhaps because the authors first language was not English?), but I have not found any techical errors in the command-line instructions of the older files.
To be honest, I occassionally took some liberties, trying to optimize the steps that they had outlined. But, if things didn't work the way I expected, I have always had success when I did it exactly the way they typed it.
As someone who owns and operates on a bunch of older Cyclades gear, I'm very thankful that Avocent has retained the old Cyclades files at http://global.cyclades.com. Access to the old information has proven to be invaluable for upgrading older units to newer code.
And, as a fan of documentation (see my Console Connection Guides...), I tip my hats to those who made sure that the original instructions were spot-on correct, every time! Thanks, folks! I hope the Avocent team will follow well in your footsteps.
-Z-
To be honest, I occassionally took some liberties, trying to optimize the steps that they had outlined. But, if things didn't work the way I expected, I have always had success when I did it exactly the way they typed it.
As someone who owns and operates on a bunch of older Cyclades gear, I'm very thankful that Avocent has retained the old Cyclades files at http://global.cyclades.com. Access to the old information has proven to be invaluable for upgrading older units to newer code.
And, as a fan of documentation (see my Console Connection Guides...), I tip my hats to those who made sure that the original instructions were spot-on correct, every time! Thanks, folks! I hope the Avocent team will follow well in your footsteps.
-Z-
20070408
Going back to the beginning
"I am waiting for you, Vizzni! You told me to go back to the beginning, so I have. When a job went wrong, you went back to the beginning!" - Inigo Montoya, Princess Bride -
Today's topic is a lesson in Change Control, sort of. More like Pre-Change Control.
Most manufacturers post their upgrade instructions. Some of these instructions are fine. Some are even exceptional. But few manufacturers offer instructions for how to downgrade their code. Yet, if an upgrade goes wrong, you will probably want a fall-back plan, to get you back where you were, at least until you can get the vendor to help past the problem(s).
Most console server devices store their OS and boot files in FLASH. But many of them cannot store more than one version of the OS in the on-board FLASH at a time. So, the upgrade process will overwrite the old version. If you plan to revert to the original version, you need to gather a few things before you start;
If your device has MD5 implemented, so that you can check the new version for integrity, you should also use MD5 to display the checksum for the current version before you start. (If you need to load the old version back again, you should know what the checksum will be.)
You keep all of the software CDs that came with your devices when they are new, right? (At least ONE copy for each platform, and for each different software version as you collect them.) You should also mark them (the CDs, Manuals, etc.) for the software version they contain. The cost of a bookcase is cheaper then an evening of your wasted time because you cannot easily find the information you need.
"But my vendor keeps all of this stuff on-line. Why bother to keep the old stuff?" you ask. Let me offer a few cases that happen more often than you might expect;
Keep the versions in separate directories, since sometimes the image names are identical, and you don't want to overwrite the old files when you add the newer files. Then, you will be ready to upgrade when the time comes, and you'll also be ready to downgrade again, if needed.
-Z-
Today's topic is a lesson in Change Control, sort of. More like Pre-Change Control.
Most manufacturers post their upgrade instructions. Some of these instructions are fine. Some are even exceptional. But few manufacturers offer instructions for how to downgrade their code. Yet, if an upgrade goes wrong, you will probably want a fall-back plan, to get you back where you were, at least until you can get the vendor to help past the problem(s).
Most console server devices store their OS and boot files in FLASH. But many of them cannot store more than one version of the OS in the on-board FLASH at a time. So, the upgrade process will overwrite the old version. If you plan to revert to the original version, you need to gather a few things before you start;
- A copy of the current code version
(You may be able to TFTP the file to a server from the device!) - At least a byte-count for the installed currently installed version
- Better is to have the MD5 checksum hash for the currently installed version
If your device has MD5 implemented, so that you can check the new version for integrity, you should also use MD5 to display the checksum for the current version before you start. (If you need to load the old version back again, you should know what the checksum will be.)
You keep all of the software CDs that came with your devices when they are new, right? (At least ONE copy for each platform, and for each different software version as you collect them.) You should also mark them (the CDs, Manuals, etc.) for the software version they contain. The cost of a bookcase is cheaper then an evening of your wasted time because you cannot easily find the information you need.
"But my vendor keeps all of this stuff on-line. Why bother to keep the old stuff?" you ask. Let me offer a few cases that happen more often than you might expect;
- The vendor goes out of business, and the website goes away.
- The vendor sells the product line, and the new vendor kills it off.
- The vendor gets acquired, and the old website goes away.
- The vendor decides old software is "support", and is worth money.
- Newer versions of software are too big to fit into the FLASH in your device.
- The upgrade instructions had a mistake.
- The new code doesn't like your old hardware.
- The vendor decides to remove old versions from the web, to encourage the adoption of newer versions.
Keep the versions in separate directories, since sometimes the image names are identical, and you don't want to overwrite the old files when you add the newer files. Then, you will be ready to upgrade when the time comes, and you'll also be ready to downgrade again, if needed.
-Z-
Subscribe to:
Posts (Atom)