Ready to "Wring Out the Old"...

 2017 was wrapped up with a weeks-long 12+ hour days effort to move a data center "live" (that is, keep as much online as possible, while shifting it all to another building!), in an effort that ended 2 days before Christmas, because the building we had been in was going to be demolished! We got it done, but that was just the start of maintaining that operation while the campus around us was demolished, and significant renovations were being made to the building that we were now in. Going to the data center included suiting up in boots, jeans, hardhat, safety vest and safety glasses. Construction continues even today, on 5 of the 6 sides of the current data center. COVID-19  certainly played a part in extending the length of the construction project, and the end is now foreseeable.

  In January of 2021, I expect to see President Joe Biden sworn in as the 46th President of the US. We may even see the first inoculations for COVID-19 start in the US. In the Spring of 2021, I expect to be relocating to SW Oregon, and selling my California home. We've all been adapting to "working from home", as companies have had to change and adapt to remote working. With that move comes the decisions of what to pack, and where will it live in the new house. Older console servers from the test lab? Manuals and CDs from past vendors (example: Cyclades), with docs and firmware for older hardware. Even bins of serial adapters and specialty cables. Some of that stuff isn't going to "make the cut". This will be much of the "Wringing Out" in the next few months...

  In the past 3 years, I've been making LED Art with Arduino boards for fun (see https://www.instructables.com/IKEA-Star-With-ATtiny-and-NeoPixels/ for an example) and environmental monitors. I also got a Prusa MK2S 3D printer, and I'm learning how to make things. I was a supporter of https://www.hackerdojo.com, but COVID-19 has closed most of our maker spaces. I helped teach embedded computing to Jr High students for a year (1-day every couple weeks, in an Introduction to Technology class). I was at every (SF) Bay Area Maker Faire until there was no more Maker Faire. 

  I was wrestling with whether to start a new Maker-oriented blog as a fork, and try to keep that up after the move. But all the good names I could think of has been taken on blogspot, many years ago. Sadly, most of them had far fewer entries... some seemed to only try a few test posts, and then they want idle. Some started back in 2001, but none updated after 2014. :-( As a result, I think I'll stick with this blog, and post what I like going forward (versus only posting console-related things). 

  I still get the occasional email, asking for something that's not on my pages at https://www.conserver.com/consoles/, but I haven't needed to set up any hardware to recreate a problem for a long while. It may be Murphy's Law that after I toss out old hardware, that's when folks will have a problem that I'd want to look at myself. But since Murphy isn't going to help me load and unload the trucks, I'll learn to live with not having the hardware.

  I'm also exploring Raspberry Pi as I explore new Digital modes in Ham Radio (DMR, New Packet, and some HF modes next year), as I decide which monitors will be useful for Pi projects. There may be some interesting serial-related posts about these in the future.

  I hope for good luck, and good health for all of us in the new year.

Is USB the new Serial Console?

I'm happy to report that I finally have access to a Cisco device with a USB console (a gently-used C1900 router), so I can experiment a bit with it. I also have had an interesting briefing about how these USB console interconnects came to be at Cisco, and some caveats.

As I understand it, essentially they have embedded a USB-to-EIA-232 bridge chip into the Cisco device. When you connect a PC, and you have suitable drivers, it simply appears to your machine to be a simple COM port. (I'm presuming that a MAC will see s suitable TTY and etc.)

When the serial bridge chip comes on-line, it steals the line console 0 data path from the RJ-45 console connection(!). I need to check if that means that some speed-shifting is also happening. That is, if my line console 0 is set for 9600 8-N-1, does it shift to 115,200 when it swings to the USB channel?

When the USB chip powers down (when you unplug the USB), the data path is supposed to drop back to the RJ-45 port. But, if you leave the USB connected, you could find yourself without access to the RJ45 console for remote administration. Fortunately, there is an idle timer to revert back to RJ45, but this wasn't implemented in hardware in the earliest versions of equipment to have the USB console. I'll summarize more of the details in the next couple weeks. I'm excited to finally get a chance to explore this, since many folks have asked me questions about it already.

This sounds like a great addition for quick desktop administration, or for jacking into equipment with your laptop in a data center. After all, fewer laptops have a built-in DE-9 connection these days, and everything has a USB port, right? With a USB console, you only need the single USB(A)-to-USBmini(B) cable. But, it hasn't (yet) become a ubiquitous connection, so we probably still also need to carry our USB-to-Serial dongle, and also our DE-9 to Cisco-RJ45 cable as well.

Check back in August to read about the results of my hacking.

My first impressions of the Redpark serial cable and Get Console

I'm reviewing them together, as I don't see an easy way to use the cable with another app (yet).
My first attempts were hampered by some Bluetooth weirdness between my iPad 1 and my ZAGGmate keyboard. A hard-reset of the iPad seemed to straighten that out. Since then, I have had good luck using the ZAGGmate with Get Console in both portrait and landscape modes. I'm a bit let down that the top speed is 'only' 57.6 Kbps, as I have a handful of console ports running at 115.2 Kbps. That said, it does perform well at those speeds which it does support.
There is no built-in method to email your log files, but you can copy them to the cut-and-paste buffer,and then paste it into the email app of your choice.
It makes the Escape key sequences visible! This is a BIG plus in my book! If you are working with conserver, this will help you spot devices which are prone to fill your logs with extra characters.
The earlier assertion that the application will only work with Cisco devices seems to be incorrect! That is, with a proper adapter, you can connect the cable to almost any device. Using a CFDTE92 adapter, I can connect to the DE9M com port on most Intel-based servers. (Check my Cisco console connection guide page for more clues, http://www.conserver.com/consoles/Cisco/ciscocons.html )
The cable is about 6' long, with a slender cable. The jacket feels a bit like Teflon, and the RJ45M connector has a molded strain relief, which will likely improve the service life. (That would be great, given the cost of the cable!) It is lightweight, and it looks like a high-quality part. The folks at Redpark have been known for the high quality of their products in the past, and I expect good things from this new product.
I have tried the combination with a variety of non-network devices as well, and I seem to be able to talk to them without needing to do any fussing with the signaling wires. I can't promise that it will work with anything, but the Get Console app running 'stand-alone' (without needing to connect to their website) seems to be a useful basic terminal emulator, if you need a wired serial connection. The cable has made a good addition to my iPad gear bag.
Testing the Get Console website service will need to wait for another post.

October Double-header! Two new useful apps!

I finally got a Redpark serial cable, and the Get Console app, and it's almost everything I wanted to make my iPad a quick tool for accessing all sorts of serial consoles!

But, I also finally found the Blogger app, which will make it easier to update my blog when I've got stuff to say. AND, I'll be able to do it from my older, pre-3g iPhone! Thank you Google!

The Blogger app also pointed out that I had yet to actually publish a couple older blog articles! That was easy to fix! Now I can get back to testing the Get Console app and the Redpark cable.

UPDATE, Spring 2012: The Get Console app has been handy on the iPad, and the Redpark cable has been reliable, so far. However, I've run into bluetooth reliability issues when using the Zagg keyboard with the iPad. When the keyboard is working, this threesome is a GREAT combination for field work.

Finally, a serial cable for my ipad!

More precisely, there is a hardware cable (cost is about $60 US) from Redpark, which plugs into the 30-pin ipod/iphone/ipad docking connector, and does all the right stuff to provide an RS-232 serial connection. But, to make practical use of it, you also need the GetConsole iPad application. And it all works without jailbreaking your device!

The application should work on the iPhone as well, but I don't have a cable yet to play with the application. I expect to get them both next month, but I have been tracking the progress getting the cable to the market for a few months!

The Redpark folks have a long history of development for Apple systems, and their legacy includes the Keyspan RS-232 serial adapters, another favorite of mine from many years ago.

The GetConsole application is part of a larger project, though, which may become a great field service tool! I'm a big fan of secure remote access to serial consoles (especially the Conserver console server management application). While the GetConsole application will let you attach your IOS device to the serial console of some other device for local interaction, the application also has the ability to be accessed via the GetConsole website. In this way, a field engineer could use his phone to connect to a serial console, but a senior engineer could then use that the phone network to reach through the phone to interact with the serial port.

I look forward to the chance to play with the cable and the application. I'll likely get the Cisco RJ45 version, since I have a large collection of adapters for them (or you can build your own with these adapter schematics). I'm curious if the application will work on my V2 iPhone (which is limited to running IOS 3.x), or if the application requires IOS 4.x. Either way, I remain excited, and I'll post an update in a month or so.

Ring out the old? Serial Consoles STILL going strong!

The last couple of weeks have been very busy, helping acquire a small company for a customer. The majority of my role included assessing, and then upgrading the network gear, as well as helping with the various server gear.

After the initial assessment, I installed Conserver on a local host, and I re-deployed a Cisco 3640 with a pair of NM-32A asynch modules. The serial consoles were VITAL to take over network gear for which the previous administrator could offer no admin credentials. Some of the network gear was also equipment with older operating software, which I had little experience with, so I needed to engage technical help from the equipment vendor. The first two questions were always "Can we do a remote desktop conference?" and "Do you have access to the serial console?" It's always nice to be able to answer "yes" to both questions. There is also some comfort in knowing that Conserver will be logging the session, so you can look back over the logs after the fact to review what happened.

During this session, I was also working with some Juniper gear, and I'm REALLY happy that Juniper also use the RJ45 wiring schema that Cisco uses. I'm a big fan of that symmetrical wiring format.

It's worth noting that SUN Microsystems newer machines (starting back with the Netra T-1) have also used this wiring schema for their TTYs0, and some Lantronix console servers have also used the wiring schema. Opengear has made this wiring schema an option for many of their products, and it is becoming the primary wiring format for some of their newest products. This makes hooking up the majority of the equipment that I work simple, AND it gives me some flexibility to choose different hardware vendors with confidence that they will drop into my existing infrastructure nicely.

Finally, I wanted to point out that serial port capability is also still being built into embedded processors! I've had some time to hack with the ATMEL processors used in the Arduino family of development boards. One port is normally integrated with a bootloader and connected to a USB interface (typically with an FTDI interface chip) to make it easy to talk to the processor. However, these 3.3-to-5 volt "TTL" interfaces can also be tied to an RS-232 interface chip when the ATMEL chip is plugged into an embedded system. (As an alternative, there is another pair of pins which can be a data-only 9600 bps (max) serial port to communicate between the ATMEL chip an another device. And the larger ATMEGA actually has 3 extra serial ports available!) My newest project is to make a small processor which can communicate via a serial interface to another system.

As we enter 2011, the venerable RS-232 serial interface lives on. I wish you all health and happiness, and hopefully a bit of prosperity as well.

         -Z-

New info about OpenGear from the USENIX LISA '10 conference

I'm just back from the USENIX LISA '10 conference (San Jose, CA). OpenGear was there (for their third year), but they were the only Console Server vendor in the hall. As a result, they got a significant amount of mindshare from the attendees. I spent a good deal of time asking questions, and I'm happy to say that they had sent some technically-deep folks to staff the booth. (They also have a lot of Conserver experience, which qualifies for extra points in my book. ;-)  I wish more vendors would have come to the LISA conference. With a great international draw, the LISA attendees have significant technical depth and come from some very large corporations. It would seem to be a fertile ground to collect contact info, and provide information to be carried back to universities and big corporations. We'll see you next year in Boston, right?

It's another example of Moore's Law... even in the last year, there are fewer Console Server vendors (Emerson acquired Avocent (who had acquired Cyclades...)), and OpenGear has made many significant improvements in their hardware and software. But, more important to me, is that OpenGear has continued to be attentive and responsive to their customers, not only for me!

What's in it for you, as a Console Server user/customer? Plenty! Customer needs have driven them to provide some really interesting new hardware this year, and they have also given customers the option to use the OpenGear RJ-45 wiring scheme, or order hardware matching the Cisco/Lantronix/Sun symmetrical RJ-45 scheme. Thanks to Jared Mallett and Todd Rychecky for sharing good info with booth visitors, here's some of what I heard;

    They have upgraded the CPU and storage in their smaller devices!
    They have a 4-port model, running on DC power...
    Some smaller units have 2 network connections (dual Ethernet, or Eth and G3 cell modem!)
    They have also made provisions for external FLASH for logging, etc.
    Security improvements include Single Sign-On, and FIPS 140-2 Crypto module
    Participating in the RSA Secured Partner Program SecurID integration

Two weeks ago, I learned of a project that wanted a 4-port device... last week, OpenGear showed me their Dual Ethernet 4-port device, shipping by default with Cisco RJ-45 wiring. Problem solved!

As we approach the holiday season, the daylight gets shorter, and (with the holidays, and folks taking extra time off) the calendar itself seems to get shorter, and deadlines start to loom large. If you need console servers, take a look at OpenGear, and see if you can find what you need quickly.

       Happy holidays!     -Zonker-

Assiging DHCP addresses to APC Power Strips

This is certainly a post related to serial consoles. But, you are limited in the amount of useful monitoring data that you can pull via the CLI, so folks often want to use SNMP or SYSLOG to monitor these devices. Usually, that's where they want CLI access.

In my case, we are re-numbering the management subnet, and we've decided to try to make this job easier for the next time a re-number is required. We're telling the devices to use DHCP addressing, and using MAC Address reservations for all of the devices that we care about. Two complications arose; How to connect to the serial CLI, and how to set the device to use DHCP.

The complication with using DHCP addressing on the AP-7868 PDU (and likely many other APC devices), is that their default configuration REQUIRES that your DHCP server set option 43 to a specific vendor code. If you don't do this, the PDU will not accept the DHCP Offer from your server.

APC AP-7868 Smart PDU Serial Pinouts

APC RJ-13 serial console, 4-wire, data-only connection
APC Console     Lantronix TS (Cisco, Opengear with Cisco wiring, etc.)
(RJ-13)         (RJ-45)
1  ---(???) nc            default settings:  9600-8-N-1
2  ---(GND)---  5
3  ---(TxD)-->  6         default username:  apc
4  <--(RxD)---  3         default password:  apc
5  ---(GND)---  
6  ---(???) nc 
( nc = No Connection )

NOTE: You must configure a default gateway for these devices (APC-7868), even if you are only talking on your own subnet!

On your Conserver host

This section includes what you need to type on the CLI console, to configure the PDU to ignore Option 43 and accept the DHCP Offer. I use the spare port on one of my console servers to do this, and I always name my unused ports in my Conserver deployments, so that I can use them at a moments notice. In this case, I'm using port 30, on Console Server 5...

NOTE: the CLI is not good about sanitizing user input, or detecting errors! You can type commas (or TEXT!) in an IP address field, and non-standard TLDs in the domain field! Be very careful about your typing, and visually check your data before accepting the changes!

console unused-cs5-30     (this is the port I picked...)
[Enter]
[Enter]     (twice, to wake-up the CLI interface)
apc [Enter]     (default login name: apc  case-sensitive!)
apc [Enter]     (default password: apc  case-sensitive!)
2 [Enter]     (Network)
1 [Enter]     (TCP/IP)
(If you need the MAC address, you'll find it here)
4 [Enter]     (Boot Mode   only if currently set for Manual...)
2 [Enter]     (DHCP Only)
2 [Enter]     (Advanced)
1 [Enter]     (Device Name)
ps-6.12i [Enter]
2 [Enter>     (Domain Name
garage.com [Enter]
8 [Enter]     (DHCP Cookie Is: ...)
1 [Enter]     (Not Required to accept offer)
9 [Enter]     (Accept the pending changes)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
4 [Enter]     (logout)

When you log out of the PDU, the device will reload the TCP stack, and in a few minutes, your PDU should accept the new IP address via DHCP.

Configuring your DHCP Server for Option 43

You need to define you option 43. How (or IF) you can do this will depend on your DHCP server software. But, this section includes the particular HEX characters that the APC PDU expects to see (or it will refuse the DHCP Offer).

These clues are for the Internet Software Consortium (ISC) DHCP server. See the dhcp-options man page - this describes all the standard DHCP options. Look for the section titled "VENDOR ENCAPSULATED OPTIONS", which is option 43. If you already have a class for APC units, then that would be a good place to define option 43 as it will be defined for all devices that match the class. The simplest way to define the value is something like this:

     option vendor-encapsulated-options 01:04:31:41:50:43;

DHCP option names and numbers are listed in RFC1533, and in the ISC DHCPd source file common/tables.c

(Explanation: I forget the first character (01), but I believe that it designates to expect HEX characters. The second says there are (4) characters in the field. The last three characters are ASCII for "APC". I don't know why the "31" is there...)

NOTE: All APC PDU's should be protected by UPS power.

In an unprotected environment, if the utility power flickers off and on quickly, the onboard network card may not power back on. In these instances, the PDU may power on but the LED display will be blank and any type of access (web, telnet, console) to the PDU will be disabled. In this case, you must power cycle the PDU again, by pulling the power cord, waiting at least 5 seconds, and re-applying power.

The on-board network card was not designed for quick off/on scenarios. Customers protecting their PDU with a UPS, as designed, will not have this problem.

There are also some APC UPS clues on my console site, http://www.conserver.com/consoles/Clues/cons-apc.html
 
Regards,

-Zonker-

APC Serial Ports, and websites, wikis, and backups, oh my!

How many of you use APC brand UPS gear? Raise your hands! Ok, you can put your hands down.

Now, how many of you have connected the UPS serial port to your computer, to get power information, or to have the UPS tell your computer to shut down? Not so many? How many tried, but had trouble? 

Question 1: Were you using the APC-supplied cable (with the correct end plugged into the UPS)?

I ask, because the APC serial ports are unusual! If you use one of the cables you have lying around your shop, there is a good chance that the UPS shut down, and would not turn on again until you disconnected the cable. 

When I first tried this, the cable was missing, and I tried various cables in the shop (because I didn't have passive signal tracers at the time). Nothing worked, and I sometimes lost power. 

When I had another UPS, with the APC cable, I found that the cable wasn't long enough, and some of the cables I tried using to extend the run caused troubles. That's when I dug deeper into the problem. Here's what I've found;

• The APC port puts data and ground on unusual pins!
  
• The UPS uses the two outputs and inputs for other purposes!

You can actually make a "Big Red Switch" for your small APC, using a couple wires and the DE-9 connection. Because of the signals being on unusual pins, when you plug in a PC with a non-APC cable, you are actually applying a "shut-down" hard-wired signal to the UPS.

I made an APC Clues web page, documenting the findings, in late 2007. But, for some reason, I guess I didn't post it. Only when I needed the information again (and couldn't find it) did the panic set in, because some of the legacy information is no longer on the web.

That brings us to....

Question 2: How many of you are making backups of your critical files?

You can replace the word "critical" with Photos, Homework, Research, Family Tree, Tax Info, you get the idea. The data that you do not want to lose in a hard disk crash, or on a lost Flash Memory Drive. Where do you keep it, and how often do you update it?

I use an automated system, backing up critical files to a disk "in the cloud", for $99 a year. This runs every time I start up my computer, and I put a lot of faith in it. But, I need to pay money to keep it alive.

I also make manual copies to a 1 GB external hard disk (but I have to remember to move new files). I should really set this up to run automatically.

For some of my hobbies, and console stuff, I put copies of the web. (Technically, I made them for display on the web, so I guess the copies on my home machine are my backups.)

For some data sets (photos for some of my hobbies or family, or my console-related files, for example) will be occasionally copied to a CD or a DVD, and distributed to friends, family, or colleagues. While it isn't complete, this may be one of my best backup methods!

My point here is that disks die, memory sticks get lost, laptops get stolen, and even servers or hosting companies die without prior notice. Data stored on websites may still be transient. If it is your data, you should make multiple, diverse copies, and make more copies over time as new media becomes popular. (And, if the data is sensitive, ENCRYPT IT!)

When I couldn't find my APC web page (I was certain that I had made one), I started checking the Wacky Wayback Machine to try to find the information. Sometimes this comes up empty, if the domain/pages you seek were not deemed interesting enough.

In closing, if you have any APC UPS gear, please check out my APC Clues page. Save a printed copy, because you may want that information again someday.  ;)

    Regards,

       -Z-

What is 'Normal', anyway?

Yow! It's already well into May, and April got by without a post. It was a busy month, with a LOT of discovery tasks happening. I'll try to get a few more posts in this month to summarize. But first, I wanted to get this thought online...

Those who know me are not surprised if the come into a dark wiring closet, and find me sitting on a milk crate, staring at the 'blinkly lights'. (Frankly, I've startled plenty of people who don't know me this way!) Most just write it of as a fixation on blinky lights, but it is not. This is how I can 'visually spot trends' of activity on devices and interfaces. It's also a chance to remember which link status indicators are on, off, in an alarm condition. And, I'm also listening... to fans, air conditioner inlets, hard drive spindle bearings, modem speakers, relays clicking.

This is how I get to know what "Normal" looks like, sounds like, and smells like in the data center and wiring closets. (Yes, smell... does it 'usually' smell damp in this room? Is the 'ozone' smell something that's always here, or does it indicate a component failure. Scent is a strong trigger for memory!)

Knowing what is 'Normal' helps us spot what is unusual. When I have a network failure, I can go to the associated wiring closet and look, listen, and smell...I don't need to ponder "has that always been like this?", because I'll remember. "That light is usually blinking...so there isn't traffic on that interface!" Knowing what is normal is a key to fast troubleshooting.

The RRD tool has been a great resource for graphing monitoring data, allowing you to visualize 'normal', to see 'now', and to identify trends. You find this under MANY open source tools, such as MRTG, Cricket, Cacti, and many more.

But, can you tell what's 'normal' with a serial console? Yes, you can! The key is, you need to LOOK when things are operating normally. Look when the system/device/network is idle some night or weekend. Look again when backups are running. Look again when the network is busy, but not failing. Then, compare your notes, or, your LOGS!

You can do a LOT interactively, using a simple terminal emulator and a cable. You have a lot more flexibility with a Console Server and multiple Telnet sessions (for example, you can monitor many consoles simultaneously, and cut-and-paste between them). The real benefits are had when you combine the console servers with a Serial Console Management Application such as Conserver, or ConsoleWorks, and you can compare historic data with today's results.

I have a handful of devices which have "diagnostics' ports, that only the field engineers will use. They are not for normal use by customers. However, when you connect to these ports, you can find some of the devices are 'beaconing' about events, or reporting regular status messages. Even if the port doesn't say much, if you hit a carriage return, you'll probably get a prompt...doing this occasionally will tell you it's still alive. And, sometimes, you can also leverage the simple diagnostics that are built in. It's good to have a baseline log of what the devices are reporting to these diagnostics when they are healthy, so you can compare them if the device starts misbehaving.

But, to do any of these things, you need to start by looking when things are working as expected, doing their job the way you want them to. That's when you want to get your first look, and save that data for later comparison. (Speaking of comparing log data, try SPLUNK! Check out splunkbase, and the SPLUNK Forums! Splunk is worth a few blog articles by itself...later.)

The bottom line: Do you KNOW what you are missing? Do you WANT to know? Then LOOK!

-Z-

Saving time, and saving soles...

Whether you wear sports shoes, cowboy boots, or birkenstocks, the life of your shoe's soles are determined partly by how much walking you do in them. I'm a big fan of recreational walking, but I'd like to minimize the running around I need to do at the shop.

In the event of an emergency, getting on a serial console saves me from pushing a cart around, waiting for elevators, carrying a laptop around (and all the extra adapters, cables and power packs), just so I can get on the console of various devices, just so I can check configurations.

Yes, if the network is working, I can check a lot of the configurations across the network. But, sometimes interfaces die. Sometimes a typographic error will change an interface or network setting. Sometimes a cable gets unplugged accidentally. That's when your normal access breaks, and leaves you scrambling. Unless you have a console management network in place, and some strategicly placed console servers. (Better still, you should have something logging all of those remote consoles, but that's a topic for another post.)

Today I watched a friend cutting over to a new network over the holidays. Progress was being made, but it was slower than planned, and the days are starting to run out. He had developers from the new network gear vendor helping to debug the lack of interaction. While they all had their laptops, I watched as they wheeled a cart around the buildings, and up and down elevators, trying to check configurations because the network wasn't yet stable.

What was missing was a stable, simple console server deployment, with 8-16 ports in each of the main and intermediate frame rooms (MDF and IDFs). There was already fiber between the rooms. The console net could have been simple, stable, and independent of the main network. And it would have allowed them to be logged into many consoles at one time. They could have been watching errors and events on many devices as they tried tuning various settings.

Sure, this costs a bit of money to set up. But the price per port is low for simple, reliable gear. Consider the time for a couple contractors, and three developers, working over the holidays, trying to debug a problem. (I guess if your doing that work as an hourly worker, it's not so bad...but if you're the person in finance trying to close out the end-of-year books, and the network is being flaky, I imagine that your perspective about how soon the network should be stable would be different.)

I've written elsewhere about my portable Emergency Kit (a small hub, console server, adapters, cables, and canned telnet configurations to the console server). Today, I'm trying to lobby for you to consider a simple configuration, to support your current devices, with a bit of room to grow. Hopefully, you may find yourself trying to add more gear in preparation for a cutover, adding extra devices in every wiring closet, and you'll save yourself time and steps if you have some extra ports ready in each location.

Remote offices/sites deserve this consideration as well. I know many places that install a modem on the core router at their remote sites. But, what happens when the router relies on a TACACS or RADIUS server at that location, and the problem is not the router? You can dial in, get a prompt, but you can't authenticate, so you can't get into the router, or the authentication server. If you had a small console server there, and the modem allowed you into the console server, you would have a better chance of getting the access you need. (Even if you only see errors, you'd understand how to resolve the problems later...maybe the authentication server is trying to log your attempt by resolving to a DNS host that it cannot reach (since the network to the main office is down)? You know you need a DNS host in that office, or at least some static entries in the local hosts table for that host.)

Trust me about this...adding console servers isn't going to make you lazy. It WILL save you time, which you can spend doing some of the other tasks on your plate. It is well worth the investment of time and money to set it up.

-Zonker-

When did my console service become Critical Infrastructure?

Many Conserver instances were started as an experiment. It was something to add to an existing console server deployment. It's usually installed on a seldom-used machine, probably older hardware, and maybe without support. But, it was certainly "enough to do the job, for now."

Once it was working, a few other folks started to use it. Then you offered to add more ports into the config for other system and network administrators. Soon, you've pulled in most of the original console server ports in the shop, and you're buying more console servers, and you're starting to look for more RAM and bigger hard drives. You're wondering if you are backing this system up, since the logs have become useful data. Then, one day, some data retention policy comes along, and you realize your console data just became Vital Records, and needs to be protected. It's time to think about an upgrade, and a service contract, and a way to write directly to archival media.

Why didn't you think about those things sooner? That's the topic for today's blog. What started as a demonstration just became critical infrastructure at one site I support. Here's what we are considering as we are looking for making this Conserver deployment a "Production Service".

Supported Hardware. Rather than a hand-me-down or a "Frankenstein machine", consider getting newer hardware. This will give you a longer product life (read that as "you can get tech support and replacement parts from the vendor" for 3-4 years), and you should consider the support contract, since this will likely become Critical Infrastructure if it hasn't already.

If you are trying to do this "on the cheap", you could try using a hand-me-down machine. Make sure you get a spare chassis (with power supply and motherboard), and and many drives and RAM! Remember, older drives and RAM get to be more expensive when they are no longer the new stuff! You'll also need to be able to service your own gear, on your own time.

Redundant Power Supplies. Unless your data center has fancy power distribution units that source two circuits to a since power cable, you should consider using a chassis that has dual power supplies. Make sure that the chassis can run fine (fully configured) on just one power supply! You should make sure that you are sourcing the power supplies from two different circuits. Also, find out if the power supplies have to be on the same PHASE of power, and find out BEFORE you plug them in. (Have I mentioned the value of a support contract for your hardware?)

I need RAM. Lots and lots of RAM. But how much is "lots". This depends on the number of consoles you plan to support, including your "someday" scenarios. Remember that Conserver starts with it's own process, and then spawns children for (generally) every 16 ports. Your OS will want some. And any other tools and scripts will need some. (If you are going to be editing large log files, searching large data sets, or processing many large log files, you're going to want a LOT of RAM, so that you avoid swapping memory to the hard drives.) Don't skimp on RAM.

Dedicated Log Data Drive(s). You don't want your logs on the same drive a your main OS. (If your console logs fill the disk over a holiday weekend, and your system can't write it's own system logs, your Monday morning is going to be a LONG morning!) How much space do you need? This is the hardest thing to estimate, since it really depends on how much you use your consoles. But, here is a good ballpark to start - estimate 20 MBytes for every console you plan to support, plus 1-2 GBytes as a buffer for when some logs start filling up faster than you expected.

Since Conserver can auto-rotate the logfiles when they reach a certain size, you want to consider how large a file you can open with the editing tools you want to use. I rotate my logs between 10-20 MB, and I use grep and PERL to find things. But if you use other tools that can't open files larger than, say, 5 MB, then you should adjust your log rotation size accordingly.

Use RAID to help protect your data. Even basic mirroring of your log drive will help protect the data set (your vital log files). If you are capturing a lot of very busy log files, you might want to consider striping the data. You may also want to consider hardware RAID support. (This can save you system some CPU overhead, but it also adds some hardware complication. Still, many sites are successfully using hardware RAID. If this is your first attempt to use hardware RAID, you should consider getting the hardware support contract for your machine.)

Consider using a RAID pair to protect your OS, scripts and tools as well! Many newer machines can host five 2.5-inch drives in a 1-Rack-Unit chassis. That gives you a pair for the OS, and a pair for log data.

Backing up your critical log files to archival media. CD media is cheap, but so is DVD media. (Heck, DVD drives are cheap, too!) You can get 5+ GBytes on a single-layer DVD, and must laptops and other workstations can read them. It's an ideal media for the day when the auditor comes and asks you to produce the log files for certain machines across a range of dates.

Remember, too, that most log data is going to be ASCII data. It's VERY compressible, and you can use PERL and cron jobs to compress the latest newly-rotated log files to a gzip version. This will let you store more log data longer, and lets you delete some of the older non-zipped versions. (Compressing the log files means you are going to be backing up the compressed logs, so you can store more of them on the DVD media...this means some savings in the number of discs you need to write over time.)


So, what would I recommend? Let me start by giving you the information that I'm using to base the decision, and then I'll tell you.

Given:
650 console ports today, could grow to 1024
(20 MB x 1,024 = 20 GB of drive, plus overhead, 25 GB minimum)
I want to store large amounts of compressed log files.
(3 yr retention needs for some files, 60 GB minimum)
1024 consoles means 65 processes, minimum, but more if my processes are really busy due to verbose logging. Could be 128+ Conserver processes later.
I want to run Splunk for log checking (RAM and drive implications.)
I also need drive space for backup, and log report manipulation.

What I'm proposing;
Dell 2950, with two 1.8 GHz Quad-core processors (Energy Smart)
Pick your OS (I'll pick Suse Linux, 3 yr license for the OS support)
8 GB of RAM (four 2 GB Dual Ranked DIMMs, Energy Smart)
Hardware RAID
76 GB, 15k-RPM drive threesome for the OS
146 GB, 10k-RPM drive threesome for the logs
Dual PS

This gives me a pair of drives for the OS, plus an on-site warm-spare, plus a pair for the logs with a warm spare. The drives are hot-swappable, so if I have a failed drive that RAID cannot recover, I'll yank the failed drive and swap in the warm spare, then let RAID rebuild it, and then I'll call Dell support for a next-day replacement, for the next three years. If I were really worried about hardware failure, I might opt for the 4-hour on-site contract.

Total cost comes in at $8.5k(US) today, from Dell, though the price may be better through other channels. (That works out to about $2,900 per year, or about $240 per month, to support 1,000 vital ports in the shop.)

Can you get by for less? Certainly. But, what is the cost to you if you lose log files you need to be retaining? What will the business impact be if the server is down for a day or more, and your administrators have to scramble to manage their servers and network gear 'the old fashioned way'? The cost isn't too bad, for Critical Infrastructure, now that it's proven itself to be useful and reliable. Shouldn't you put it on reliable hardware as well?

-Z-

Making progress, leaving footprints.

It's a bit depressing to see that I'm down to about one blog per month. At least I'm making a bit of effort hre, as well as making progress on the other projects that compete for my time.

As I've changed jobs, I appreciate the 'Oral History Project' that most companies use to store and share knowledge about network infrastructure, legacy reasons for doing things, even network architecture. But, I only like them a little, mainly because they aren't a stable source. People leave companies, or even just transfer to other departments. Stories change with time, and details tend to degrade very rapidly. I like documents better. (That's one large reason I make my web pages, and keep this blog.)

I learned a long time ago, from two examples, that the time it takes to make some basic documentation is largely paid ack when you use the documentation 3-4 times. Whether it's time saved when you refer to it yourself at a later date, or whether it's time saved when you give the document to someone else, and they can do a task, without you needing to give them the information via storytelling and sitting with them while they do the task. (I won't put the example stories in this article, to save space. Maybe I'll post them in a separate article, if there is some demand for them.)

When I'm debugging a problem, I keep notes, usually in a 'notepad' or 'teach text' file. (Save the file with a useful name, and save it often during the project!) If I'm working on a device configuration, I keep a diary of the changes I made, in case I need to revert anything. I also paste in any error messages I get during the process.

BUT, here's the message for this blog... During the process, I make notes about some of the useful options for some of the commands I'm using. (I needed to look through the syntax for this effort, and I learned something...so I save it for a future effort!) When I'm done, and I've figured out the proper sequence of commands, I now have a basic document that shows me the correct steps, and a few added notes for related commands. If I take an extra 30-60 mintes, I can add some comments for the next time... things to be aware of, pitfalls to avoid, information to gather before you start, how long it might take... these all get into the document.

At the end of the effort, I now save a copy of the file in a "Clues" directory. This is my 'canonical' place for the most current version of my clues, for anything I've worked on. I've written my stories down, and everyone on the team knows where to look for them if they need to work on a project. If I'm unavailable, my team will still have access to useful knowledge on an array of tasks. (And, if it's easy to hand off a project, it's more likely I'll be able to hand it off and do something more interesting the next time.)

Make a clues repository, start gathering simple process docs, and get the other folks in your department to share in filling the repository.

-Z-

The Emergency Kit

I've put together a small kit, which can give me quick access to the serial consoles of various machines, for those little emergency trips to the colocation facility, or a customer site. It's been so successful and handy that colleagues are borrowing it for small cluster jumpstarts, and even for a Disaster Recovery exercise. So, I'd also describe it here.

The gist is, I started with a small 8-port console server, so I could use my laptop, a pre-configured telnet client, a crossover-Ethernet cable, and a handful of adapters to connect to various devices. I found that I often needed to be connected to the machine's network as well, so I added a small hub to the kit, as well as a small power outlet strip to my laptop bag.

I picked the Cyclades TS-800 8-port console server because it was small, was BREAK-safe, and ran from a 'wall wart' (power pack). I really wanted one that could run on 12-volts DC, so I could run the kit on car battery power, but couldn't find one. My laptop can also run from 12-volts DC (which could be handy in an emergency). Sadly, the TS-800 units are no longer sold by Avocent, but there are still a variety of small console servers on the market.

The small console server fit in the large pocket of my laptop bag, but limited how many cables I could carry, along with all the adapters. I finally fell to using a small, extra bag for the console bits.

I added a small 10-Mbps hub initially, so I could also do some packet sniffing, since most small switches don't have the ability to 'span' or 'mirror' ports. But, that bottleneck also throttled network access to the attached device, and I wasn't doing so much packet sniffing, so I 'traded up' to a Allied Telesyn AT-FS708LE 8-port 10/100 Mbps switch. This is a 12-volt DC device. It's small, lightweight, sturdy, and still available cheaply.

The kit now includes the switch, the console server, two crossover-Ethernet cables (1'- and 5'-long), an assortment of lengths of regular 8-wire CAT-5 cables, and an assortment of 24 Cyclades serial adapters. I also have a few 1'-long Cyclades null-modem (DCE-to-DTE crossed RJ-45 cables, and a few CAT-5e-rated female RF45 couplers. Add in the wall warts, and the kit lives in a 6-quart Sterilite plastic container about the size of a shoebox. Easy for security folks to see what is inside, but you don't want to put this in the belly of an airplane.

The Console Server is really the key to this kit, so here's what I've done to make it work;

First, configured the device as a console server, then copied the config file so I have an on-board copy, in case someone feels the need to change settings while they borrow it.
cp /etc/portslave/pslave.conf /etc/portslave/pslave.bkup

Make sure the backup config is saved into the unit's NVRAM. On the Cyclades, this means adding the file into /etc/config_files. (This is one reason I strip most of the comments out of my console server config files, since I'm basically saving two copies of the configuration file.) Now, if someone needs to change the config file, it's quick and easy to get back on the console, and copy the config back over the pslave.conf file, then 'saveconf' and reboot.

I also add an entry for my laptop into the /etc/hosts file on the console server, to reduce delays when making your reverse-TCP connections to the console server.

I had originally assigned different port speeds (19.2k, 38.8, 57.6, and 115.2) to ports 5 through 8, to make it quick and easy to access a high-speed port. But, I found that most ports are STILL at 9.6kbps as a default, and sometimes I needed more than 4 ports. In those cases when I needed a higher speed, I often needed more than one port at the higher speed at the same time. Since I found myself making many configuration changes I decided to revert the base configuration to 9.6k all 'round, but that's just explaining my reasons. You may want to do it your own way, and that's up to you.

I also pointed all the "extra" services (Radius server, syslog, etc) that the console server might want to use to the laptop IP address. I already use a small but of RFC-1918 address space, to reduce the chances of a conflict with the network that I may attach to. Pointing just to my laptop reduces the chance of the TS tripping any Intrusion Detection System (IDS) that might be watching the network. (I don't use the Syslog much anymore, but if I want to try it, that's one less part I need to configure.)

The current laptop is a PC, using PuTTY as the client tool. This lets me save the basic settings for "ts-1" through "ts-8" (configured to connect to the IP address of the console server, at ports 7001-7008, respectively). I use a distinctive window size and text/background colors for these windows, and each connection is configured with a unique 'window name', and a unique log-file name. The distinctive window settings make it easy to tell which open windows are serial connections, versus network sessions when I'm debugging.

An advantage to using PuTTY for this, is I can also 'export' a copy the Windows Registry settings for these sessions onto my FLASH fob, and easily give the settings to someone who is going to borrow the kit, and VIOLA! They are ready to connect to the ports quickly and easily. PuTTY is small, and runs without needing to be 'installed', so I have a copy of that on the FLASH fob as well.

The final bit that makes the kit as easy to loan as it is for me to use it, is a 1-page README document (also on the FLASH fob...with a canned Conserver config snippet for this device, just in case), which outlines the default IP settings, serial speed settings, adapter and cabling clues, and basics about how to open a reverse-TCP session to a serial port. I also include clues and the password for configuring the device. (Add a few useful labels to the hub and console server, mark the wall-wart connectors with a color-code at the device-end, and things go pretty smoothly.)

I don't use it very often, but when I need it, it's ready to go! When your time is worth money, having a kit on the shelf is money well spent!

-Z-

Use the correct serial adapters, and trust them.

Today was a service window, and I was working in an old part of the network. I stumbled onto a broken DE9-toRJ45 adapter, dangling by it's wires. I swapped it out for a new one, from the proper vendor for the terminal servers in use, but there was no response via conserver.

"It was working fine last week." I'm told.

Looking further, I found a poor RJ45 termination, but the signals on on my signal tracer looked odd as well. This led me to the other end of the cable, where I found that I was looking at a USOC-Rolled cable (old CDDI-type wiring). This would have been a 'null-modem' if they had been using Cisco console servers, but they were not. Instead, they had only a couple leads correctly connected, and signal ground was not one of them. After a bit of frustration reaching more consoles, and reterminating those cables to be straight-through again, the network was back under our control. But why had they failed?

"But they've always worked like that." I'm told.

The wiring of the Cisco adapters, the null-modem (USOC-Rolled) cables, and the non-Cisco console servers wiring scheme had meant that signal reference was really be derived from the equipment chassis, across the power systems ground, between the console servers and the controlled devices. Today was they day that newer, fatter power circuits were installed, and moving the gear to different Power Distribution Units were too much.

The signal reference ground pin is extremely important, to prevent data errors, possible equipment damage, and possible injury to staffl handling the connections! (I'll share my 'bad ground, smoking gun' story another day, when I have more time.)

Using the correct adapters, and correct wiring, will lead to predictable results. Mixing and matching parts you have laying around with specialty cables will eventually cause you more headaches, and possibly more money than you will save. And the chances are good that you'll need to 'do it the right way' sooner or later. If your shop is as busy as the places I've been, you may as well do it right in the first place, because you may not have time to do it right later without sacrificing weekends or vacation time.

Crossing a VPN to get to your Console Ports

You may not have tried this yet, but you may want to read the note anyway, so you'll be prepared for the day you will need to cross a VPN. As security gets tightened, and networks are segmented, the VPNs will be coming to a LAN near you.

I hadn't thought about the issue of "dropped links" for years. My modems would often time-out from inactivity. Sometimes ISDN connections would drop due to idle timers, because there weren't that many B-channels available in the pool. I understand why you get timed out, and I empathize with the implementers. But, sometimes you need to keep a link up, without thinking too hard, or remembering to type a key.

In the modem days, I'd start a PING across a PPP link. Sometimes I'd set my email to poll every 5 minutes, just short of the timer, in order to keep links active. But broadband access and SSH to hosts was ubiquitous enough that I'd forgotten about this inconvenience.

Today, I was bitten bye a VPN with idle-timeouts set uselessly low for working remotely on my consoles. While email, and web browsers don't care so much (since they are always initiating a new TCP connection), it was invisible at first. But, my SSH sessions would suddenly be frozen. with no indication of the problem. Meanwhile, my idle TCP connection at the far host was abandoned, and would need to be killed off by someone with administrative privileges.

The answer was the TCP-keepalive options available in my SSH client. Under Microsoft graphical OSs, I use PuTTY, and you can set the option there for sending keep-alive null packets every n seconds, and/or TCP keep-alives. This solves the problem when it needs solving (only when I've got an SSH session I care about), and its automatically done (no need to remember special hacks), with minimal load on the network link.

Now, I admit that I am hogging a resource (the VPN session), but I'm usually logged in for a good reason (such as wrenching on the network), so I'm doing it for a good cause.

If you are going to SSH into console servers (or even reverse-TCP), a dropped VPN connection will cause you a lot of grief, since the port you were connected to will be 'busy' until someone can kill off that process. If you are using SSH to log into a Console Management Application Server (such as Conserver), you will be leaving abandoned sessions on the application server host, but the console ports themselves will still be available to the next person who needs to get on them. If you run into this symptom, check for the keep-alive options in your telnet/SSH clients, and see if that doesn't fix your timeouts. Just remember to log out of those sessions when you are done, so the VPN access can continue to be shared.

-Z-