20080106

Bluetooth console adapters, short time only

I thought I had mentioned these before, but I can't find them by keywords, so here we go.

If you know about the BlueConsole2 bluetooth serial adapters, and you wanted to buy any, order them NOW, since the manufacturer is closing his doors this month (JAN 2008). The ordering page on his site is already closed, but he still has stock, and is trying to make deals to move the last few hundred.

I waited 6 months, trying to get someone else to buy them, so I could try them. OK, now I'm the guy that bought them. They are a bit pricey, but the are clearly the best implementation in this niche product space.

I'm using mine with the TriDone TriConnect software on my Palm Treo 650. The free version of the software throws an extra carriage return (so I'll probably upgrade to the Pro version), but it's a great console with scrollback for watching a device start up, and doing some CLI configuration.)

There are plenty of times where I don't have a laptop handy when I might want one. This tool is an investment in my time. When I need a terminal quickly, unplanned, I'll be able to use the Treo and the BlueConsole.

If you want to save your time, then don't waste any time. Get yours today, and save yourself the prie of trying to buy one on ebay later.

-Z-

20080103

It's OK to talk back...

With the new year came some new toys, but I still have the same old Doctor's Bag, so I needed to find out what needed to stay, and what could be combined or made smaller.

One of the relatively large things were a couple of DB25 Loopback modules, from Cyclades (one included with each server, so I have a bunch). To use this useful tool, I also have a collection of DB25 adapters for a few of the console servers I often work with.

I'd tried using some simple loopback plugs...just short wires into an RJ45 plug (sometimes called an 'ice cube') like the one you also get with a new Cyclades console server...but these get lost in places, sometimes even lost in console server ports if bigger gear is mounted above and below the console server, so I gave up on them.

My next attempt was a 'pigtail'...a 4"-6" section of cable into the 'ice cube'. This was a 'flag', to help me find it sticking out of the console server, but I still had the trouble getting these out of console servers which had been mounted in tight corners.

All the while, I still needed a way to check the signal at the other end of the cable. My real concern was usually "do I have good connectivity up to the back of the machine?", and this required a loopback at the device in question, so I started packing the loopback modules.

I'd thought about taking an RJ45 socket out of a patch-panel, and looping wires around in that. It would be smaller and lighter, but how do I label them. I thought about using color, like the hood colors on the console server adapters. (As baby boomers get older, color coding lets one get by without reading glasses, at least some of the time.) I'd tried a variety of discarded sockets, with some success, but they all had flaws.

I finally stumbled on a part that I like for the purpose! Panduit's Minicom line of CAT-3 patch panel parts. You can get them in the common resistor color-code colors, they are compact, rugged, you can buy them in small quantites, and they have a flat spot you can use for labeling the jack. I'll try to get a couple pictures on Flickr, so I can link them here (and update my Doctor's Bag page). The only real modification to make is to round the sharp corners with some sandpaper, so they don't cause problems in the bag.

-Z-

20071230

Happy New Year! It's about time...

Time for a New Years Resolution! Establish an NTP server at every site I work on, and point as many devices to it as I can. (I'll do this to save everyone some time.)

Consider taking a look around your installation. How long would it take to check the uptimes of the devices, and also check their on-board Real Time Clocks. How many do you think are correctly showing the current time, within +/- 10 minutes? For your network gear, how many are showing log timestamps as the "from startup" uptime stamp, instead of a date/time notation?

Now, consider that, in the time it would take to EVALUATE your clocks and timestamp settings, that you could probably FIX THEM, not just record the results! If you plan to go and touch all the gear anyway, why not save some time, and SET the clocks, instead of checking them?

But, when was the last time anyone bothered to do this, in your installation? Maybe the clocks on your devices only get set when each device is installed. (If you ship devices between data centers, across timezones, do you make a point of changing the device clocks when you receive them into your installation?) Let's face it, these clocks are cheap, and they ALL have some drift in them. Even if the drift is minimal, the error is cumulative, and can be significant across 3 or more years. You complicate the drift when you don't use a stable starting point. (What time reference does everyone use when they set device clocks? Probably their independent wrist watch?)

In the same way that you need to spend some money to make more money, you need to spend time to make more time! NTP is a great way to set clocks, and keep them close to the current time. (Even a daily time sync will all but eliminate the effect of drift on the cheap clocks.) Most devices have a no-cost client available, and many servers and some network gear have a no-cost NTP Server option available. If you have a complicated network, with multiple customers or DMZs, you can invest in an appliance that uses GPS or the cellular phne service to provide a drop-in connection on an isolated network. You just need to take some time to set up a server, and then set your devices to be clients.

Technically, all of this work (and invested time setting up NTP) doesn't save time, per se. What it will do is reduce the time you spend tracking down problems. Calculating time-math is hard. Correlating time offsets in log files between devices with unsynchronized clocks takes a LOT of time. But, when you have an "event" in your installation, and you need to find the Root Cause, you need to compare a bunch of logs, to figure out what happend first, then second, and etc.

If you take the time to set up an NTP server, you make it quick and easy to 'set and forget' the clocks on new devices coming into your installation. When you take the time to point all of your existing devices to the NTP server, you'll save time comparing the logs on all future 'events'. It's hard to guess how much time, or how soon you will recover that time, but it WILL happen. Just give it time.

-Zonker-

Some NTP appliance resources I've found to be interesting:
VMware NTP appliance VM info
Symmetricom S200
J-Time (Meinberg USA) Lantime M300
Brandywine Communications NTV-100RG

20071228

Saving time, and saving soles...

Whether you wear sports shoes, cowboy boots, or birkenstocks, the life of your shoe's soles are determined partly by how much walking you do in them. I'm a big fan of recreational walking, but I'd like to minimize the running around I need to do at the shop.

In the event of an emergency, getting on a serial console saves me from pushing a cart around, waiting for elevators, carrying a laptop around (and all the extra adapters, cables and power packs), just so I can get on the console of various devices, just so I can check configurations.

Yes, if the network is working, I can check a lot of the configurations across the network. But, sometimes interfaces die. Sometimes a typographic error will change an interface or network setting. Sometimes a cable gets unplugged accidentally. That's when your normal access breaks, and leaves you scrambling. Unless you have a console management network in place, and some strategicly placed console servers. (Better still, you should have something logging all of those remote consoles, but that's a topic for another post.)

Today I watched a friend cutting over to a new network over the holidays. Progress was being made, but it was slower than planned, and the days are starting to run out. He had developers from the new network gear vendor helping to debug the lack of interaction. While they all had their laptops, I watched as they wheeled a cart around the buildings, and up and down elevators, trying to check configurations because the network wasn't yet stable.

What was missing was a stable, simple console server deployment, with 8-16 ports in each of the main and intermediate frame rooms (MDF and IDFs). There was already fiber between the rooms. The console net could have been simple, stable, and independent of the main network. And it would have allowed them to be logged into many consoles at one time. They could have been watching errors and events on many devices as they tried tuning various settings.

Sure, this costs a bit of money to set up. But the price per port is low for simple, reliable gear. Consider the time for a couple contractors, and three developers, working over the holidays, trying to debug a problem. (I guess if your doing that work as an hourly worker, it's not so bad...but if you're the person in finance trying to close out the end-of-year books, and the network is being flaky, I imagine that your perspective about how soon the network should be stable would be different.)

I've written elsewhere about my portable Emergency Kit (a small hub, console server, adapters, cables, and canned telnet configurations to the console server). Today, I'm trying to lobby for you to consider a simple configuration, to support your current devices, with a bit of room to grow. Hopefully, you may find yourself trying to add more gear in preparation for a cutover, adding extra devices in every wiring closet, and you'll save yourself time and steps if you have some extra ports ready in each location.

Remote offices/sites deserve this consideration as well. I know many places that install a modem on the core router at their remote sites. But, what happens when the router relies on a TACACS or RADIUS server at that location, and the problem is not the router? You can dial in, get a prompt, but you can't authenticate, so you can't get into the router, or the authentication server. If you had a small console server there, and the modem allowed you into the console server, you would have a better chance of getting the access you need. (Even if you only see errors, you'd understand how to resolve the problems later...maybe the authentication server is trying to log your attempt by resolving to a DNS host that it cannot reach (since the network to the main office is down)? You know you need a DNS host in that office, or at least some static entries in the local hosts table for that host.)

Trust me about this...adding console servers isn't going to make you lazy. It WILL save you time, which you can spend doing some of the other tasks on your plate. It is well worth the investment of time and money to set it up.

-Zonker-

20071129

All Together Now...

You use Console Servers to manage the serial ports of various devices, right? So, it only makes sense that a 'Best Practice' would be to connect the console ports of your Console Servers to a port on a console server, so you can monitor the Console Servers, too, right?

BUT, to do this 'right', you need to connect it to another, different console server. After all, when do you really need to use the console? When the machine is having a problem, and WHEN THE DEVICE REBOOTS! (All Caps used for emphasis...sorry if I offended anyone. ;-) When the device reboots, your serial console is how you see what the machine is doing until it gets going sufficiently for the network to be of use to you. So, if you connect your serial port to another port on the same Console server, you can monitor it when all is going well, but when things go badly, you've just lost your console access as well.

Tonight, I hit a bit of a snag...because I had connected some Console Server console ports to other "nearby" Console Servers. (Imagine 18" console jumper cables...why not, the 'different' units were close enough.) "What could be the problem with that", I had thought.

Tonight was the night to upgrade the software on this set of Console Servers...complete with systematic reboots between revision levels. Doing the three in parallel, to take advantage of 'cut-and-paste' to minimize the chances for typographical errors. Halfway through the reboots, I stop seeing progress...because the reboot of Console Server A had taken out the console access for Console Server B...

So, I type this note as I wait for the Console Servers to reboot and for my console access to stabilize again, and I consider whether my tie would have been worth the extra ~125 feet of cable to make the console runs to diversely-disparate Console Servers, to minimize the chance of a 'cluster service' outage taking out closely-grouped Console Servers. Food for thought.

-Z-

20071023

Whether we need DCD more than symetry

I'm very much in favor of a symmetrical RJ45 wiring schema for console servers, because it could minimize how many adapters I would need to stock. The advantages of a symmetrical pinout include using a simple 'rolled' cable to make a 'null-modem' connection between devices and adapters.

Now, I'm not picking any particular vendors wiring scheme as 'the best'. I'm espousing symmetry as the 'best way'. The problem is, this means having two Signal Ground pins, which usually means that DCD won't have a pin. (On an 8-wire RJ45 interface, you would have two ground leads, two data leads, two hardware flow control leads, and two hardware handshaking leads.)

Remember that today's Console Server has its roots in yesteryears Terminal Servers. The old TS used modems or terminals, to let remote users connect to hosts across the network. The hardware handshaking was used by the terminals to indicate if they were connected. With modems, they could be connected and powered, but it was DCD signals from the modem to the TS which told the TS whether an interface was 'on-line' with a remote user.

But, these days in console service, modems are a pretty rare thing. Still, I'll be the first to recognize that modems could be used to dial into a remote site and try to connect when the main WAN links are down. However, you can configure most modems to toggle the DSR lead to reflect the status of DCD, and you can configure most console servers to recognize the DSR lead as a signal that the device isn't ready. (So, I don't think we really need DCD in console service. I'll leave Dial-up Networking for later article.)

If you look at my Signals Page, in the RJ45 section, you'll notice that there are already a couple of symmetrical pinout schemes. But you can also see the wide variety of combinations that I've found so far. While pin 4 seems to be the most common pin for signal ground, its clear that not everyone uses pin for for the signal ground. That makes it harder to make an RJ45 signal tracer that works in all cases, so there is less incentive for a vendor to take on that project.

So, for the existing symmetrical RJ45 pinouts on my signals page, you'll notice that they use pins 4 and 5 as ground pins. Data pins are on pins 3 and 6. Handshaking is on pins 2 and 7, and flow control are on pins 1 and 8.

If you're asking "why can't we use pin 5 for DCD?", then you may not have thought about the issue deeply. The first problem comes when you use a rolled cable as a null modem, and you now connect two devices signal leads, but not their signal ground pins! You'll be replying on the chassis ground connections to provide the signal reference. Using these console servers in large data centers means that you may be connecting devices on different phases of the AC circuit, or even across two different Power Distrubtion Units (PDUs) or Uninterruptible Power Supplies (UPSs). The result could be burned out serial ports on your console server and the attached devices, or more sever damage to the units.

So, to summarize the vendors I know about today;

Symmetrical, with 2 grounds (9+ of 26):
Cisco, Logical Solutions SCS, Lantronics SCS-1600, Digi PortServer II, Xyplex 1600/1800, iTouch/MRV in-Reach lines, Juniper Networks (J-series devices, and many NetScreen devices), Server Technology CDU consoles, and Sun Microsystems console/TTY ports

Symmetrical, except for 1 ground and DCD (3 of 26):
Digi CM, Digi STS-1610, Opengear CM

Of the 27 console connections I've documented, 15 use pin 4 for ground, and 9 use pin 6 for ground. (Keep that in mind if you are making your own RJ45 signal tracers.) And of the 9 symmetrical pinouts, eight of them use pins 4 and 5 for ground, and one uses pin6 3 and 6.

It seems to me that other vendors (who are not making console servers) have been choosing the Cisco wiring scheme (Juniper, Sun, Server Technology). Lantronix picked the null-modem compliment to the Cisco pinout. Those choices have been good for me, since it simplifies some of the deployments that I work on. It's interesting to note that the Xyplex/iTouch/MRV pinouts are almost identical, except that pins 1 and 8 (the flow control leads) are inverted from the Cisco schema. (So, if you don't care about flow control, you can mix-and-match these devices with the Cisco-schema cousins.)

I'm under the impression that some vendors are considering adopting a different wiring scheme for their future product series. I'd strongly urge that they consider adopting a symmetrical schema, and I'd also strongly encourage them to consider using an existing schema, rather than trying to come up with a unique schema. And I'm always interested in discussing it in person, though an email on the topic is also welcome.

20070920

Sometimes you want it raw...

Your console connection, that is!

When you are using a console server for remote access to serial consoles, you're defaults are usually a telnet-type listerner on the console server. That means a 7-bit session, with the high-order bit normally being "0". (That's why you sometimes have a session that displays unusual characters...something trips the high-order bit to "1", and you need to reset the session/port to clear the bit.)

But, what if you NEED to pass 8-bit data? Sam Crooks recently had the need to upgrade the OS on a router, but didn't have a the ability to use the network to get the code to the unit. He could, however, use XMODEM-1K from a locally-attached laptop serial port to the console of the unit. Except that his laptop was across the network. The gotcha here was the 8th-bit was set to "1" in much of the data, and certainly in the file transfer sequence numbers. This botches up the connection, and you need to start over again. (I'll explain about the modes first, but I'll also share notes from Sam's solution towards the end of this article.)

Fortunately, most console servers today have legacy roots in "Terminal Server" service, as in dial-in modem pools, and binary file transfers. So, besides their 'default' 7-bit session mode, most have an "8-bit, escapable" mode, and some even have a full "raw 8-bit, non-escapable" mode. You just may have to dig around a bit to find out how to access them. To demonstrate the issues, we'll use Cisco access servers as the example.

With the Cisco devices, to reach across the network to a particular serial port, you normally use a telnet client, to the IP address of the access server, at the TCP port 2000 + n, where n equals the (async) line number. That is, line 21, would be at TCP 2021. This is for the normal, 7-bit session. So far, so good?

If you need to use the 8th bit regularly (specialty keys, alternate characters, etc.), you can get the "escapable" 8-bit mode by using the TCP port 4000 + n. (The same n as before.) The 'escapable' pat means that you still have a couple of the 254 possible characters that you cannot use as data, because these will be used for the client to escape from sending data, so it can send control info (like "quit this session"). Most folks could use the 7-bit mode or the 8-bit mode interchangeably.

But, if you need to send ALL 256 possible characters (think the block sequence numbers, if you think that your data wouldn't have all the characters), then you need to use a "raw 8-bit" mode. But, if you can't escape from the data session (because your escape characters are "just data" now...), how do you close the session? You need to get the device connected to the async port to toggle it's handshake line. (Think of it as a modem call ending, and the modem drops the DCD line, or it cycles the DSR line...because that where that code has it's roots.)

In the Cisco access servers, you can invoke the "raw 8-bit" mode at TCP port 6000 + n.

Here are the notes from Sam's solution, once he discovered these different modes;

The serial speed of 9,600 bps is too slow for big file transfers, so he increases the port speeds to 115,200 bps. (In his case, he uses autobaud on one side, and manually sets the speed on the other. I presume this is so the speeds will resort to 9,600 after everything's finished.)

Next, he sets the port for the data transfer;

Switch#terminal download
Switch#
Switch#copy xmodem: flash:/c3560-advipservicesk9-mz.122-37.SE1.bin
Destination filename [c3560-advipservicesk9-mz.122-37.SE1.bin]?
Begin the Xmodem or Xmodem-1K transfer now...
CCCCCCCCC
now here, we go to TermTerm's, menus....
File../Transfer->/XMODEM->Send...
Pick the file, select 1K at the bottom of the window (vs CRC or checksum)...
and off the file transfer goes....hopefully to completion, and in a reasonable time...

Finally, when the transfer completes, he needs to break the session from the far end, so he uses the command clear line n (where n is the line number of the serial port you are using for the transfer).

Cisco also has a good, older page about the various commands related to configuring access servers for modem usage, which has some useful clues as well, including transport in and out, and changing your escape characters. You'll find references to these binary modes on this page, in the section relating to 'rotary'.

You can also find useful information searching for "configuring lines for modem support". For example, you can find a brief summary related to contacting the AUX port on a Cisco DSLAM using these three modes here.

Related Cisco commands (Anyone can search the public Documentation part of the Cisco website, but Cisco support contract holders can find more information if they are logged in.)
  • reverse telnet
  • configuring modem line rotary
  • telnet break-on-ip
  • exec (look for telnet-related BREAK info)
Good luck with that!

-Z-