Four years living near the sea was enough

 Not long after my last post, we started to pack our house. We moved that next Spring to the Southwest corner of Oregon, far from family, to a place with a few friends. The pace was slower, I'd get to "work remote" most of the time. And, my best friend there died of a stroke just after Thanksgiving (before we moved up). It took a while, but we finally decided that small town life was not for us. There were a litany of small reasons, which we'd tell ourselves that we were willing to put up with for the slower pace, the 1-mile walk to the beach, and watching the local wildlife wander through our yard and neighborhood (deer and elk!), but the elephant in the room was a lack of nearby medical care. In an emergency, time is of the essence. But, in a non-emergency, you might wait MANY months to get an appointment with a "traveling" specialist, or you may need to drive 2-4.5 hours (each way) to places with more specialists would would take you on as a patient.

We spent much of 2024, looking for other places, finally landing a place just outside of Portland. Close to a big city, but also a smaller-town vibe. But, good healthcare options, plenty of shopping and restaurant options. It also meant that I'd be working in the office again, but my drive is 2 miles on city streets (only 5 miles to a data center I also service), and I also have great public transit options. I can take the MAX train to the airport, instead of leaving the car in long-term parking. I was lucky that my work team gave us the latitude we needed for all of our house hunting trips!

As we approach the first anniversary in this home, I'm grateful to be in a place where I think we can age gracefully, and we have a guest room for when friends and family want to come visit. The current Republican-driven government shutdown may block our air travel for Thanksgiving, if TSA and Air Traffic Controllers are still not getting paycheck. I'm glad the Democrats are holding the line for our healthcare, and I hope the elections today show a Blue Wave. If they don't, it's time to investigate the tabulation of those votes... too many loyalists have been put in place, and I think one-party rule equals authoritarianism. Contrary to what "voices" on the right will parrot, Portland isn't burning (that footage was during COVID times). I'm building community with my new neighbors, and paying in shelf-stable foods ahead of any General Strikes. 

Get to know your neighbors, my friends. We need to start talking about our common ground, and how we want to rebuild our democracy to move forward... because the current regime is trying to take us backwards.  Happy Thanksgiving.

Ready to "Wring Out the Old"...

 2017 was wrapped up with a weeks-long 12+ hour days effort to move a data center "live" (that is, keep as much online as possible, while shifting it all to another building!), in an effort that ended 2 days before Christmas, because the building we had been in was going to be demolished! We got it done, but that was just the start of maintaining that operation while the campus around us was demolished, and significant renovations were being made to the building that we were now in. Going to the data center included suiting up in boots, jeans, hardhat, safety vest and safety glasses. Construction continues even today, on 5 of the 6 sides of the current data center. COVID-19  certainly played a part in extending the length of the construction project, and the end is now foreseeable.

  In January of 2021, I expect to see President Joe Biden sworn in as the 46th President of the US. We may even see the first inoculations for COVID-19 start in the US. In the Spring of 2021, I expect to be relocating to SW Oregon, and selling my California home. We've all been adapting to "working from home", as companies have had to change and adapt to remote working. With that move comes the decisions of what to pack, and where will it live in the new house. Older console servers from the test lab? Manuals and CDs from past vendors (example: Cyclades), with docs and firmware for older hardware. Even bins of serial adapters and specialty cables. Some of that stuff isn't going to "make the cut". This will be much of the "Wringing Out" in the next few months...

  In the past 3 years, I've been making LED Art with Arduino boards for fun (see https://www.instructables.com/IKEA-Star-With-ATtiny-and-NeoPixels/ for an example) and environmental monitors. I also got a Prusa MK2S 3D printer, and I'm learning how to make things. I was a supporter of https://www.hackerdojo.com, but COVID-19 has closed most of our maker spaces. I helped teach embedded computing to Jr High students for a year (1-day every couple weeks, in an Introduction to Technology class). I was at every (SF) Bay Area Maker Faire until there was no more Maker Faire. 

  I was wrestling with whether to start a new Maker-oriented blog as a fork, and try to keep that up after the move. But all the good names I could think of has been taken on blogspot, many years ago. Sadly, most of them had far fewer entries... some seemed to only try a few test posts, and then they want idle. Some started back in 2001, but none updated after 2014. :-( As a result, I think I'll stick with this blog, and post what I like going forward (versus only posting console-related things). 

  I still get the occasional email, asking for something that's not on my pages at https://www.conserver.com/consoles/, but I haven't needed to set up any hardware to recreate a problem for a long while. It may be Murphy's Law that after I toss out old hardware, that's when folks will have a problem that I'd want to look at myself. But since Murphy isn't going to help me load and unload the trucks, I'll learn to live with not having the hardware.

  I'm also exploring Raspberry Pi as I explore new Digital modes in Ham Radio (DMR, New Packet, and some HF modes next year), as I decide which monitors will be useful for Pi projects. There may be some interesting serial-related posts about these in the future.

  I hope for good luck, and good health for all of us in the new year.

Happy 2017!

I didn't realize that I'd left this blog fallow for a couple years! I'm still supporting console info, but I lost my old server (used for posting my web pages, although I still have my development server). It was the domain which was lost at the same time, which was the undoing of mirrors, including the primary at Conserver.com. I've been unable to rebuild that bridge.

I have replicated my repository on a new server, but that server doesn't support server-side includes for security reasons. My pages relied on that feature for page stats and for header and footer info. And I've been to lazy/preoccupied to go manually apply static header and footer into to all the pages. (Part of me had hoped to restore is back they way it was. And then I let myself get busy with other projects, and here we are in the beginning of 2017...)

While I've continued to track new servers, and the changes in the console server industry, there hasn't been any groundshaking news which might have triggered more posts. I've also been hacking with Arduino projects for the past few years as well. And now I find myself at a point where I'm considering a pivot...

I enjoy your emails with questions about something that you haven't been able to find on my web pages or elsewhere. But I'm not going to make any ad revenue off of my web pages. I've considered writing a book, or an ebook, but I can't see much of a market for that. And I realize that I was mainly working on the web pages because I liked new puzzles (the pages answer the easy questions, and then your emails brought me new puzzles).

I'm now looking at an intersection of a couple hobbies. Ham radio (primarily APRS and weather reporting), plus Arduino, having serial communications as a common link. But I don't think I can get many articles annually about APRS and weather. (But, given the storms in the past few months, maybe weather is going to generate more articles than I expect.)

Time will tell, but perhaps this blog will start pointing more towards Arduino in the months ahead. It's a creative outlet for me, and may generate more articles. I'll still field the serial questions, and post interesting new features as I come across them.

Please try to make 2017 a better year, especially where politics and your relationships with friends and family. Last year was rough on many, and this year may be rougher on others. I hope you will join me in trying to be more compassionate, and a better listener, and willing to speak up when your voice is needed to intervene or to support, for the betterment of the many.

From whence the Command Line Interface?

I've taken typing ASCII strings into serial ports as a normal interaction for decades now. Sometimes the strings seemed like arcane incantations that made little sense to me, but seemed to do the right things (in the end) on the device that I was talking to. I've written programs and scripts that took input from a user (via a keyboard, serial port, and other inputs), and I learned to "sanitize the input", and how to parse it, and how to cough up an error if the input was something unexpected, or not in the format that I wanted.

Over the years, I've had to sift through manuals, and command-completion information screens, in order to be reminded what my options are, on this device, running that version of firmware. Mainly because it seems so familiar, overall, but the occasional command options, or the order of arguments, may have been changed.

Today, I was sifting the web, looking for information to configure the serial port settings on an Avocent CCM4850. It turns out that line had roots from Equinox, with a different command syntax from the Cyclades line (which was also absorbed into Avocent, many years ago). But It also looks like the command syntax used for Lantronix and others. And this is when I decided to write this article.

Earlier in this month, Cisco Networks announced it was going to sue Arista Networks (Cisco's announcement) over what looked like copying of their Command Line Interface structures (and a few other items). Arista posted a couple early responses (from the Legal Counsel and the CTO), and I'm curious to see where it goes, and what effect it may have on the decade-old networking company. But, part of the argument begs the question about whether you can protect your Command Line Interface syntax. Are some command lines intuitive (after decades of predecessors...), and are some "common" enough that they become open to the public domain? Would the market benefit if a vendor could lock-in a particular set of commands, while competitors had to find unique ways to configure the same settings?

It harkens me back to the days of the big re-write of the Point-to-Point (PPP) protocols in the IETF Request For Comments... there were only three RFCs then, and there was confusion among the companies trying to write compatible versions. Many companies (Microsoft, Novell, Telebit, Morningstar) were trying to work on vendor-specific adaptations, and some were trying to find a way to aggregate multiple modem links into one larger pipe. The rewrite moved us into 13 different, new RFCs, but they helped lay the groundwork for interoperability. The PPP Consortium grew monthly, as more vendors jumped aboard to make things work together. And many command strings started to look alike. A LOT alike! And, at the time, that seemed like a very good thing, since I didn't have to reference many manuals to get a link running.

So, today, I'm wondering what the downside is, if different network vendors can parse the same command line string to provision VLAN or Spanning Tree settings for a switchport. And what may happen in the Console Server world if a ruling comes out that vendors must have unique command line syntaxes. How many syntaxes would you want to support, or remember?

Happy New Year! I can hardly wait to see what's ahead.

More about Cisco's USB Console Port

This past fall, I was able to work with a Cisco 1941 router, allowing me to explore the USB console in a bit more depth. Because it was the first time I've had an opportunity to work with that interface on the Cisco gear, I came to the installation process as a new user, without any previously  loaded software for this Cisco interface. This was actually beneficial, because it exposed a few other issues that might affect you if you were to approach it today.

When I first plugged the blue USB cable into my computer, nothing happened. That is, no USB device was yet detected. It appears that the USB chip in the cable gets its power from the Cisco device. Once I had the USB cable connected to the router, that's when the USB device was detected.

On a Windows 8.1 system, the USB driver is not found, and a generic interface failed to load properly, resulting in new communications. Searching the Cisco site, you will want to search for “Cisco Windows USB console driver”. The most relevant links I found work installation guides for my hardware. However, the only software download links which were still valid were dated in 2010, and the installer file for windows only supported Windows Vista, XP, and older Windows Server 2000 software.

The good news here, is that the Vista drivers included 64-bit code, and this happily installed under Windows 8.1, properly identifying the Cisco adapter as a Cypress (chip manufacturer) device,  and loaded appropriate drivers… requiring a reboot of the system.


  Device: Cypress 
  Driver added:  CiscoSerial
  Device Installed: ciscoserial.inf
  Device installed: cypressserial.inf
  Driver Management has concluded the process to add Service CiscoSerial...


On Cisco equipment which includes a USB console, there is also an LED next to the USB console interface. NOTE: When the internal interface chip is activated, this LED will illuminate, and that is a visual indication that the local RJ-45 interface has been deactivated because the USB console is active.  If the USB chip in the cable isn't initialized, the LED won't light, and the console won't be swapped.

Caution: When swapping the console channel between the RJ-45 interface and the USB interface does not alter the console process state. That is to say, if you are logged in on the RJ-45 port and already using the enabled mode, and the USB interface is activated, the USB user is now logged in and in the enabled mode!  Conversely, if you leave a session login and enabled, and then remove the USB interface, that enabled session is still now active on the RJ-45 interface. This is a strong argument for having the exec–timeout command set on your serial console line 0.

The USB serial console has a few minor differences, depending on the Cisco hardware family and IOS version. For example, not all versions will support the usb-inactivity-timeout 30 command, but this is a useful method to try and revert to the RJ-45 console, if you have a deployed console server implementation.

Swapping between RJ-45 to USB, or USB to RJ-45, does not force a logout of the consul processed. There was no visual indication (no characters received) on the RJ-45 interface which indicated that the serial process has been diverted to the USB port, or been returned to RJ-45.

 While you can connect multiple Cisco USB interfaces via a USB Hub, each Cisco device does take up a COM port on your Windows machine, or a TTY port on your Linux or OS X machine. As a result, I cannot see this as a practical method to connect the serial console for many machines in data center service. (Using the RJ-45 serial console to a console server for remote access to dozens of machines is still a much more scalable solution.) However, I think that the USB console on a Cisco product is an elegant way to use a laptop or desktop device for setup of new equipment coming in prior to deployment. It offers a quick solution for devices which only have USB interfaces, such as tablets or netbooks, to use a single cable for the configuration and quick debugging, rather than needing to support a USB-to-DE9 interface.

I am grateful for access to the 1941 ISR to be able to do this testing, and for a few candid conversations with Cisco folks who are close to the USB serial console project.

Is USB the new Serial Console?

I'm happy to report that I finally have access to a Cisco device with a USB console (a gently-used C1900 router), so I can experiment a bit with it. I also have had an interesting briefing about how these USB console interconnects came to be at Cisco, and some caveats.

As I understand it, essentially they have embedded a USB-to-EIA-232 bridge chip into the Cisco device. When you connect a PC, and you have suitable drivers, it simply appears to your machine to be a simple COM port. (I'm presuming that a MAC will see s suitable TTY and etc.)

When the serial bridge chip comes on-line, it steals the line console 0 data path from the RJ-45 console connection(!). I need to check if that means that some speed-shifting is also happening. That is, if my line console 0 is set for 9600 8-N-1, does it shift to 115,200 when it swings to the USB channel?

When the USB chip powers down (when you unplug the USB), the data path is supposed to drop back to the RJ-45 port. But, if you leave the USB connected, you could find yourself without access to the RJ45 console for remote administration. Fortunately, there is an idle timer to revert back to RJ45, but this wasn't implemented in hardware in the earliest versions of equipment to have the USB console. I'll summarize more of the details in the next couple weeks. I'm excited to finally get a chance to explore this, since many folks have asked me questions about it already.

This sounds like a great addition for quick desktop administration, or for jacking into equipment with your laptop in a data center. After all, fewer laptops have a built-in DE-9 connection these days, and everything has a USB port, right? With a USB console, you only need the single USB(A)-to-USBmini(B) cable. But, it hasn't (yet) become a ubiquitous connection, so we probably still also need to carry our USB-to-Serial dongle, and also our DE-9 to Cisco-RJ45 cable as well.

Check back in August to read about the results of my hacking.

Dell R6xx, 7xx series servers (iDRAC and Console)

Here are two clues that will help you plan for installing the new Dell server families.

They seem to be great servers... lots of capacity for RAM, lots of CPU strength, and I love the rack mounting rail system. I also like the iDRAC service processors.

What's not to like? The back of the chassis is too thick (or the DE9 console port is recessed too far). Either way, most of the serial console adapters (DE9 to RJ45) won't fit well, and provide reliable connections, unless you grind down the cases of the adapters. And, since the adapter vendors don't offer to sell you a "trimmed" version, that job is going to cost you extra, somehow. Hey, maybe we can get the intern to do it? ;-)

I'm *really* glad that the serial consoles are still on the back of the servers, but if you are going to put them there, it seems to me that making the serial adapters fit should be part of the plan to have them on the server.

The one adapter I've found that works on the R620 series servers is from Cyclades (ADB0036). Yes, you can still get them from CDW. Fortunately, our latest batch went into racks with Cyclades TS2000. But, I did need to modify SO MANY of the Lantronix adapters that I made a custom jig for a Dremel router, to get consistent results.

Clue number two: On the R620, ttyS0 goes tot he iDRAC, and ttyS1 goes to the DE9 on the rear of the server. You will need to start an agetty for BOTH of these, if you want to be able to use both of them.

My other big projects are nearly done, and I plan to post much more often this summer. Thanks for reading.

My first impressions of the Redpark serial cable and Get Console

I'm reviewing them together, as I don't see an easy way to use the cable with another app (yet).
My first attempts were hampered by some Bluetooth weirdness between my iPad 1 and my ZAGGmate keyboard. A hard-reset of the iPad seemed to straighten that out. Since then, I have had good luck using the ZAGGmate with Get Console in both portrait and landscape modes. I'm a bit let down that the top speed is 'only' 57.6 Kbps, as I have a handful of console ports running at 115.2 Kbps. That said, it does perform well at those speeds which it does support.
There is no built-in method to email your log files, but you can copy them to the cut-and-paste buffer,and then paste it into the email app of your choice.
It makes the Escape key sequences visible! This is a BIG plus in my book! If you are working with conserver, this will help you spot devices which are prone to fill your logs with extra characters.
The earlier assertion that the application will only work with Cisco devices seems to be incorrect! That is, with a proper adapter, you can connect the cable to almost any device. Using a CFDTE92 adapter, I can connect to the DE9M com port on most Intel-based servers. (Check my Cisco console connection guide page for more clues, http://www.conserver.com/consoles/Cisco/ciscocons.html )
The cable is about 6' long, with a slender cable. The jacket feels a bit like Teflon, and the RJ45M connector has a molded strain relief, which will likely improve the service life. (That would be great, given the cost of the cable!) It is lightweight, and it looks like a high-quality part. The folks at Redpark have been known for the high quality of their products in the past, and I expect good things from this new product.
I have tried the combination with a variety of non-network devices as well, and I seem to be able to talk to them without needing to do any fussing with the signaling wires. I can't promise that it will work with anything, but the Get Console app running 'stand-alone' (without needing to connect to their website) seems to be a useful basic terminal emulator, if you need a wired serial connection. The cable has made a good addition to my iPad gear bag.
Testing the Get Console website service will need to wait for another post.

October Double-header! Two new useful apps!

I finally got a Redpark serial cable, and the Get Console app, and it's almost everything I wanted to make my iPad a quick tool for accessing all sorts of serial consoles!

But, I also finally found the Blogger app, which will make it easier to update my blog when I've got stuff to say. AND, I'll be able to do it from my older, pre-3g iPhone! Thank you Google!

The Blogger app also pointed out that I had yet to actually publish a couple older blog articles! That was easy to fix! Now I can get back to testing the Get Console app and the Redpark cable.

UPDATE, Spring 2012: The Get Console app has been handy on the iPad, and the Redpark cable has been reliable, so far. However, I've run into bluetooth reliability issues when using the Zagg keyboard with the iPad. When the keyboard is working, this threesome is a GREAT combination for field work.

Finally, a serial cable for my ipad!

More precisely, there is a hardware cable (cost is about $60 US) from Redpark, which plugs into the 30-pin ipod/iphone/ipad docking connector, and does all the right stuff to provide an RS-232 serial connection. But, to make practical use of it, you also need the GetConsole iPad application. And it all works without jailbreaking your device!

The application should work on the iPhone as well, but I don't have a cable yet to play with the application. I expect to get them both next month, but I have been tracking the progress getting the cable to the market for a few months!

The Redpark folks have a long history of development for Apple systems, and their legacy includes the Keyspan RS-232 serial adapters, another favorite of mine from many years ago.

The GetConsole application is part of a larger project, though, which may become a great field service tool! I'm a big fan of secure remote access to serial consoles (especially the Conserver console server management application). While the GetConsole application will let you attach your IOS device to the serial console of some other device for local interaction, the application also has the ability to be accessed via the GetConsole website. In this way, a field engineer could use his phone to connect to a serial console, but a senior engineer could then use that the phone network to reach through the phone to interact with the serial port.

I look forward to the chance to play with the cable and the application. I'll likely get the Cisco RJ45 version, since I have a large collection of adapters for them (or you can build your own with these adapter schematics). I'm curious if the application will work on my V2 iPhone (which is limited to running IOS 3.x), or if the application requires IOS 4.x. Either way, I remain excited, and I'll post an update in a month or so.

Ring out the old? Serial Consoles STILL going strong!

The last couple of weeks have been very busy, helping acquire a small company for a customer. The majority of my role included assessing, and then upgrading the network gear, as well as helping with the various server gear.

After the initial assessment, I installed Conserver on a local host, and I re-deployed a Cisco 3640 with a pair of NM-32A asynch modules. The serial consoles were VITAL to take over network gear for which the previous administrator could offer no admin credentials. Some of the network gear was also equipment with older operating software, which I had little experience with, so I needed to engage technical help from the equipment vendor. The first two questions were always "Can we do a remote desktop conference?" and "Do you have access to the serial console?" It's always nice to be able to answer "yes" to both questions. There is also some comfort in knowing that Conserver will be logging the session, so you can look back over the logs after the fact to review what happened.

During this session, I was also working with some Juniper gear, and I'm REALLY happy that Juniper also use the RJ45 wiring schema that Cisco uses. I'm a big fan of that symmetrical wiring format.

It's worth noting that SUN Microsystems newer machines (starting back with the Netra T-1) have also used this wiring schema for their TTYs0, and some Lantronix console servers have also used the wiring schema. Opengear has made this wiring schema an option for many of their products, and it is becoming the primary wiring format for some of their newest products. This makes hooking up the majority of the equipment that I work simple, AND it gives me some flexibility to choose different hardware vendors with confidence that they will drop into my existing infrastructure nicely.

Finally, I wanted to point out that serial port capability is also still being built into embedded processors! I've had some time to hack with the ATMEL processors used in the Arduino family of development boards. One port is normally integrated with a bootloader and connected to a USB interface (typically with an FTDI interface chip) to make it easy to talk to the processor. However, these 3.3-to-5 volt "TTL" interfaces can also be tied to an RS-232 interface chip when the ATMEL chip is plugged into an embedded system. (As an alternative, there is another pair of pins which can be a data-only 9600 bps (max) serial port to communicate between the ATMEL chip an another device. And the larger ATMEGA actually has 3 extra serial ports available!) My newest project is to make a small processor which can communicate via a serial interface to another system.

As we enter 2011, the venerable RS-232 serial interface lives on. I wish you all health and happiness, and hopefully a bit of prosperity as well.

         -Z-

New info about OpenGear from the USENIX LISA '10 conference

I'm just back from the USENIX LISA '10 conference (San Jose, CA). OpenGear was there (for their third year), but they were the only Console Server vendor in the hall. As a result, they got a significant amount of mindshare from the attendees. I spent a good deal of time asking questions, and I'm happy to say that they had sent some technically-deep folks to staff the booth. (They also have a lot of Conserver experience, which qualifies for extra points in my book. ;-)  I wish more vendors would have come to the LISA conference. With a great international draw, the LISA attendees have significant technical depth and come from some very large corporations. It would seem to be a fertile ground to collect contact info, and provide information to be carried back to universities and big corporations. We'll see you next year in Boston, right?

It's another example of Moore's Law... even in the last year, there are fewer Console Server vendors (Emerson acquired Avocent (who had acquired Cyclades...)), and OpenGear has made many significant improvements in their hardware and software. But, more important to me, is that OpenGear has continued to be attentive and responsive to their customers, not only for me!

What's in it for you, as a Console Server user/customer? Plenty! Customer needs have driven them to provide some really interesting new hardware this year, and they have also given customers the option to use the OpenGear RJ-45 wiring scheme, or order hardware matching the Cisco/Lantronix/Sun symmetrical RJ-45 scheme. Thanks to Jared Mallett and Todd Rychecky for sharing good info with booth visitors, here's some of what I heard;

    They have upgraded the CPU and storage in their smaller devices!
    They have a 4-port model, running on DC power...
    Some smaller units have 2 network connections (dual Ethernet, or Eth and G3 cell modem!)
    They have also made provisions for external FLASH for logging, etc.
    Security improvements include Single Sign-On, and FIPS 140-2 Crypto module
    Participating in the RSA Secured Partner Program SecurID integration

Two weeks ago, I learned of a project that wanted a 4-port device... last week, OpenGear showed me their Dual Ethernet 4-port device, shipping by default with Cisco RJ-45 wiring. Problem solved!

As we approach the holiday season, the daylight gets shorter, and (with the holidays, and folks taking extra time off) the calendar itself seems to get shorter, and deadlines start to loom large. If you need console servers, take a look at OpenGear, and see if you can find what you need quickly.

       Happy holidays!     -Zonker-

Assiging DHCP addresses to APC Power Strips

This is certainly a post related to serial consoles. But, you are limited in the amount of useful monitoring data that you can pull via the CLI, so folks often want to use SNMP or SYSLOG to monitor these devices. Usually, that's where they want CLI access.

In my case, we are re-numbering the management subnet, and we've decided to try to make this job easier for the next time a re-number is required. We're telling the devices to use DHCP addressing, and using MAC Address reservations for all of the devices that we care about. Two complications arose; How to connect to the serial CLI, and how to set the device to use DHCP.

The complication with using DHCP addressing on the AP-7868 PDU (and likely many other APC devices), is that their default configuration REQUIRES that your DHCP server set option 43 to a specific vendor code. If you don't do this, the PDU will not accept the DHCP Offer from your server.

APC AP-7868 Smart PDU Serial Pinouts

APC RJ-13 serial console, 4-wire, data-only connection
APC Console     Lantronix TS (Cisco, Opengear with Cisco wiring, etc.)
(RJ-13)         (RJ-45)
1  ---(???) nc            default settings:  9600-8-N-1
2  ---(GND)---  5
3  ---(TxD)-->  6         default username:  apc
4  <--(RxD)---  3         default password:  apc
5  ---(GND)---  
6  ---(???) nc 
( nc = No Connection )

NOTE: You must configure a default gateway for these devices (APC-7868), even if you are only talking on your own subnet!

On your Conserver host

This section includes what you need to type on the CLI console, to configure the PDU to ignore Option 43 and accept the DHCP Offer. I use the spare port on one of my console servers to do this, and I always name my unused ports in my Conserver deployments, so that I can use them at a moments notice. In this case, I'm using port 30, on Console Server 5...

NOTE: the CLI is not good about sanitizing user input, or detecting errors! You can type commas (or TEXT!) in an IP address field, and non-standard TLDs in the domain field! Be very careful about your typing, and visually check your data before accepting the changes!

console unused-cs5-30     (this is the port I picked...)
[Enter]
[Enter]     (twice, to wake-up the CLI interface)
apc [Enter]     (default login name: apc  case-sensitive!)
apc [Enter]     (default password: apc  case-sensitive!)
2 [Enter]     (Network)
1 [Enter]     (TCP/IP)
(If you need the MAC address, you'll find it here)
4 [Enter]     (Boot Mode   only if currently set for Manual...)
2 [Enter]     (DHCP Only)
2 [Enter]     (Advanced)
1 [Enter]     (Device Name)
ps-6.12i [Enter]
2 [Enter>     (Domain Name
garage.com [Enter]
8 [Enter]     (DHCP Cookie Is: ...)
1 [Enter]     (Not Required to accept offer)
9 [Enter]     (Accept the pending changes)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
[Esc]      (go to the previous menu)
4 [Enter]     (logout)

When you log out of the PDU, the device will reload the TCP stack, and in a few minutes, your PDU should accept the new IP address via DHCP.

Configuring your DHCP Server for Option 43

You need to define you option 43. How (or IF) you can do this will depend on your DHCP server software. But, this section includes the particular HEX characters that the APC PDU expects to see (or it will refuse the DHCP Offer).

These clues are for the Internet Software Consortium (ISC) DHCP server. See the dhcp-options man page - this describes all the standard DHCP options. Look for the section titled "VENDOR ENCAPSULATED OPTIONS", which is option 43. If you already have a class for APC units, then that would be a good place to define option 43 as it will be defined for all devices that match the class. The simplest way to define the value is something like this:

     option vendor-encapsulated-options 01:04:31:41:50:43;

DHCP option names and numbers are listed in RFC1533, and in the ISC DHCPd source file common/tables.c

(Explanation: I forget the first character (01), but I believe that it designates to expect HEX characters. The second says there are (4) characters in the field. The last three characters are ASCII for "APC". I don't know why the "31" is there...)

NOTE: All APC PDU's should be protected by UPS power.

In an unprotected environment, if the utility power flickers off and on quickly, the onboard network card may not power back on. In these instances, the PDU may power on but the LED display will be blank and any type of access (web, telnet, console) to the PDU will be disabled. In this case, you must power cycle the PDU again, by pulling the power cord, waiting at least 5 seconds, and re-applying power.

The on-board network card was not designed for quick off/on scenarios. Customers protecting their PDU with a UPS, as designed, will not have this problem.

There are also some APC UPS clues on my console site, http://www.conserver.com/consoles/Clues/cons-apc.html
 
Regards,

-Zonker-

How did the year get by me?

Wow, I'm surprised that a year has passed since my last post. Largely, it has been due to my inability to get the editing tools to play well with the iPhone and now the iPad. Normally, I don't update my stuff on employer time, or employer equipment. Couple this with a VERY busy year, and me putting updates lower on the priority list, and here we are on Memorial Day weekend.

A lot has happened in the past 12 months. Too much for a single post, but let me sum up.

I'm working with a variety of Service Processors this year, making the work, and documenting what I've learned. Besides working at the BIOS level, I'm trying to determine if/how the settings can be modified from the OS's involved, and that is more difficult than I supposed. The main problem seems like the websites holding the IPMI drivers or libraries have disappeared after 3-4 years. (The biggest lesson here is to capture, collect, and archive these files for any hardware you acquire. Mergers and acquisitions contribute to the loss of the files, even as the instructions make it into a knowledge base.)

An old lesson re-learned, is that older Sun gear will ignore their serial console settings as soon as you plug a keyboard in! Your only clue, if you are logging the console output with Conserver or some other application, is a small line about detecting a keyboard. Removing the keyboard does NOT revert back to serial console output, you need to reboot the server with no keyboard attached. I knew this long ago, but then was thwarted when folks in a remote office attached a keyboard, and we didn't notice that console I/O had gone away.

I've also been doing a LOT of work with SNMP in the past 6 months, and I found an oddity with an MGE (which sold the line to APC, who then sold the line to Schneider Electric) room-sized UPS. The brains of the UPS uses a serial line to pass a data stream to the Network Interface shelf. If this streaming interface is removed, the SNMP cards do not get an update... But the cards simply report the LAST value for any queried OID. As a result, your only clue of a disconnect is seeing all results "flat-line". BUT, if you are also set up forSNMP traps, you will get also see a string that communications was lost with the UPS. This is a case where you really need to be able to parse Syslog and SNMP traps for interesting strings and send an alert.

Other UPS problems led to the integration of a UPS of Last Resort, so that core devices could track and log events with the big UPS. This included watching the serial output of the APC Smart-UPS 2200XL, plus SNMP integration into our monitoring system.

Another Big Project in the past year was due to a planned network outage. This was due to a planned partial power outage in the building. Unfortunately, some key devices also lost power, causing trouble, which could not send alerts via email because the network was down. We decided that we needed to make sure there was a second path out of the data center for critical alerts and messages. This led to integration with a FoxBox, a Network-connected cell-phone device, allowing SMS alerting AND control!

I've also added many more hosts to the host-to adapter database, but I only have a basic Cyclades console connection guide posted, and I still need to make a Digi page. (Cyclades was another fine example of collecting information from the web as soon as you hear about an acquisition!)

I'll make stronger efforts to post more frequently through the rest of 2010. thanks for reading.

-Zonker-

Look at all the pretty lights

This week, I needed to reconfigure a previously-used Cyclades ACS console server. Finding the right adapters was easy enough, but getting the USB serial dongle to play was the tricky bit. Without a signal tracer, I was stuck with the evidence... "it's not working"...and I was trying different hardware combinations, and a few software changes. I wasted 2.5 hours going through different combinations, and I put a reminder in my calendar to bring my signal tracers and my Serial Doctors Bag the next day.

Once I had my trusty signal tracers, it was a matter of 10 minutes to find the proper USB settings, COM port configuration in my terminal emulator, and start the reconfiguration process. Blinky lights will save you time, every time. Good quality signal tracers are worth a high cost, because they will save you time often, and serve you well for many years. One day, blinky lights will probably save my life...

So, having solved Problem #1, and with kit in hand, I tackled Problem #2;

Years ago, a couple of hundred APC smart power strips (OK, "0-RU, metered Power Distribution Units") were installed in a data center. Fast forward to last week, when one unit is failing to respond to SNMP after a high-temperature 'event' in the room. Since it was in critical service, we wanted to try using the serial console to see if the brains were still alive. Unfortunately, nobody could find a serial cable for the job, and APC doesn't want folks making their own cables so APC doesn't post any signaling clues. I did find one website with clues about the wiring for the 940-0144 cable.

The clues indicated that the 6-pin RJ-14 interface was, indeed, RS-232 (and not TTL, for example), I could do some experimenting. A new toy in my Doctors Bag (not in the pictures (yet), is a MODAPT from Siemon. With this, and my trusty Digital Volt Meter (DVM), I was able to confirm some signalling that wasn't mentioned in the LuxNET web page.

Pin1 -2v (in reference to Pin2)
Pin2 GND (0v potential to Pin5)
Pin3 TxD output (-5.5v in reference to Pin2)
Pin4 RxD input (0.2V in reference to Pin2)
Pin5 GND (0v potential to Pin2)
Pin6 -2v (in reference to Pin2)

With this information, I could then make an adapter cable, and then use the RJ45 signal tracer to confirm that the signals are on the right pins. (I found that I could use just 3 wires, Ground - RxD - TxD) for communications to the APC AP7868 rPDU. I did not need to loop RJ-14 pins 1 and 6, and I didn't need to connect RJ-14 pins 1 or 6 to ground to establish communication.)

I'll update my APC console page and my Signals page soon with some of this information.

APC Serial Ports, and websites, wikis, and backups, oh my!

How many of you use APC brand UPS gear? Raise your hands! Ok, you can put your hands down.

Now, how many of you have connected the UPS serial port to your computer, to get power information, or to have the UPS tell your computer to shut down? Not so many? How many tried, but had trouble? 

Question 1: Were you using the APC-supplied cable (with the correct end plugged into the UPS)?

I ask, because the APC serial ports are unusual! If you use one of the cables you have lying around your shop, there is a good chance that the UPS shut down, and would not turn on again until you disconnected the cable. 

When I first tried this, the cable was missing, and I tried various cables in the shop (because I didn't have passive signal tracers at the time). Nothing worked, and I sometimes lost power. 

When I had another UPS, with the APC cable, I found that the cable wasn't long enough, and some of the cables I tried using to extend the run caused troubles. That's when I dug deeper into the problem. Here's what I've found;

• The APC port puts data and ground on unusual pins!
  
• The UPS uses the two outputs and inputs for other purposes!

You can actually make a "Big Red Switch" for your small APC, using a couple wires and the DE-9 connection. Because of the signals being on unusual pins, when you plug in a PC with a non-APC cable, you are actually applying a "shut-down" hard-wired signal to the UPS.

I made an APC Clues web page, documenting the findings, in late 2007. But, for some reason, I guess I didn't post it. Only when I needed the information again (and couldn't find it) did the panic set in, because some of the legacy information is no longer on the web.

That brings us to....

Question 2: How many of you are making backups of your critical files?

You can replace the word "critical" with Photos, Homework, Research, Family Tree, Tax Info, you get the idea. The data that you do not want to lose in a hard disk crash, or on a lost Flash Memory Drive. Where do you keep it, and how often do you update it?

I use an automated system, backing up critical files to a disk "in the cloud", for $99 a year. This runs every time I start up my computer, and I put a lot of faith in it. But, I need to pay money to keep it alive.

I also make manual copies to a 1 GB external hard disk (but I have to remember to move new files). I should really set this up to run automatically.

For some of my hobbies, and console stuff, I put copies of the web. (Technically, I made them for display on the web, so I guess the copies on my home machine are my backups.)

For some data sets (photos for some of my hobbies or family, or my console-related files, for example) will be occasionally copied to a CD or a DVD, and distributed to friends, family, or colleagues. While it isn't complete, this may be one of my best backup methods!

My point here is that disks die, memory sticks get lost, laptops get stolen, and even servers or hosting companies die without prior notice. Data stored on websites may still be transient. If it is your data, you should make multiple, diverse copies, and make more copies over time as new media becomes popular. (And, if the data is sensitive, ENCRYPT IT!)

When I couldn't find my APC web page (I was certain that I had made one), I started checking the Wacky Wayback Machine to try to find the information. Sometimes this comes up empty, if the domain/pages you seek were not deemed interesting enough.

In closing, if you have any APC UPS gear, please check out my APC Clues page. Save a printed copy, because you may want that information again someday.  ;)

    Regards,

       -Z-

How about Ubiquitous USB Serial Consoles

Happy New Year! Sorry I've missed posting in a couple months.

It's odd to me, when sysadmins tell me that "Serial Consoles are Dead. USB killed them." But, when I ask them what they mean, I get different answers.

The first meaning is: "RS-232 is dead, long live USB!" So, then I ask them what they do when they need to get to the serial console, and they tell me about needing to find a USB-to-DE9 serial adapter, get that to work, and point the serial console getty to that device. ("I bet that's tough to do when you are having trouble with a server" I offer. They usually agree that it is...)

The second meaning is: "You don't need DE-9 connectors anymore, you can do it all with USB". This has more insidious meanings...

2a) Most new servers don't have a DE-9 connector on the chassis, just USB ports.

2b) If I need a serial port, I can add a USB-to-DE9 dongle, and use minicom.

Have you ever tried plugging in a USB-connected external modem, and then tried to dial into it, to access the serial console (getty) on your server? The adventure is fraught with problems, which must be solved with a Crash Cart or other form of KVM or remote desktop (which means you need to think about it BEFORE you have a problem, so that it's a useful avenue when trouble presents itself).

Remember, the serial console has been your friend, when the network or the network-facing configuration goes awry. Do you think USB is ready to replace that role? Can you go to Fry's, and buy a USB-to-USB cable which you can simply plug in between two servers, and automatically establish a "null-modem"-type data path? I haven't seen one. Why Not?

Someone needs to develop a little, microprocessor pod, with two USB cables. Power the unit from the 5-volt signal from either USB device. Maybe even add some TX and RX data status LEDs, to show which USB cable is talking. You don't even need to do the conversion from TTL to RS-232 and back...just keep it all at TTL levels! But the microprocessor needs to be able to talk to each USB port when the computer inquires "Who are you, what are you?"...this is the key!

USB relies on drivers, residing on your computer(s), to recognize the ID sent from a USB-attached device when you first plug it in. To make things easy, many basic drivers are bundled into Operating System packages, which lets you plug in many devices without needing to install the drivers first. (This is the difference between Play and Pray in "Plug-n-Play" architecture.)

Looking at the basic USB-to-DE9 dongles, most are derivative works from two basic hardware types. For example, since Brands A, G, M and Q all use "Chip X", and Brands D, I and S all use "Chip Z", it's easy to see that just having two basic (generic) drivers will let you plug in most of the devices from these 7 brands of dongles. Getting Microsoft, and the various Linux distributions is an important step in making things easier for the folks who use these operating systems with serial devices.

But, if you make this USB cable I'm describing, and you try to use either of these USB-to-Serial chips as a base, you will find that most systems will think that you are attaching a plain serial device, and they'll use the generic driver. So, you'll need a bit more engineering...you need a driver that will determine if it needs to find (or create) a getty process, and connect it to the I/O address for the cable. Since you are going to need to make your own driver, you just gave yourself the flexibility to define how this type of cable communicates with the computers.

In a basic "Crash Cart" scenario, I want to plug the "Server" cable into my server chassis. If there is a console getty process running, I want it to be directed to this new device (even if it thinks it was connected to some other previous USB-to-Serial dongle, since that may have failed - causing me to need the crash cart).

When I plug in the "Client" cable to my crash cart, or a laptop, I want them to think that it's just another serial port. (For real Ease Of Use, let's default the serial speed to 9600 8N1.) If it's a Windows box, I'd like the option to choose (in the driver configuration) which COM port this device is assigned to (so it's going to ALWAYS be there...even if it needs to steal the assignment from a previous device). I'd also like to be able to associate this interface with an application, so that the OS will try to launch that application when the cable is plugged in.

I want to walk up to a server, just out of the box, with my laptop, and plug this cable into the server and into the laptop...and have the laptop launch my copy of ProCom, or TerraTerm, or HyperTerm...and have the server configure it's getty to talk to the new port, if the OS is healthy.

If we can loop in a few of the big BIOS makers, the BIOS might notice this device on startup, if BIOS redirection has been turned on, and use the cable (as a preference above the usual COM A assignment).

Of course, until we can get to this point, USB hasn't replaced the venerable RS-232 serial port, at least in my data center.

Good luck with this winter!

-Z-

Why aren't service processors "easy to use"?

You could try to tell me that it's because serial consoles are sensitive access points to vital hardware, and that Security is inversely proportional to Ease Of Use, therefore Service Processors cannot be Easy, because they need to be Secure. And, doing my best Jeremy Clarkson (Top Gear) impression, I'd tell you that "you'd be wrong".

I'll tell you that it's because most the consumers are not asking for ease of use, or for interoperability. Because it seems (to me) to be on the minds (and mouths, and blogs) of a relatively small subset of vocal customers, and the 'collective voice' of this subset is further diminished because we are asking different vendors. (That is, I ask Dell, you ask HP, she asks Sun...but darn few of us are asking more than one or two vendors.)

Security features are only just now beginning to be built-in, either already enabled, or very easy to configure, largely because the US government has tied such requirements to their purchasing rules. If you want to sell the US government a desktop computer, there is now a whole checklist of security items to tick on the list before your kit might be considered. The US government, across many departments, now speaks with a large, unified voice (through the purchasing rules and policies)

Similarly, in the early days of the Point-to-Point Protocol (the follow-on to S.L.I.P. for IP-over-modem communications), PPP would allow more computer network protocols to connect and exchange data. As a customer, you'd have thought that vendor A's PPP (the choice inYOUR shop) would talk to Vendor B's gear at some Strategic Partner's network. But you'd have been wrong. Coupled with the birth of the Commercial Internet (it wasn't just for DARPA anymore), many companies were trying to get onto the Internet, or to connect to other companies to exchange information.

There was a large dissatisfacton with a lack up interoperability. And many companies spoke with the vendors to complain. BUT, besides complaining to the vendor who supplied the kit to YOU, you also would have complained to the vendors of the other guys gear as well. You might complain to 5 or 10 vendors, between routers and modems.

As a result of the dissatisfaction of LOTS of users, 6 companies started the PPP Consortium, with a week-long meeting at Telebit Corp in Sunnyvale, CA, with the express purpose of trying to test all of their gear with each other, and test across the matrix of then-new (and still evolving) PPP protocol options. At then end of it, the scorecard wasn't too bad, and all of the participants had a list of things to fix. They also hashed out some of the ambiguities in the draft Request For Comments (RFCs) describing the options. (Many of the vendors were part of the IEEE, and were helping define the RFCs, but the meeting helped settle misunderstandings about the options.)

Because the group of six could now claim a large amount of interoperability with the others in the group, it helped their marketing. Because the customers saw progress, they harped on the vendors who were NOT part of the original group, and they wanted to get in on the consortium. Six months after the first group, I believe there were 13 members, and 18 at the next. By the third meeting, PPP was Stable. PPP 'just worked', most of the time. PPP had become "easy", and SLIP was largely relagated to antique links on old equipment.

When I look at the various implementations of BIOS redirection to the serial port, they almost exclusively presume (or require) a modem connection, which then requires hardware level wiring issues if you are using a Console Server connection. The mapping of Function Keys, command sequences, and the timing issues to invoke the commands make using them tricky on a Console Server. It makes them impractical on any link slower than 19.2 Kbps. Yet the few BIOS makers don't see a need to try to do better, let alone to work with the other vendors to provide some consistancy of user interface to their customers. Do they know whether you care?

Now that Service Processors are becoming prevalent on new servers, it's hard to get a server without one. Yet they suffer from problems very similar to BIOS. The user interfaces are different from vendor to vendor. But what about processors from the same vendor. What about within the same vendor, across a hardware platform? You'd think that a vendor might want to keep their command sequences consistent, for the sake of their customers (who might be scripting support sequences for their machines). You'd probably like to think that...but you'd be wrong. A couple vendors seem to change at will, but I'm guessing that they are just buying server designs from other companies...and those same vendors probably haven't taken the time to design that Human User Interface, they probably simply rely on the hardware maker to fulfill the tasks (power cycle the server, set the IP address of the Ethernet interface for service processor, implement a web interface with certain features), leaving the details of keystrokes to the hardware maker...and when the vendor chooses another hardware maker, they get the same latitude with the design, and make different choices. (I wonder what Donald Norman thinks about tis problem.) Do your vendors know if you care?

When you tell a sales person that you want some new feature on their next year's model, the usual response is "OK, if we can get that done, how many will you buy?". It's going to cost them labor hours, engineering and QA time, technical documentation, maybe changing part of a training class. While you may only want 6, or 12, or 40, your voice will be tallied. When they from enough of you, it's going to be worth it to make a change. But we need to tip that scale by asking, and probably tying that request to some money. (My dad would sometimes ask me to "write your request on a $5 bill, and I'll consider it", but I have seen this over and over again, from different seats in different companies. For the vendor, it's all about money.)

My point here, is that you are going to spend money on new servers at some point. Make your money count! Tie the money to your requests for easier-to-use service processors. Decide what you want, and tell your possible vendors about your requirements. They all want your money...so vote with your wallet! Reward the responsive vendors! If the unresponsive vendors don't get the order, they'll at least know why, and they'll learn that you were serious. As more of us tie our requirements to purchasing, we can make a difference.

As you get into your busy end-of-year, some of you are already going through your budget cycle for 2009. Use any slowdown you get to think about requirements to go along with those purchases. Get them on paper, and in email, so they are ready to send to your vendors with your Request for Quote. And good luck in the new year.

-Z-

A little perl of wisdom

When it comes to finding some quick help (after the Helpdesk has closed), or those tidbits of knowledge and experience of someone who has tread the path before me, Google has been my friend for many years (taking up the mantle after AltaVista!).

But, there is a downside to learning all about a topic from Google. It is that you, the reader, are guiding the learning. You are getting the answers to the questions you ask...but it doesn't always offer things that you should know (or at least consider), based on your searches.

I taught myself perl, reading books, and using Google for answers, and examples...but I sometimes found myself confused when reading someone else's code. This year, I'm taking formal classes, from a few sources, hoping to patch a few holes in my foundation of knowledge.

While I'm glad to have taken this approach, to have someone guiding me through the learning, systematically, and answering my questions...I'm finding that the Teacher in this equation is even more important than the content. Even if someone knows the content well, that doesn't mean that they will be a good teacher. (Answering the occasional question does not equal teaching.)

After 5 weeks in a semester-long class at a local community college, I too a full-day Perl 101 course from Bayview Training...and then I coded for 15 hours straight, because that class had made a number of concepts more concrete for me (several "light-bulb" moments), and I was excited about the possibilities for my future projects.

I heartily recommend Bill Ward as an instructor. While he teaches primarily in Silicon Valley, he does travel for some classes and consulting as well. The cost of the course was money well spent! If you really need to learn some perl, check out Bayview Training and Bill Ward!

Big News! SSH clients for iPhone!

I haven't had any 'big news' on the console front for a couple months now. I have been missing my BlueConsole adapters (which I've blogged about before), and SSH access from my iPhone, BUT that has changed!

New in the Apple Apps Store are four SSH applications (use the search dialogue, look for "ssh"...). I've already bought two to try, and one has already met my most common needs. In a nutshell, here are the four I found, with URLs to the maker's websites, in case you want to do your own homework on the topic.

TouchTerm http://www.jbrink.net/touchterm/index.html $2.99 This one looks pretty good, from the online docs. There is only basic information on the maker's home page, BUT, check the SUPPORT page! Key Exchange, font colors, different input modes (immediate, batch, command history). The developer offers an email address for comments and suggestions. This looks like a good pick!

pTerm http://www.instantcocoa.com/products/pTerm/ $4.99 This maker also lists an email address, as well as a Google discussion group. The web page posts a short-term development schedule. They also boast that this app is based on PuTTY, and I'm a big PuTTY fan (but I understand that this app is only scratching the surface on features). I've sent email to the developer with my wants, and I already received a positive reply. He is also willing to work on the bluetooth-serial-devices interfacing as well.

iSSH http://www.zinger-soft.com/iSSH_features.html $4.99 This one also looks good. The main web page is odd... there are small, light-grey arrows under a few of the descriptions...they indicate that there are additional items at this menu level of the web page. Clicking on them will show you the other topics. This one also supports shared keys, but doesn't seem as flexible (to me, yet), as TouchTerm.

SSH http://www.throughput.biz/ $3.99 This looks simple, and it word-wraps the screen. There is little information on the maker's homepage. I'm currently not inclined to try it.

So, with this in mind, I have bought the first two (TouchTerm and pTerm), and I'm spending my efforts to put TouchTerm through it's paces first, because pTerm doesn't have RSA/DSA keys yet (though keys are on the short-range list for new features, so I'll be giving them both a good workout in September). I was able to set up a basic profile, with screen and font colors, RSA-2048 key, emailed the key to another host, so I could place the key, and use the session. It will take a bit of getting used to, since I'll need to leave the fonts a bit larger, and pan around the screen (using mutt was kinda awkward, but that won't be my primary use of SSH). I like it so far, and I hope to be able to put my BlueConsole adapters back into service sometime in the near future.

I hope you had a chance to enjoy some vacation this summer as well.

-Z-