The last couple of weeks have been very busy, helping acquire a small company for a customer. The majority of my role included assessing, and then upgrading the network gear, as well as helping with the various server gear.
After the initial assessment, I installed Conserver on a local host, and I re-deployed a Cisco 3640 with a pair of NM-32A asynch modules. The serial consoles were VITAL to take over network gear for which the previous administrator could offer no admin credentials. Some of the network gear was also equipment with older operating software, which I had little experience with, so I needed to engage technical help from the equipment vendor. The first two questions were always "Can we do a remote desktop conference?" and "Do you have access to the serial console?" It's always nice to be able to answer "yes" to both questions. There is also some comfort in knowing that Conserver will be logging the session, so you can look back over the logs after the fact to review what happened.
During this session, I was also working with some Juniper gear, and I'm REALLY happy that Juniper also use the RJ45 wiring schema that Cisco uses. I'm a big fan of that symmetrical wiring format.
It's worth noting that SUN Microsystems newer machines (starting back with the Netra T-1) have also used this wiring schema for their TTYs0, and some Lantronix console servers have also used the wiring schema. Opengear has made this wiring schema an option for many of their products, and it is becoming the primary wiring format for some of their newest products. This makes hooking up the majority of the equipment that I work simple, AND it gives me some flexibility to choose different hardware vendors with confidence that they will drop into my existing infrastructure nicely.
Finally, I wanted to point out that serial port capability is also still being built into embedded processors! I've had some time to hack with the ATMEL processors used in the Arduino family of development boards. One port is normally integrated with a bootloader and connected to a USB interface (typically with an FTDI interface chip) to make it easy to talk to the processor. However, these 3.3-to-5 volt "TTL" interfaces can also be tied to an RS-232 interface chip when the ATMEL chip is plugged into an embedded system. (As an alternative, there is another pair of pins which can be a data-only 9600 bps (max) serial port to communicate between the ATMEL chip an another device. And the larger ATMEGA actually has 3 extra serial ports available!) My newest project is to make a small processor which can communicate via a serial interface to another system.
As we enter 2011, the venerable RS-232 serial interface lives on. I wish you all health and happiness, and hopefully a bit of prosperity as well.
-Z-
20101113
New info about OpenGear from the USENIX LISA '10 conference
I'm just back from the USENIX LISA '10 conference (San Jose, CA). OpenGear was there (for their third year), but they were the only Console Server vendor in the hall. As a result, they got a significant amount of mindshare from the attendees. I spent a good deal of time asking questions, and I'm happy to say that they had sent some technically-deep folks to staff the booth. (They also have a lot of Conserver experience, which qualifies for extra points in my book. ;-) I wish more vendors would have come to the LISA conference. With a great international draw, the LISA attendees have significant technical depth and come from some very large corporations. It would seem to be a fertile ground to collect contact info, and provide information to be carried back to universities and big corporations. We'll see you next year in Boston, right?
It's another example of Moore's Law... even in the last year, there are fewer Console Server vendors (Emerson acquired Avocent (who had acquired Cyclades...)), and OpenGear has made many significant improvements in their hardware and software. But, more important to me, is that OpenGear has continued to be attentive and responsive to their customers, not only for me!
What's in it for you, as a Console Server user/customer? Plenty! Customer needs have driven them to provide some really interesting new hardware this year, and they have also given customers the option to use the OpenGear RJ-45 wiring scheme, or order hardware matching the Cisco/Lantronix/Sun symmetrical RJ-45 scheme. Thanks to Jared Mallett and Todd Rychecky for sharing good info with booth visitors, here's some of what I heard;
They have upgraded the CPU and storage in their smaller devices!
They have a 4-port model, running on DC power...
Some smaller units have 2 network connections (dual Ethernet, or Eth and G3 cell modem!)
They have also made provisions for external FLASH for logging, etc.
Security improvements include Single Sign-On, and FIPS 140-2 Crypto module
Participating in the RSA Secured Partner Program SecurID integration
Two weeks ago, I learned of a project that wanted a 4-port device... last week, OpenGear showed me their Dual Ethernet 4-port device, shipping by default with Cisco RJ-45 wiring. Problem solved!
As we approach the holiday season, the daylight gets shorter, and (with the holidays, and folks taking extra time off) the calendar itself seems to get shorter, and deadlines start to loom large. If you need console servers, take a look at OpenGear, and see if you can find what you need quickly.
Happy holidays! -Zonker-
It's another example of Moore's Law... even in the last year, there are fewer Console Server vendors (Emerson acquired Avocent (who had acquired Cyclades...)), and OpenGear has made many significant improvements in their hardware and software. But, more important to me, is that OpenGear has continued to be attentive and responsive to their customers, not only for me!
What's in it for you, as a Console Server user/customer? Plenty! Customer needs have driven them to provide some really interesting new hardware this year, and they have also given customers the option to use the OpenGear RJ-45 wiring scheme, or order hardware matching the Cisco/Lantronix/Sun symmetrical RJ-45 scheme. Thanks to Jared Mallett and Todd Rychecky for sharing good info with booth visitors, here's some of what I heard;
They have upgraded the CPU and storage in their smaller devices!
They have a 4-port model, running on DC power...
Some smaller units have 2 network connections (dual Ethernet, or Eth and G3 cell modem!)
They have also made provisions for external FLASH for logging, etc.
Security improvements include Single Sign-On, and FIPS 140-2 Crypto module
Participating in the RSA Secured Partner Program SecurID integration
Two weeks ago, I learned of a project that wanted a 4-port device... last week, OpenGear showed me their Dual Ethernet 4-port device, shipping by default with Cisco RJ-45 wiring. Problem solved!
As we approach the holiday season, the daylight gets shorter, and (with the holidays, and folks taking extra time off) the calendar itself seems to get shorter, and deadlines start to loom large. If you need console servers, take a look at OpenGear, and see if you can find what you need quickly.
Happy holidays! -Zonker-
20100711
Assiging DHCP addresses to APC Power Strips
This is certainly a post related to serial consoles. But, you are limited in the amount of useful monitoring data that you can pull via the CLI, so folks often want to use SNMP or SYSLOG to monitor these devices. Usually, that's where they want CLI access.
In my case, we are re-numbering the management subnet, and we've decided to try to make this job easier for the next time a re-number is required. We're telling the devices to use DHCP addressing, and using MAC Address reservations for all of the devices that we care about. Two complications arose; How to connect to the serial CLI, and how to set the device to use DHCP.
The complication with using DHCP addressing on the AP-7868 PDU (and likely many other APC devices), is that their default configuration REQUIRES that your DHCP server set option 43 to a specific vendor code. If you don't do this, the PDU will not accept the DHCP Offer from your server.
APC AP-7868 Smart PDU Serial Pinouts
On your Conserver host
This section includes what you need to type on the CLI console, to configure the PDU to ignore Option 43 and accept the DHCP Offer. I use the spare port on one of my console servers to do this, and I always name my unused ports in my Conserver deployments, so that I can use them at a moments notice. In this case, I'm using port 30, on Console Server 5...
NOTE: the CLI is not good about sanitizing user input, or detecting errors! You can type commas (or TEXT!) in an IP address field, and non-standard TLDs in the domain field! Be very careful about your typing, and visually check your data before accepting the changes!
When you log out of the PDU, the device will reload the TCP stack, and in a few minutes, your PDU should accept the new IP address via DHCP.
Configuring your DHCP Server for Option 43
You need to define you option 43. How (or IF) you can do this will depend on your DHCP server software. But, this section includes the particular HEX characters that the APC PDU expects to see (or it will refuse the DHCP Offer).
These clues are for the Internet Software Consortium (ISC) DHCP server. See the dhcp-options man page - this describes all the standard DHCP options. Look for the section titled "VENDOR ENCAPSULATED OPTIONS", which is option 43. If you already have a class for APC units, then that would be a good place to define option 43 as it will be defined for all devices that match the class. The simplest way to define the value is something like this:
option vendor-encapsulated-options 01:04:31:41:50:43;
DHCP option names and numbers are listed in RFC1533, and in the ISC DHCPd source file common/tables.c
(Explanation: I forget the first character (01), but I believe that it designates to expect HEX characters. The second says there are (4) characters in the field. The last three characters are ASCII for "APC". I don't know why the "31" is there...)
NOTE: All APC PDU's should be protected by UPS power.
In an unprotected environment, if the utility power flickers off and on quickly, the onboard network card may not power back on. In these instances, the PDU may power on but the LED display will be blank and any type of access (web, telnet, console) to the PDU will be disabled. In this case, you must power cycle the PDU again, by pulling the power cord, waiting at least 5 seconds, and re-applying power.
The on-board network card was not designed for quick off/on scenarios. Customers protecting their PDU with a UPS, as designed, will not have this problem.
There are also some APC UPS clues on my console site, http://www.conserver.com/consoles/Clues/cons-apc.html
Regards,
-Zonker-
In my case, we are re-numbering the management subnet, and we've decided to try to make this job easier for the next time a re-number is required. We're telling the devices to use DHCP addressing, and using MAC Address reservations for all of the devices that we care about. Two complications arose; How to connect to the serial CLI, and how to set the device to use DHCP.
The complication with using DHCP addressing on the AP-7868 PDU (and likely many other APC devices), is that their default configuration REQUIRES that your DHCP server set option 43 to a specific vendor code. If you don't do this, the PDU will not accept the DHCP Offer from your server.
APC AP-7868 Smart PDU Serial Pinouts
APC RJ-13 serial console, 4-wire, data-only connection APC Console Lantronix TS (Cisco, Opengear with Cisco wiring, etc.) (RJ-13) (RJ-45) 1 ---(???) nc default settings: 9600-8-N-1 2 ---(GND)--- 5 3 ---(TxD)--> 6 default username: apc 4 <--(RxD)--- 3 default password: apc 5 ---(GND)--- 6 ---(???) nc ( nc = No Connection )
NOTE: You must configure a default gateway for these devices (APC-7868), even if you are only talking on your own subnet!
On your Conserver host
This section includes what you need to type on the CLI console, to configure the PDU to ignore Option 43 and accept the DHCP Offer. I use the spare port on one of my console servers to do this, and I always name my unused ports in my Conserver deployments, so that I can use them at a moments notice. In this case, I'm using port 30, on Console Server 5...
NOTE: the CLI is not good about sanitizing user input, or detecting errors! You can type commas (or TEXT!) in an IP address field, and non-standard TLDs in the domain field! Be very careful about your typing, and visually check your data before accepting the changes!
console unused-cs5-30 (this is the port I picked...) [Enter] [Enter] (twice, to wake-up the CLI interface) apc [Enter] (default login name: apc case-sensitive!) apc [Enter] (default password: apc case-sensitive!) 2 [Enter] (Network) 1 [Enter] (TCP/IP) (If you need the MAC address, you'll find it here) 4 [Enter] (Boot Mode only if currently set for Manual...) 2 [Enter] (DHCP Only) 2 [Enter] (Advanced) 1 [Enter] (Device Name) ps-6.12i [Enter] 2 [Enter> (Domain Name garage.com [Enter] 8 [Enter] (DHCP Cookie Is: ...) 1 [Enter] (Not Required to accept offer) 9 [Enter] (Accept the pending changes) [Esc] (go to the previous menu) [Esc] (go to the previous menu) [Esc] (go to the previous menu) 4 [Enter] (logout)
When you log out of the PDU, the device will reload the TCP stack, and in a few minutes, your PDU should accept the new IP address via DHCP.
Configuring your DHCP Server for Option 43
You need to define you option 43. How (or IF) you can do this will depend on your DHCP server software. But, this section includes the particular HEX characters that the APC PDU expects to see (or it will refuse the DHCP Offer).
These clues are for the Internet Software Consortium (ISC) DHCP server. See the dhcp-options man page - this describes all the standard DHCP options. Look for the section titled "VENDOR ENCAPSULATED OPTIONS", which is option 43. If you already have a class for APC units, then that would be a good place to define option 43 as it will be defined for all devices that match the class. The simplest way to define the value is something like this:
option vendor-encapsulated-options 01:04:31:41:50:43;
DHCP option names and numbers are listed in RFC1533, and in the ISC DHCPd source file common/tables.c
NOTE: All APC PDU's should be protected by UPS power.
In an unprotected environment, if the utility power flickers off and on quickly, the onboard network card may not power back on. In these instances, the PDU may power on but the LED display will be blank and any type of access (web, telnet, console) to the PDU will be disabled. In this case, you must power cycle the PDU again, by pulling the power cord, waiting at least 5 seconds, and re-applying power.
The on-board network card was not designed for quick off/on scenarios. Customers protecting their PDU with a UPS, as designed, will not have this problem.
There are also some APC UPS clues on my console site, http://www.conserver.com/consoles/Clues/cons-apc.html
Regards,
-Zonker-
20100530
How did the year get by me?
Wow, I'm surprised that a year has passed since my last post. Largely, it has been due to my inability to get the editing tools to play well with the iPhone and now the iPad. Normally, I don't update my stuff on employer time, or employer equipment. Couple this with a VERY busy year, and me putting updates lower on the priority list, and here we are on Memorial Day weekend.
A lot has happened in the past 12 months. Too much for a single post, but let me sum up.
I'm working with a variety of Service Processors this year, making the work, and documenting what I've learned. Besides working at the BIOS level, I'm trying to determine if/how the settings can be modified from the OS's involved, and that is more difficult than I supposed. The main problem seems like the websites holding the IPMI drivers or libraries have disappeared after 3-4 years. (The biggest lesson here is to capture, collect, and archive these files for any hardware you acquire. Mergers and acquisitions contribute to the loss of the files, even as the instructions make it into a knowledge base.)
An old lesson re-learned, is that older Sun gear will ignore their serial console settings as soon as you plug a keyboard in! Your only clue, if you are logging the console output with Conserver or some other application, is a small line about detecting a keyboard. Removing the keyboard does NOT revert back to serial console output, you need to reboot the server with no keyboard attached. I knew this long ago, but then was thwarted when folks in a remote office attached a keyboard, and we didn't notice that console I/O had gone away.
I've also been doing a LOT of work with SNMP in the past 6 months, and I found an oddity with an MGE (which sold the line to APC, who then sold the line to Schneider Electric) room-sized UPS. The brains of the UPS uses a serial line to pass a data stream to the Network Interface shelf. If this streaming interface is removed, the SNMP cards do not get an update... But the cards simply report the LAST value for any queried OID. As a result, your only clue of a disconnect is seeing all results "flat-line". BUT, if you are also set up forSNMP traps, you will get also see a string that communications was lost with the UPS. This is a case where you really need to be able to parse Syslog and SNMP traps for interesting strings and send an alert.
Other UPS problems led to the integration of a UPS of Last Resort, so that core devices could track and log events with the big UPS. This included watching the serial output of the APC Smart-UPS 2200XL, plus SNMP integration into our monitoring system.
Another Big Project in the past year was due to a planned network outage. This was due to a planned partial power outage in the building. Unfortunately, some key devices also lost power, causing trouble, which could not send alerts via email because the network was down. We decided that we needed to make sure there was a second path out of the data center for critical alerts and messages. This led to integration with a FoxBox, a Network-connected cell-phone device, allowing SMS alerting AND control!
I've also added many more hosts to the host-to adapter database, but I only have a basic Cyclades console connection guide posted, and I still need to make a Digi page. (Cyclades was another fine example of collecting information from the web as soon as you hear about an acquisition!)
I'll make stronger efforts to post more frequently through the rest of 2010. thanks for reading.
-Zonker-
A lot has happened in the past 12 months. Too much for a single post, but let me sum up.
I'm working with a variety of Service Processors this year, making the work, and documenting what I've learned. Besides working at the BIOS level, I'm trying to determine if/how the settings can be modified from the OS's involved, and that is more difficult than I supposed. The main problem seems like the websites holding the IPMI drivers or libraries have disappeared after 3-4 years. (The biggest lesson here is to capture, collect, and archive these files for any hardware you acquire. Mergers and acquisitions contribute to the loss of the files, even as the instructions make it into a knowledge base.)
An old lesson re-learned, is that older Sun gear will ignore their serial console settings as soon as you plug a keyboard in! Your only clue, if you are logging the console output with Conserver or some other application, is a small line about detecting a keyboard. Removing the keyboard does NOT revert back to serial console output, you need to reboot the server with no keyboard attached. I knew this long ago, but then was thwarted when folks in a remote office attached a keyboard, and we didn't notice that console I/O had gone away.
I've also been doing a LOT of work with SNMP in the past 6 months, and I found an oddity with an MGE (which sold the line to APC, who then sold the line to Schneider Electric) room-sized UPS. The brains of the UPS uses a serial line to pass a data stream to the Network Interface shelf. If this streaming interface is removed, the SNMP cards do not get an update... But the cards simply report the LAST value for any queried OID. As a result, your only clue of a disconnect is seeing all results "flat-line". BUT, if you are also set up forSNMP traps, you will get also see a string that communications was lost with the UPS. This is a case where you really need to be able to parse Syslog and SNMP traps for interesting strings and send an alert.
Other UPS problems led to the integration of a UPS of Last Resort, so that core devices could track and log events with the big UPS. This included watching the serial output of the APC Smart-UPS 2200XL, plus SNMP integration into our monitoring system.
Another Big Project in the past year was due to a planned network outage. This was due to a planned partial power outage in the building. Unfortunately, some key devices also lost power, causing trouble, which could not send alerts via email because the network was down. We decided that we needed to make sure there was a second path out of the data center for critical alerts and messages. This led to integration with a FoxBox, a Network-connected cell-phone device, allowing SMS alerting AND control!
I've also added many more hosts to the host-to adapter database, but I only have a basic Cyclades console connection guide posted, and I still need to make a Digi page. (Cyclades was another fine example of collecting information from the web as soon as you hear about an acquisition!)
I'll make stronger efforts to post more frequently through the rest of 2010. thanks for reading.
-Zonker-
Subscribe to:
Posts (Atom)