You could try to tell me that it's because serial consoles are sensitive access points to vital hardware, and that Security is inversely proportional to Ease Of Use, therefore Service Processors cannot be Easy, because they need to be Secure. And, doing my best Jeremy Clarkson (Top Gear) impression, I'd tell you that "you'd be wrong".
I'll tell you that it's because most the consumers are not asking for ease of use, or for interoperability. Because it seems (to me) to be on the minds (and mouths, and blogs) of a relatively small subset of vocal customers, and the 'collective voice' of this subset is further diminished because we are asking different vendors. (That is, I ask Dell, you ask HP, she asks Sun...but darn few of us are asking more than one or two vendors.)
Security features are only just now beginning to be built-in, either already enabled, or very easy to configure, largely because the US government has tied such requirements to their purchasing rules. If you want to sell the US government a desktop computer, there is now a whole checklist of security items to tick on the list before your kit might be considered. The US government, across many departments, now speaks with a large, unified voice (through the purchasing rules and policies)
Similarly, in the early days of the Point-to-Point Protocol (the follow-on to S.L.I.P. for IP-over-modem communications), PPP would allow more computer network protocols to connect and exchange data. As a customer, you'd have thought that vendor A's PPP (the choice inYOUR shop) would talk to Vendor B's gear at some Strategic Partner's network. But you'd have been wrong. Coupled with the birth of the Commercial Internet (it wasn't just for DARPA anymore), many companies were trying to get onto the Internet, or to connect to other companies to exchange information.
There was a large dissatisfacton with a lack up interoperability. And many companies spoke with the vendors to complain. BUT, besides complaining to the vendor who supplied the kit to YOU, you also would have complained to the vendors of the other guys gear as well. You might complain to 5 or 10 vendors, between routers and modems.
As a result of the dissatisfaction of LOTS of users, 6 companies started the PPP Consortium, with a week-long meeting at Telebit Corp in Sunnyvale, CA, with the express purpose of trying to test all of their gear with each other, and test across the matrix of then-new (and still evolving) PPP protocol options. At then end of it, the scorecard wasn't too bad, and all of the participants had a list of things to fix. They also hashed out some of the ambiguities in the draft Request For Comments (RFCs) describing the options. (Many of the vendors were part of the IEEE, and were helping define the RFCs, but the meeting helped settle misunderstandings about the options.)
Because the group of six could now claim a large amount of interoperability with the others in the group, it helped their marketing. Because the customers saw progress, they harped on the vendors who were NOT part of the original group, and they wanted to get in on the consortium. Six months after the first group, I believe there were 13 members, and 18 at the next. By the third meeting, PPP was Stable. PPP 'just worked', most of the time. PPP had become "easy", and SLIP was largely relagated to antique links on old equipment.
When I look at the various implementations of BIOS redirection to the serial port, they almost exclusively presume (or require) a modem connection, which then requires hardware level wiring issues if you are using a Console Server connection. The mapping of Function Keys, command sequences, and the timing issues to invoke the commands make using them tricky on a Console Server. It makes them impractical on any link slower than 19.2 Kbps. Yet the few BIOS makers don't see a need to try to do better, let alone to work with the other vendors to provide some consistancy of user interface to their customers. Do they know whether you care?
Now that Service Processors are becoming prevalent on new servers, it's hard to get a server without one. Yet they suffer from problems very similar to BIOS. The user interfaces are different from vendor to vendor. But what about processors from the same vendor. What about within the same vendor, across a hardware platform? You'd think that a vendor might want to keep their command sequences consistent, for the sake of their customers (who might be scripting support sequences for their machines). You'd probably like to think that...but you'd be wrong. A couple vendors seem to change at will, but I'm guessing that they are just buying server designs from other companies...and those same vendors probably haven't taken the time to design that Human User Interface, they probably simply rely on the hardware maker to fulfill the tasks (power cycle the server, set the IP address of the Ethernet interface for service processor, implement a web interface with certain features), leaving the details of keystrokes to the hardware maker...and when the vendor chooses another hardware maker, they get the same latitude with the design, and make different choices. (I wonder what Donald Norman thinks about tis problem.) Do your vendors know if you care?
When you tell a sales person that you want some new feature on their next year's model, the usual response is "OK, if we can get that done, how many will you buy?". It's going to cost them labor hours, engineering and QA time, technical documentation, maybe changing part of a training class. While you may only want 6, or 12, or 40, your voice will be tallied. When they from enough of you, it's going to be worth it to make a change. But we need to tip that scale by asking, and probably tying that request to some money. (My dad would sometimes ask me to "write your request on a $5 bill, and I'll consider it", but I have seen this over and over again, from different seats in different companies. For the vendor, it's all about money.)
My point here, is that you are going to spend money on new servers at some point. Make your money count! Tie the money to your requests for easier-to-use service processors. Decide what you want, and tell your possible vendors about your requirements. They all want your money...so vote with your wallet! Reward the responsive vendors! If the unresponsive vendors don't get the order, they'll at least know why, and they'll learn that you were serious. As more of us tie our requirements to purchasing, we can make a difference.
As you get into your busy end-of-year, some of you are already going through your budget cycle for 2009. Use any slowdown you get to think about requirements to go along with those purchases. Get them on paper, and in email, so they are ready to send to your vendors with your Request for Quote. And good luck in the new year.
-Z-
20081020
A little perl of wisdom
When it comes to finding some quick help (after the Helpdesk has closed), or those tidbits of knowledge and experience of someone who has tread the path before me, Google has been my friend for many years (taking up the mantle after AltaVista!).
But, there is a downside to learning all about a topic from Google. It is that you, the reader, are guiding the learning. You are getting the answers to the questions you ask...but it doesn't always offer things that you should know (or at least consider), based on your searches.
I taught myself perl, reading books, and using Google for answers, and examples...but I sometimes found myself confused when reading someone else's code. This year, I'm taking formal classes, from a few sources, hoping to patch a few holes in my foundation of knowledge.
While I'm glad to have taken this approach, to have someone guiding me through the learning, systematically, and answering my questions...I'm finding that the Teacher in this equation is even more important than the content. Even if someone knows the content well, that doesn't mean that they will be a good teacher. (Answering the occasional question does not equal teaching.)
After 5 weeks in a semester-long class at a local community college, I too a full-day Perl 101 course from Bayview Training...and then I coded for 15 hours straight, because that class had made a number of concepts more concrete for me (several "light-bulb" moments), and I was excited about the possibilities for my future projects.
I heartily recommend Bill Ward as an instructor. While he teaches primarily in Silicon Valley, he does travel for some classes and consulting as well. The cost of the course was money well spent! If you really need to learn some perl, check out Bayview Training and Bill Ward!
But, there is a downside to learning all about a topic from Google. It is that you, the reader, are guiding the learning. You are getting the answers to the questions you ask...but it doesn't always offer things that you should know (or at least consider), based on your searches.
I taught myself perl, reading books, and using Google for answers, and examples...but I sometimes found myself confused when reading someone else's code. This year, I'm taking formal classes, from a few sources, hoping to patch a few holes in my foundation of knowledge.
While I'm glad to have taken this approach, to have someone guiding me through the learning, systematically, and answering my questions...I'm finding that the Teacher in this equation is even more important than the content. Even if someone knows the content well, that doesn't mean that they will be a good teacher. (Answering the occasional question does not equal teaching.)
After 5 weeks in a semester-long class at a local community college, I too a full-day Perl 101 course from Bayview Training...and then I coded for 15 hours straight, because that class had made a number of concepts more concrete for me (several "light-bulb" moments), and I was excited about the possibilities for my future projects.
I heartily recommend Bill Ward as an instructor. While he teaches primarily in Silicon Valley, he does travel for some classes and consulting as well. The cost of the course was money well spent! If you really need to learn some perl, check out Bayview Training and Bill Ward!
20080828
Big News! SSH clients for iPhone!
I haven't had any 'big news' on the console front for a couple months now. I have been missing my BlueConsole adapters (which I've blogged about before), and SSH access from my iPhone, BUT that has changed!
New in the Apple Apps Store are four SSH applications (use the search dialogue, look for "ssh"...). I've already bought two to try, and one has already met my most common needs. In a nutshell, here are the four I found, with URLs to the maker's websites, in case you want to do your own homework on the topic.
TouchTerm http://www.jbrink.net/touchterm/index.html $2.99 This one looks pretty good, from the online docs. There is only basic information on the maker's home page, BUT, check the SUPPORT page! Key Exchange, font colors, different input modes (immediate, batch, command history). The developer offers an email address for comments and suggestions. This looks like a good pick!
pTerm http://www.instantcocoa.com/products/pTerm/ $4.99 This maker also lists an email address, as well as a Google discussion group. The web page posts a short-term development schedule. They also boast that this app is based on PuTTY, and I'm a big PuTTY fan (but I understand that this app is only scratching the surface on features). I've sent email to the developer with my wants, and I already received a positive reply. He is also willing to work on the bluetooth-serial-devices interfacing as well.
iSSH http://www.zinger-soft.com/iSSH_features.html $4.99 This one also looks good. The main web page is odd... there are small, light-grey arrows under a few of the descriptions...they indicate that there are additional items at this menu level of the web page. Clicking on them will show you the other topics. This one also supports shared keys, but doesn't seem as flexible (to me, yet), as TouchTerm.
SSH http://www.throughput.biz/ $3.99 This looks simple, and it word-wraps the screen. There is little information on the maker's homepage. I'm currently not inclined to try it.
So, with this in mind, I have bought the first two (TouchTerm and pTerm), and I'm spending my efforts to put TouchTerm through it's paces first, because pTerm doesn't have RSA/DSA keys yet (though keys are on the short-range list for new features, so I'll be giving them both a good workout in September). I was able to set up a basic profile, with screen and font colors, RSA-2048 key, emailed the key to another host, so I could place the key, and use the session. It will take a bit of getting used to, since I'll need to leave the fonts a bit larger, and pan around the screen (using mutt was kinda awkward, but that won't be my primary use of SSH). I like it so far, and I hope to be able to put my BlueConsole adapters back into service sometime in the near future.
I hope you had a chance to enjoy some vacation this summer as well.
-Z-
New in the Apple Apps Store are four SSH applications (use the search dialogue, look for "ssh"...). I've already bought two to try, and one has already met my most common needs. In a nutshell, here are the four I found, with URLs to the maker's websites, in case you want to do your own homework on the topic.
TouchTerm http://www.jbrink.net/touchterm/index.html $2.99 This one looks pretty good, from the online docs. There is only basic information on the maker's home page, BUT, check the SUPPORT page! Key Exchange, font colors, different input modes (immediate, batch, command history). The developer offers an email address for comments and suggestions. This looks like a good pick!
pTerm http://www.instantcocoa.com/products/pTerm/ $4.99 This maker also lists an email address, as well as a Google discussion group. The web page posts a short-term development schedule. They also boast that this app is based on PuTTY, and I'm a big PuTTY fan (but I understand that this app is only scratching the surface on features). I've sent email to the developer with my wants, and I already received a positive reply. He is also willing to work on the bluetooth-serial-devices interfacing as well.
iSSH http://www.zinger-soft.com/iSSH_features.html $4.99 This one also looks good. The main web page is odd... there are small, light-grey arrows under a few of the descriptions...they indicate that there are additional items at this menu level of the web page. Clicking on them will show you the other topics. This one also supports shared keys, but doesn't seem as flexible (to me, yet), as TouchTerm.
SSH http://www.throughput.biz/ $3.99 This looks simple, and it word-wraps the screen. There is little information on the maker's homepage. I'm currently not inclined to try it.
So, with this in mind, I have bought the first two (TouchTerm and pTerm), and I'm spending my efforts to put TouchTerm through it's paces first, because pTerm doesn't have RSA/DSA keys yet (though keys are on the short-range list for new features, so I'll be giving them both a good workout in September). I was able to set up a basic profile, with screen and font colors, RSA-2048 key, emailed the key to another host, so I could place the key, and use the session. It will take a bit of getting used to, since I'll need to leave the fonts a bit larger, and pan around the screen (using mutt was kinda awkward, but that won't be my primary use of SSH). I like it so far, and I hope to be able to put my BlueConsole adapters back into service sometime in the near future.
I hope you had a chance to enjoy some vacation this summer as well.
-Z-
20080713
Social Networking, professional connections
The question comes up, from time to time... Why do System Administrators seem to be better at (Social) Networking than Network Administrators. You'd think that Netadmins would have the communications thing down cold. But, I've found, we tend to have smaller social networks, but with links we can rely upon to do our jobs.
Linked in? Yep, I've been there a while...
But, why are there so many more Systems-related conferences than Network-related? OS related trade shows, versus Network trade shows.
I just find it odd, that's all.
-Z-
Linked in? Yep, I've been there a while...
But, why are there so many more Systems-related conferences than Network-related? OS related trade shows, versus Network trade shows.
I just find it odd, that's all.
-Z-
20080704
Can you hear me now? Wait, now I can't hear you!
California's two new laws regarding hands-free cellphone use while driving have Bluetooth headsets in the news, and on the front page of many papers. I still haven't found the right combination in any solution that I've tried. Every one has had some deficit.
Since I wear glasses, I don't like anything that has an ear-loop. None have been comfortable so far. I also prefer an in-ear (ear-bud) style, versus an over-the-ear solution with a headband. This certainly limits my choices, but there have been a few good choices. Good, but none were Great.
For listening to calls, the best for me has ben the Sony HB-808 headset. Good talk-time, and lightweight, with sufficient audio to the earpiece, and the microphone has some automatic gain control, but it is not so good about noise cancellation. It comes with an earloop, which can be reversed to wear on either ear (that's fairly standard), but the art near my ear was also uncomfortable, because it pressed against my ear when I used the earloop. But, the diameter of the earphone was slightly smaller than an original iPhone earphone, so I tried using the Griffin EarJam earbud adapters for the iPod earphones...This was a HUGE improvement! It now fits in my ear without an earloop. It's light enough to stay in my ear. And, because the earbud is in my ear, it seals out local noises, while allowing me to turn the earpiece volume down! Even in a data center, staring down dozens of fans, I can hear my calls just fine...but, that's where it all breaks down. Because the microphone has gain control, it hears all of the fans, so I constantly have to juggle the phone to mute and unmute the microphone during calls in the data center to service gear. It frustrating for me.
I finally bought an original Jawbone, because of the noise canceling technology. It really is great, for the people I call. I don't have to mute my microphone in the data center (in most cases), which has been really useful. But, the device has an awkward earloop, since the device MUST rest on your cheek in order for the noise canceling to work. And, since it's pressing the microphone aginst my cheek, the earpiece hovers above my ear. Folks can ear me, but I've got to turn the volume up to 11 to hear them over the local noise.
I tried all 4 of the ear-bits that came with the Jawbone, but none were comfortable, and only one came close to helping me lower the volume of the earpiece. The earpiece is also slightly smaller than the iPod earphone, so the Griffin earbud trick wouldn't seat securely. Even trying to secure the Griffin part, it would fit in my ear, but then the microphone wouldn't press against my cheek (defeating the noise cancellation), so that trick didn't work. BUT, at MacWorld SF 2007, I found Comply's "Whoomp!" earbud enhancers. They are snugger, and more flexible than the Griffin part, with a foam earpiece on a slightly slanted axis. This held the Jawbone snugly, pressed the mirophone against my cheek, and sealed the outside noise nicely...but, the foam bit was hard to clean. I still use it, but each earbud only lasts about 3-4 months for me.
A friend turned me on to another option for the original Jawbone; Using the Jabra Ear-gels (available from Hello Direct for about $6US), which gives you another set of 4 sizes, which SNUGLY fit the Jawbone. hey are more supple than the original Jawbone ear-bits, but they still aren't as comfortable as I would like. My friend has already tried the new Jawbone, and has been favorably impressed.
I've tried using commercial adapters (cable, switch and microphone) with my own iPod earbuds, but the microphone still hears the fans. (It's a dichotomy; If I can hear you well, you can't hear me well...why can't they make a headset that allows both?)
I'm fine using a headset in the car, but I received a Tom-Tom for my birthday, since it has bluetooth hands-free speaker phone capability. This works OK, but I understand it's a high-theft-rate target. I can't leave it in the car, lest someone breaks a window to steal it. (Even if you take it inside, I'm told that leaving the suction-cup mount on your windshield is enough attraction that a thief may smash your window to see if you stashed it inside the car.) As a result, I now have one more thing to stow and carry back and forth.
Should I try using the Tom-Tom as a speakerphone in the data center? They don't brag about noise cancellation, but folks seem to hear me pretty good when I'm in the car. :-)
-Z-
Since I wear glasses, I don't like anything that has an ear-loop. None have been comfortable so far. I also prefer an in-ear (ear-bud) style, versus an over-the-ear solution with a headband. This certainly limits my choices, but there have been a few good choices. Good, but none were Great.
For listening to calls, the best for me has ben the Sony HB-808 headset. Good talk-time, and lightweight, with sufficient audio to the earpiece, and the microphone has some automatic gain control, but it is not so good about noise cancellation. It comes with an earloop, which can be reversed to wear on either ear (that's fairly standard), but the art near my ear was also uncomfortable, because it pressed against my ear when I used the earloop. But, the diameter of the earphone was slightly smaller than an original iPhone earphone, so I tried using the Griffin EarJam earbud adapters for the iPod earphones...This was a HUGE improvement! It now fits in my ear without an earloop. It's light enough to stay in my ear. And, because the earbud is in my ear, it seals out local noises, while allowing me to turn the earpiece volume down! Even in a data center, staring down dozens of fans, I can hear my calls just fine...but, that's where it all breaks down. Because the microphone has gain control, it hears all of the fans, so I constantly have to juggle the phone to mute and unmute the microphone during calls in the data center to service gear. It frustrating for me.
I finally bought an original Jawbone, because of the noise canceling technology. It really is great, for the people I call. I don't have to mute my microphone in the data center (in most cases), which has been really useful. But, the device has an awkward earloop, since the device MUST rest on your cheek in order for the noise canceling to work. And, since it's pressing the microphone aginst my cheek, the earpiece hovers above my ear. Folks can ear me, but I've got to turn the volume up to 11 to hear them over the local noise.
I tried all 4 of the ear-bits that came with the Jawbone, but none were comfortable, and only one came close to helping me lower the volume of the earpiece. The earpiece is also slightly smaller than the iPod earphone, so the Griffin earbud trick wouldn't seat securely. Even trying to secure the Griffin part, it would fit in my ear, but then the microphone wouldn't press against my cheek (defeating the noise cancellation), so that trick didn't work. BUT, at MacWorld SF 2007, I found Comply's "Whoomp!" earbud enhancers. They are snugger, and more flexible than the Griffin part, with a foam earpiece on a slightly slanted axis. This held the Jawbone snugly, pressed the mirophone against my cheek, and sealed the outside noise nicely...but, the foam bit was hard to clean. I still use it, but each earbud only lasts about 3-4 months for me.
A friend turned me on to another option for the original Jawbone; Using the Jabra Ear-gels (available from Hello Direct for about $6US), which gives you another set of 4 sizes, which SNUGLY fit the Jawbone. hey are more supple than the original Jawbone ear-bits, but they still aren't as comfortable as I would like. My friend has already tried the new Jawbone, and has been favorably impressed.
I've tried using commercial adapters (cable, switch and microphone) with my own iPod earbuds, but the microphone still hears the fans. (It's a dichotomy; If I can hear you well, you can't hear me well...why can't they make a headset that allows both?)
I'm fine using a headset in the car, but I received a Tom-Tom for my birthday, since it has bluetooth hands-free speaker phone capability. This works OK, but I understand it's a high-theft-rate target. I can't leave it in the car, lest someone breaks a window to steal it. (Even if you take it inside, I'm told that leaving the suction-cup mount on your windshield is enough attraction that a thief may smash your window to see if you stashed it inside the car.) As a result, I now have one more thing to stow and carry back and forth.
Should I try using the Tom-Tom as a speakerphone in the data center? They don't brag about noise cancellation, but folks seem to hear me pretty good when I'm in the car. :-)
-Z-
20080605
Zonker needs a Bluetooth terminal emulator App for iPhone
The longer I go without this capability, the stronger my desire for an application for the iPhone to fill my needs. But, I don't want to hack my phone...I want a real iPhone App, and now it's worth money to me. I can't find a 'Wish List' page that hasn't been taken over by spammers. So, I'll try the Open Letter approach.
I have two needs, with a common root...both need a basic vt-100 terminal emulator underneath. ANSI color would be a nice touch, but isn't necessary. I don't need Wyse, or full-blown TN-3270 emulation...keep it simple. Some scrollback buffer would be useful (say ~2600 lines), and the ability to capture to a log file (and email the log later?) would be very handy. So, what else?
The primary need is to have an SSH v2 client, so I can SSH directly to my machines using the Edge or wifi connections. This also probably means solving the pre-shared keys issue...how do I get the keys onto the iPhone? I still can't sync a simple file to my phone. Maybe I need a key-generator on the phone, but I still need to get the public key to the hosts.
The secondary need is to communicate with a Bluetooth serial device. Specifically, a BlueConsole dongle, although there are other devices with which you should be able to interoperate. (I used to use the Tri-connect package (from Tridone) on my Treo 650. It wasn't great, but it was a good option, which let me do a lot of quick, little jobs without lugging a computer around.) I'm sorely tempted to carry the Treo around in my tool bag, for those occasions when I'd really like to use these serial devices...if I could find a good way to keep the Treo battery charged while it's in the tool bag, I'd do it. But, I'd much rather use the iPhone for this task.
If you're an iPhone developer, working on a terminal emulator that will use the Bluetooth port, let's talk! I'm willing to be a software tester, give feedback, and help with documentation. I've got good credentials in both the serial console and the communications software test realms. We may also be able to work out a loan of a BlueConsole device while you're developing.
If you're an iPhone developer, working on an SSH v2 client, I want to talk to you as well, and I'm willing to be a tester for this as well. (I'd love to know that someone is trying to fulfill this need.) My favorite SSH app is PuTTY (by Simon Tatham and others), if you want to set your sites high. If you are trying to determine what folks 'want', versus what the 'need' (and would be willing to pay for), I'd like to talk to you.
Since Father's Day is in June, I'm going to post information about some of my favorite new tools in my next posts, in case you are shopping for special gifts. (Yes, the iPhone is on the list.)
-Z-
I have two needs, with a common root...both need a basic vt-100 terminal emulator underneath. ANSI color would be a nice touch, but isn't necessary. I don't need Wyse, or full-blown TN-3270 emulation...keep it simple. Some scrollback buffer would be useful (say ~2600 lines), and the ability to capture to a log file (and email the log later?) would be very handy. So, what else?
The primary need is to have an SSH v2 client, so I can SSH directly to my machines using the Edge or wifi connections. This also probably means solving the pre-shared keys issue...how do I get the keys onto the iPhone? I still can't sync a simple file to my phone. Maybe I need a key-generator on the phone, but I still need to get the public key to the hosts.
The secondary need is to communicate with a Bluetooth serial device. Specifically, a BlueConsole dongle, although there are other devices with which you should be able to interoperate. (I used to use the Tri-connect package (from Tridone) on my Treo 650. It wasn't great, but it was a good option, which let me do a lot of quick, little jobs without lugging a computer around.) I'm sorely tempted to carry the Treo around in my tool bag, for those occasions when I'd really like to use these serial devices...if I could find a good way to keep the Treo battery charged while it's in the tool bag, I'd do it. But, I'd much rather use the iPhone for this task.
If you're an iPhone developer, working on a terminal emulator that will use the Bluetooth port, let's talk! I'm willing to be a software tester, give feedback, and help with documentation. I've got good credentials in both the serial console and the communications software test realms. We may also be able to work out a loan of a BlueConsole device while you're developing.
If you're an iPhone developer, working on an SSH v2 client, I want to talk to you as well, and I'm willing to be a tester for this as well. (I'd love to know that someone is trying to fulfill this need.) My favorite SSH app is PuTTY (by Simon Tatham and others), if you want to set your sites high. If you are trying to determine what folks 'want', versus what the 'need' (and would be willing to pay for), I'd like to talk to you.
Since Father's Day is in June, I'm going to post information about some of my favorite new tools in my next posts, in case you are shopping for special gifts. (Yes, the iPhone is on the list.)
-Z-
20080513
What is 'Normal', anyway?
Yow! It's already well into May, and April got by without a post. It was a busy month, with a LOT of discovery tasks happening. I'll try to get a few more posts in this month to summarize. But first, I wanted to get this thought online...
Those who know me are not surprised if the come into a dark wiring closet, and find me sitting on a milk crate, staring at the 'blinkly lights'. (Frankly, I've startled plenty of people who don't know me this way!) Most just write it of as a fixation on blinky lights, but it is not. This is how I can 'visually spot trends' of activity on devices and interfaces. It's also a chance to remember which link status indicators are on, off, in an alarm condition. And, I'm also listening... to fans, air conditioner inlets, hard drive spindle bearings, modem speakers, relays clicking.
This is how I get to know what "Normal" looks like, sounds like, and smells like in the data center and wiring closets. (Yes, smell... does it 'usually' smell damp in this room? Is the 'ozone' smell something that's always here, or does it indicate a component failure. Scent is a strong trigger for memory!)
Knowing what is 'Normal' helps us spot what is unusual. When I have a network failure, I can go to the associated wiring closet and look, listen, and smell...I don't need to ponder "has that always been like this?", because I'll remember. "That light is usually blinking...so there isn't traffic on that interface!" Knowing what is normal is a key to fast troubleshooting.
The RRD tool has been a great resource for graphing monitoring data, allowing you to visualize 'normal', to see 'now', and to identify trends. You find this under MANY open source tools, such as MRTG, Cricket, Cacti, and many more.
But, can you tell what's 'normal' with a serial console? Yes, you can! The key is, you need to LOOK when things are operating normally. Look when the system/device/network is idle some night or weekend. Look again when backups are running. Look again when the network is busy, but not failing. Then, compare your notes, or, your LOGS!
You can do a LOT interactively, using a simple terminal emulator and a cable. You have a lot more flexibility with a Console Server and multiple Telnet sessions (for example, you can monitor many consoles simultaneously, and cut-and-paste between them). The real benefits are had when you combine the console servers with a Serial Console Management Application such as Conserver, or ConsoleWorks, and you can compare historic data with today's results.
I have a handful of devices which have "diagnostics' ports, that only the field engineers will use. They are not for normal use by customers. However, when you connect to these ports, you can find some of the devices are 'beaconing' about events, or reporting regular status messages. Even if the port doesn't say much, if you hit a carriage return, you'll probably get a prompt...doing this occasionally will tell you it's still alive. And, sometimes, you can also leverage the simple diagnostics that are built in. It's good to have a baseline log of what the devices are reporting to these diagnostics when they are healthy, so you can compare them if the device starts misbehaving.
But, to do any of these things, you need to start by looking when things are working as expected, doing their job the way you want them to. That's when you want to get your first look, and save that data for later comparison. (Speaking of comparing log data, try SPLUNK! Check out splunkbase, and the SPLUNK Forums! Splunk is worth a few blog articles by itself...later.)
The bottom line: Do you KNOW what you are missing? Do you WANT to know? Then LOOK!
-Z-
Those who know me are not surprised if the come into a dark wiring closet, and find me sitting on a milk crate, staring at the 'blinkly lights'. (Frankly, I've startled plenty of people who don't know me this way!) Most just write it of as a fixation on blinky lights, but it is not. This is how I can 'visually spot trends' of activity on devices and interfaces. It's also a chance to remember which link status indicators are on, off, in an alarm condition. And, I'm also listening... to fans, air conditioner inlets, hard drive spindle bearings, modem speakers, relays clicking.
This is how I get to know what "Normal" looks like, sounds like, and smells like in the data center and wiring closets. (Yes, smell... does it 'usually' smell damp in this room? Is the 'ozone' smell something that's always here, or does it indicate a component failure. Scent is a strong trigger for memory!)
Knowing what is 'Normal' helps us spot what is unusual. When I have a network failure, I can go to the associated wiring closet and look, listen, and smell...I don't need to ponder "has that always been like this?", because I'll remember. "That light is usually blinking...so there isn't traffic on that interface!" Knowing what is normal is a key to fast troubleshooting.
The RRD tool has been a great resource for graphing monitoring data, allowing you to visualize 'normal', to see 'now', and to identify trends. You find this under MANY open source tools, such as MRTG, Cricket, Cacti, and many more.
But, can you tell what's 'normal' with a serial console? Yes, you can! The key is, you need to LOOK when things are operating normally. Look when the system/device/network is idle some night or weekend. Look again when backups are running. Look again when the network is busy, but not failing. Then, compare your notes, or, your LOGS!
You can do a LOT interactively, using a simple terminal emulator and a cable. You have a lot more flexibility with a Console Server and multiple Telnet sessions (for example, you can monitor many consoles simultaneously, and cut-and-paste between them). The real benefits are had when you combine the console servers with a Serial Console Management Application such as Conserver, or ConsoleWorks, and you can compare historic data with today's results.
I have a handful of devices which have "diagnostics' ports, that only the field engineers will use. They are not for normal use by customers. However, when you connect to these ports, you can find some of the devices are 'beaconing' about events, or reporting regular status messages. Even if the port doesn't say much, if you hit a carriage return, you'll probably get a prompt...doing this occasionally will tell you it's still alive. And, sometimes, you can also leverage the simple diagnostics that are built in. It's good to have a baseline log of what the devices are reporting to these diagnostics when they are healthy, so you can compare them if the device starts misbehaving.
But, to do any of these things, you need to start by looking when things are working as expected, doing their job the way you want them to. That's when you want to get your first look, and save that data for later comparison. (Speaking of comparing log data, try SPLUNK! Check out splunkbase, and the SPLUNK Forums! Splunk is worth a few blog articles by itself...later.)
The bottom line: Do you KNOW what you are missing? Do you WANT to know? Then LOOK!
-Z-
20080331
One moment please, I'll connect you...
Today's topic has been a recurring theme during the past few weeks.
On the one hand, while I'm in love with my iPhone for email and web access pretty much everywhere, I still really miss having an SSH client on my hip. (On my Treo, the pssh client did the trick. It was simple, talked SSH v2, had a poor random number generator, but could import strong pre-shared keys from a Palm memo.) I'm hoping that a simple, cool new SSH terminal app for the iPhone is just around the corner, now that the iPhone Software Developer Kit is out. I don't need fancy key-mapping, or a variety of terminal emulations. If it can offer to let me talk to Bluetooth devices as well as the Edge/WiFi connections (so I can use my Blue Console devices, it will be worth more to me. (That's a blatant plea/hint to any iPhone developers. Hopefully I've said iPhone enough times for this article to turn up in a web search. :-)
On the other hand, I've found the Service Processor on some SUN Opteron boxes to be a bit vexing. Many hosts have these Service Processors (S.P. from here on) on-board now, and each vendor is implementing it a bit differently. It's an added 'black box' on the communications line between the physical serial port and the serial I/O (tty) port on the host.
When BIOS Redirection was added, this was the first complication making serial ports provide the GETTY console login under a Linux host. Now these processors, supporting ILOM/ALOM/e-i-e-i-o communications are also listening to the line, looking for meta-sequences, waiting to interrupt the conversation and do your bidding. One gotcha is, when you start talking to the S.P., you lose any logging from the Operating System. (Why don't these special purpose computers BUFFER at least SOME amount of host data?)
The odd hang-up with the SUN units I'm wrestling seems to be an inactivity timeout. If I get into the S.P., and let it sit for a while, it seems to go to sleep, and I can't wake it up again. BUT, it never returned the connection back to the OS. Essentially, the O.S. console connection is now bottled-up, and I can't get the S.P. to answer through the serial port, or to relinquish the port back to the O.S., leaving me to reboot the system in order to get the serial console back. There MUST be a better way.
Fortunately, there is a good write-up on the SUN S.P., and I'm going to go over it this weekend. Hopefully I'll find some variables to change the meta characters which trigger the S.P., since they are too close to another sequence I need to send, and I also hope to find some timer settings to get the S.P. to drop out politely if I time out of the session.
-Z-
On the one hand, while I'm in love with my iPhone for email and web access pretty much everywhere, I still really miss having an SSH client on my hip. (On my Treo, the pssh client did the trick. It was simple, talked SSH v2, had a poor random number generator, but could import strong pre-shared keys from a Palm memo.) I'm hoping that a simple, cool new SSH terminal app for the iPhone is just around the corner, now that the iPhone Software Developer Kit is out. I don't need fancy key-mapping, or a variety of terminal emulations. If it can offer to let me talk to Bluetooth devices as well as the Edge/WiFi connections (so I can use my Blue Console devices, it will be worth more to me. (That's a blatant plea/hint to any iPhone developers. Hopefully I've said iPhone enough times for this article to turn up in a web search. :-)
On the other hand, I've found the Service Processor on some SUN Opteron boxes to be a bit vexing. Many hosts have these Service Processors (S.P. from here on) on-board now, and each vendor is implementing it a bit differently. It's an added 'black box' on the communications line between the physical serial port and the serial I/O (tty) port on the host.
When BIOS Redirection was added, this was the first complication making serial ports provide the GETTY console login under a Linux host. Now these processors, supporting ILOM/ALOM/e-i-e-i-o communications are also listening to the line, looking for meta-sequences, waiting to interrupt the conversation and do your bidding. One gotcha is, when you start talking to the S.P., you lose any logging from the Operating System. (Why don't these special purpose computers BUFFER at least SOME amount of host data?)
The odd hang-up with the SUN units I'm wrestling seems to be an inactivity timeout. If I get into the S.P., and let it sit for a while, it seems to go to sleep, and I can't wake it up again. BUT, it never returned the connection back to the OS. Essentially, the O.S. console connection is now bottled-up, and I can't get the S.P. to answer through the serial port, or to relinquish the port back to the O.S., leaving me to reboot the system in order to get the serial console back. There MUST be a better way.
Fortunately, there is a good write-up on the SUN S.P., and I'm going to go over it this weekend. Hopefully I'll find some variables to change the meta characters which trigger the S.P., since they are too close to another sequence I need to send, and I also hope to find some timer settings to get the S.P. to drop out politely if I time out of the session.
-Z-
20080301
Tools and trade-offs...
Recent circumstances gave me the opportunity to switch my phone service from a Palm Treo 650 to an Apple iPhone. I want to ignore the OS/platform discussion, and the monthly service costs, at least for a moment, and look at the move from a functionality standpoint, because it's yet another example of the trade-offs that we need to make while working with the technologies of our choosing.
Before I used the Treo, I'd had a some type of Palm device for many years. It held simple notes in memo files, and larger, illustrated documents were composed and viewed with TealDoc and TealPaint (from TealPoint software). I could 'beam' (InfraRed data transfer) information to other nearby folks with Palm devices to share data, or send the files from a desktop computer by email, or post them to a website to share them with folks farther away (in time or space). But I was happy carrying a separate cell phone, because I hadn't seen a really GOOD marriage of the two technologies yet, and I didn't want over-using one device to drain a battery to disable BOTH functions (PDA and Phone).
I even received a Treo as a gift, but I gave it away. Then, the recipient found a good VPN client, and an SSH tool tat used the data service of the phone...THAT was a feature I would find VERY useful! Later, I found the BlueConsole devices, and the TriConnect software, making the tool even more valuable. I used the Treo for two years.
The iPhone doesn't have a good terminal emulator (so far) that will let me talk to the BlueConsole devices. I can't find a good SSH tool, either. So, you might ask, "why would you want to switch?"
Because, for over a decade, I've wanted a simple, fast way to check email, AND the web. Initially, I wanted this at home, since I usually had fast access at work. But there were sometimes moments when I was away from home when I'd remember I wanted to send email to someone, or I wanted to look up something on the web. When this happened, I wrote an appointment on my Palm, to remind me to send an email, or check something. I'd have to set the time for the next time I thought I'd be near an operating computer, which was sometimes a couple days.
In some of my earlier days, I'd used X-terminals, and other thin-client devices, which booted quickly. But I don't have any of that hardware in the lab. I'd thought about using a simple terminal, and a console server with an "auto-telnet" feature, but that would only give me a quick access to command-line email. All of the options were lacking.
So, the iPhone uses the phone (AT&T Edge network) and WiFi for data connections. It easily remembers WiFi connections where you need to authenticate, and uses the phone when WiFi isn't available. The Safari browser isn't "perfect", but it is so much better than the Treo had been. And the built-in email was now reasonable to us, because I had gigabytes of fast "in-phone" storage. (Many Palm apps didn't play well if you were running them from the SD card.)
So, I have retired the Treo to being a BlueConsole terminal and file repository, since it does those better than the iPhone. But I don't carry the Treo on my belt anymore.
Now that I've used the iPhone, I've decided it was the RIGHT answer for me. I'm REALLY glad the phone doesn't tell folks where I was when I've composed/sent my email. But reading my email has replaced reading my daily CBC/BBC world news (I used to get them via AvantGo, since browsing the web wasn't practical on the Treo.) And, because I'm reading my personal email during the day, it has given me back some time in the evenings. And, being able to send and respond to email more quickly has also sped up some communications exchanges.
The web browsing aspect has also worked out very well, because of that finger-pinching technology that allows you to shrink or expand your view. I need reading glasses now, and some web pages are really fond of the H5 and H6 'fine print' header styles! It's not a problem for me anymore.
I've also put some songs on it, and left my iPod at home. And I also put some pictures on it. Oh, and the built-in camera is pretty good, too. But these weren't really on my radar for making the decision to switch.
I miss having an SSH application in my pocket, and I hope I'll get a terminal emulator that speaks Bluetooth someday, but I've grown too attached to my email and browser to go back to the Treo. The chances are good that applications will come in the future.
We all have to make trade-offs. Here's hoping you are happy with the trade-offs that you need to make.
Before I used the Treo, I'd had a some type of Palm device for many years. It held simple notes in memo files, and larger, illustrated documents were composed and viewed with TealDoc and TealPaint (from TealPoint software). I could 'beam' (InfraRed data transfer) information to other nearby folks with Palm devices to share data, or send the files from a desktop computer by email, or post them to a website to share them with folks farther away (in time or space). But I was happy carrying a separate cell phone, because I hadn't seen a really GOOD marriage of the two technologies yet, and I didn't want over-using one device to drain a battery to disable BOTH functions (PDA and Phone).
I even received a Treo as a gift, but I gave it away. Then, the recipient found a good VPN client, and an SSH tool tat used the data service of the phone...THAT was a feature I would find VERY useful! Later, I found the BlueConsole devices, and the TriConnect software, making the tool even more valuable. I used the Treo for two years.
The iPhone doesn't have a good terminal emulator (so far) that will let me talk to the BlueConsole devices. I can't find a good SSH tool, either. So, you might ask, "why would you want to switch?"
Because, for over a decade, I've wanted a simple, fast way to check email, AND the web. Initially, I wanted this at home, since I usually had fast access at work. But there were sometimes moments when I was away from home when I'd remember I wanted to send email to someone, or I wanted to look up something on the web. When this happened, I wrote an appointment on my Palm, to remind me to send an email, or check something. I'd have to set the time for the next time I thought I'd be near an operating computer, which was sometimes a couple days.
In some of my earlier days, I'd used X-terminals, and other thin-client devices, which booted quickly. But I don't have any of that hardware in the lab. I'd thought about using a simple terminal, and a console server with an "auto-telnet" feature, but that would only give me a quick access to command-line email. All of the options were lacking.
So, the iPhone uses the phone (AT&T Edge network) and WiFi for data connections. It easily remembers WiFi connections where you need to authenticate, and uses the phone when WiFi isn't available. The Safari browser isn't "perfect", but it is so much better than the Treo had been. And the built-in email was now reasonable to us, because I had gigabytes of fast "in-phone" storage. (Many Palm apps didn't play well if you were running them from the SD card.)
So, I have retired the Treo to being a BlueConsole terminal and file repository, since it does those better than the iPhone. But I don't carry the Treo on my belt anymore.
Now that I've used the iPhone, I've decided it was the RIGHT answer for me. I'm REALLY glad the phone doesn't tell folks where I was when I've composed/sent my email. But reading my email has replaced reading my daily CBC/BBC world news (I used to get them via AvantGo, since browsing the web wasn't practical on the Treo.) And, because I'm reading my personal email during the day, it has given me back some time in the evenings. And, being able to send and respond to email more quickly has also sped up some communications exchanges.
The web browsing aspect has also worked out very well, because of that finger-pinching technology that allows you to shrink or expand your view. I need reading glasses now, and some web pages are really fond of the H5 and H6 'fine print' header styles! It's not a problem for me anymore.
I've also put some songs on it, and left my iPod at home. And I also put some pictures on it. Oh, and the built-in camera is pretty good, too. But these weren't really on my radar for making the decision to switch.
I miss having an SSH application in my pocket, and I hope I'll get a terminal emulator that speaks Bluetooth someday, but I've grown too attached to my email and browser to go back to the Treo. The chances are good that applications will come in the future.
We all have to make trade-offs. Here's hoping you are happy with the trade-offs that you need to make.
20080106
Bluetooth console adapters, short time only
I thought I had mentioned these before, but I can't find them by keywords, so here we go.
If you know about the BlueConsole2 bluetooth serial adapters, and you wanted to buy any, order them NOW, since the manufacturer is closing his doors this month (JAN 2008). The ordering page on his site is already closed, but he still has stock, and is trying to make deals to move the last few hundred.
I waited 6 months, trying to get someone else to buy them, so I could try them. OK, now I'm the guy that bought them. They are a bit pricey, but the are clearly the best implementation in this niche product space.
I'm using mine with the TriDone TriConnect software on my Palm Treo 650. The free version of the software throws an extra carriage return (so I'll probably upgrade to the Pro version), but it's a great console with scrollback for watching a device start up, and doing some CLI configuration.)
There are plenty of times where I don't have a laptop handy when I might want one. This tool is an investment in my time. When I need a terminal quickly, unplanned, I'll be able to use the Treo and the BlueConsole.
If you want to save your time, then don't waste any time. Get yours today, and save yourself the prie of trying to buy one on ebay later.
-Z-
If you know about the BlueConsole2 bluetooth serial adapters, and you wanted to buy any, order them NOW, since the manufacturer is closing his doors this month (JAN 2008). The ordering page on his site is already closed, but he still has stock, and is trying to make deals to move the last few hundred.
I waited 6 months, trying to get someone else to buy them, so I could try them. OK, now I'm the guy that bought them. They are a bit pricey, but the are clearly the best implementation in this niche product space.
I'm using mine with the TriDone TriConnect software on my Palm Treo 650. The free version of the software throws an extra carriage return (so I'll probably upgrade to the Pro version), but it's a great console with scrollback for watching a device start up, and doing some CLI configuration.)
There are plenty of times where I don't have a laptop handy when I might want one. This tool is an investment in my time. When I need a terminal quickly, unplanned, I'll be able to use the Treo and the BlueConsole.
If you want to save your time, then don't waste any time. Get yours today, and save yourself the prie of trying to buy one on ebay later.
-Z-
20080103
It's OK to talk back...
With the new year came some new toys, but I still have the same old Doctor's Bag, so I needed to find out what needed to stay, and what could be combined or made smaller.
One of the relatively large things were a couple of DB25 Loopback modules, from Cyclades (one included with each server, so I have a bunch). To use this useful tool, I also have a collection of DB25 adapters for a few of the console servers I often work with.
I'd tried using some simple loopback plugs...just short wires into an RJ45 plug (sometimes called an 'ice cube') like the one you also get with a new Cyclades console server...but these get lost in places, sometimes even lost in console server ports if bigger gear is mounted above and below the console server, so I gave up on them.
My next attempt was a 'pigtail'...a 4"-6" section of cable into the 'ice cube'. This was a 'flag', to help me find it sticking out of the console server, but I still had the trouble getting these out of console servers which had been mounted in tight corners.
All the while, I still needed a way to check the signal at the other end of the cable. My real concern was usually "do I have good connectivity up to the back of the machine?", and this required a loopback at the device in question, so I started packing the loopback modules.
I'd thought about taking an RJ45 socket out of a patch-panel, and looping wires around in that. It would be smaller and lighter, but how do I label them. I thought about using color, like the hood colors on the console server adapters. (As baby boomers get older, color coding lets one get by without reading glasses, at least some of the time.) I'd tried a variety of discarded sockets, with some success, but they all had flaws.
I finally stumbled on a part that I like for the purpose! Panduit's Minicom line of CAT-3 patch panel parts. You can get them in the common resistor color-code colors, they are compact, rugged, you can buy them in small quantites, and they have a flat spot you can use for labeling the jack. I'll try to get a couple pictures on Flickr, so I can link them here (and update my Doctor's Bag page). The only real modification to make is to round the sharp corners with some sandpaper, so they don't cause problems in the bag.
-Z-
One of the relatively large things were a couple of DB25 Loopback modules, from Cyclades (one included with each server, so I have a bunch). To use this useful tool, I also have a collection of DB25 adapters for a few of the console servers I often work with.
I'd tried using some simple loopback plugs...just short wires into an RJ45 plug (sometimes called an 'ice cube') like the one you also get with a new Cyclades console server...but these get lost in places, sometimes even lost in console server ports if bigger gear is mounted above and below the console server, so I gave up on them.
My next attempt was a 'pigtail'...a 4"-6" section of cable into the 'ice cube'. This was a 'flag', to help me find it sticking out of the console server, but I still had the trouble getting these out of console servers which had been mounted in tight corners.
All the while, I still needed a way to check the signal at the other end of the cable. My real concern was usually "do I have good connectivity up to the back of the machine?", and this required a loopback at the device in question, so I started packing the loopback modules.
I'd thought about taking an RJ45 socket out of a patch-panel, and looping wires around in that. It would be smaller and lighter, but how do I label them. I thought about using color, like the hood colors on the console server adapters. (As baby boomers get older, color coding lets one get by without reading glasses, at least some of the time.) I'd tried a variety of discarded sockets, with some success, but they all had flaws.
I finally stumbled on a part that I like for the purpose! Panduit's Minicom line of CAT-3 patch panel parts. You can get them in the common resistor color-code colors, they are compact, rugged, you can buy them in small quantites, and they have a flat spot you can use for labeling the jack. I'll try to get a couple pictures on Flickr, so I can link them here (and update my Doctor's Bag page). The only real modification to make is to round the sharp corners with some sandpaper, so they don't cause problems in the bag.
-Z-
Subscribe to:
Posts (Atom)