The longer I go without this capability, the stronger my desire for an application for the iPhone to fill my needs. But, I don't want to hack my phone...I want a real iPhone App, and now it's worth money to me. I can't find a 'Wish List' page that hasn't been taken over by spammers. So, I'll try the Open Letter approach.
I have two needs, with a common root...both need a basic vt-100 terminal emulator underneath. ANSI color would be a nice touch, but isn't necessary. I don't need Wyse, or full-blown TN-3270 emulation...keep it simple. Some scrollback buffer would be useful (say ~2600 lines), and the ability to capture to a log file (and email the log later?) would be very handy. So, what else?
The primary need is to have an SSH v2 client, so I can SSH directly to my machines using the Edge or wifi connections. This also probably means solving the pre-shared keys issue...how do I get the keys onto the iPhone? I still can't sync a simple file to my phone. Maybe I need a key-generator on the phone, but I still need to get the public key to the hosts.
The secondary need is to communicate with a Bluetooth serial device. Specifically, a BlueConsole dongle, although there are other devices with which you should be able to interoperate. (I used to use the Tri-connect package (from Tridone) on my Treo 650. It wasn't great, but it was a good option, which let me do a lot of quick, little jobs without lugging a computer around.) I'm sorely tempted to carry the Treo around in my tool bag, for those occasions when I'd really like to use these serial devices...if I could find a good way to keep the Treo battery charged while it's in the tool bag, I'd do it. But, I'd much rather use the iPhone for this task.
If you're an iPhone developer, working on a terminal emulator that will use the Bluetooth port, let's talk! I'm willing to be a software tester, give feedback, and help with documentation. I've got good credentials in both the serial console and the communications software test realms. We may also be able to work out a loan of a BlueConsole device while you're developing.
If you're an iPhone developer, working on an SSH v2 client, I want to talk to you as well, and I'm willing to be a tester for this as well. (I'd love to know that someone is trying to fulfill this need.) My favorite SSH app is PuTTY (by Simon Tatham and others), if you want to set your sites high. If you are trying to determine what folks 'want', versus what the 'need' (and would be willing to pay for), I'd like to talk to you.
Since Father's Day is in June, I'm going to post information about some of my favorite new tools in my next posts, in case you are shopping for special gifts. (Yes, the iPhone is on the list.)
-Z-
20080513
What is 'Normal', anyway?
Yow! It's already well into May, and April got by without a post. It was a busy month, with a LOT of discovery tasks happening. I'll try to get a few more posts in this month to summarize. But first, I wanted to get this thought online...
Those who know me are not surprised if the come into a dark wiring closet, and find me sitting on a milk crate, staring at the 'blinkly lights'. (Frankly, I've startled plenty of people who don't know me this way!) Most just write it of as a fixation on blinky lights, but it is not. This is how I can 'visually spot trends' of activity on devices and interfaces. It's also a chance to remember which link status indicators are on, off, in an alarm condition. And, I'm also listening... to fans, air conditioner inlets, hard drive spindle bearings, modem speakers, relays clicking.
This is how I get to know what "Normal" looks like, sounds like, and smells like in the data center and wiring closets. (Yes, smell... does it 'usually' smell damp in this room? Is the 'ozone' smell something that's always here, or does it indicate a component failure. Scent is a strong trigger for memory!)
Knowing what is 'Normal' helps us spot what is unusual. When I have a network failure, I can go to the associated wiring closet and look, listen, and smell...I don't need to ponder "has that always been like this?", because I'll remember. "That light is usually blinking...so there isn't traffic on that interface!" Knowing what is normal is a key to fast troubleshooting.
The RRD tool has been a great resource for graphing monitoring data, allowing you to visualize 'normal', to see 'now', and to identify trends. You find this under MANY open source tools, such as MRTG, Cricket, Cacti, and many more.
But, can you tell what's 'normal' with a serial console? Yes, you can! The key is, you need to LOOK when things are operating normally. Look when the system/device/network is idle some night or weekend. Look again when backups are running. Look again when the network is busy, but not failing. Then, compare your notes, or, your LOGS!
You can do a LOT interactively, using a simple terminal emulator and a cable. You have a lot more flexibility with a Console Server and multiple Telnet sessions (for example, you can monitor many consoles simultaneously, and cut-and-paste between them). The real benefits are had when you combine the console servers with a Serial Console Management Application such as Conserver, or ConsoleWorks, and you can compare historic data with today's results.
I have a handful of devices which have "diagnostics' ports, that only the field engineers will use. They are not for normal use by customers. However, when you connect to these ports, you can find some of the devices are 'beaconing' about events, or reporting regular status messages. Even if the port doesn't say much, if you hit a carriage return, you'll probably get a prompt...doing this occasionally will tell you it's still alive. And, sometimes, you can also leverage the simple diagnostics that are built in. It's good to have a baseline log of what the devices are reporting to these diagnostics when they are healthy, so you can compare them if the device starts misbehaving.
But, to do any of these things, you need to start by looking when things are working as expected, doing their job the way you want them to. That's when you want to get your first look, and save that data for later comparison. (Speaking of comparing log data, try SPLUNK! Check out splunkbase, and the SPLUNK Forums! Splunk is worth a few blog articles by itself...later.)
The bottom line: Do you KNOW what you are missing? Do you WANT to know? Then LOOK!
-Z-
Those who know me are not surprised if the come into a dark wiring closet, and find me sitting on a milk crate, staring at the 'blinkly lights'. (Frankly, I've startled plenty of people who don't know me this way!) Most just write it of as a fixation on blinky lights, but it is not. This is how I can 'visually spot trends' of activity on devices and interfaces. It's also a chance to remember which link status indicators are on, off, in an alarm condition. And, I'm also listening... to fans, air conditioner inlets, hard drive spindle bearings, modem speakers, relays clicking.
This is how I get to know what "Normal" looks like, sounds like, and smells like in the data center and wiring closets. (Yes, smell... does it 'usually' smell damp in this room? Is the 'ozone' smell something that's always here, or does it indicate a component failure. Scent is a strong trigger for memory!)
Knowing what is 'Normal' helps us spot what is unusual. When I have a network failure, I can go to the associated wiring closet and look, listen, and smell...I don't need to ponder "has that always been like this?", because I'll remember. "That light is usually blinking...so there isn't traffic on that interface!" Knowing what is normal is a key to fast troubleshooting.
The RRD tool has been a great resource for graphing monitoring data, allowing you to visualize 'normal', to see 'now', and to identify trends. You find this under MANY open source tools, such as MRTG, Cricket, Cacti, and many more.
But, can you tell what's 'normal' with a serial console? Yes, you can! The key is, you need to LOOK when things are operating normally. Look when the system/device/network is idle some night or weekend. Look again when backups are running. Look again when the network is busy, but not failing. Then, compare your notes, or, your LOGS!
You can do a LOT interactively, using a simple terminal emulator and a cable. You have a lot more flexibility with a Console Server and multiple Telnet sessions (for example, you can monitor many consoles simultaneously, and cut-and-paste between them). The real benefits are had when you combine the console servers with a Serial Console Management Application such as Conserver, or ConsoleWorks, and you can compare historic data with today's results.
I have a handful of devices which have "diagnostics' ports, that only the field engineers will use. They are not for normal use by customers. However, when you connect to these ports, you can find some of the devices are 'beaconing' about events, or reporting regular status messages. Even if the port doesn't say much, if you hit a carriage return, you'll probably get a prompt...doing this occasionally will tell you it's still alive. And, sometimes, you can also leverage the simple diagnostics that are built in. It's good to have a baseline log of what the devices are reporting to these diagnostics when they are healthy, so you can compare them if the device starts misbehaving.
But, to do any of these things, you need to start by looking when things are working as expected, doing their job the way you want them to. That's when you want to get your first look, and save that data for later comparison. (Speaking of comparing log data, try SPLUNK! Check out splunkbase, and the SPLUNK Forums! Splunk is worth a few blog articles by itself...later.)
The bottom line: Do you KNOW what you are missing? Do you WANT to know? Then LOOK!
-Z-
20080331
One moment please, I'll connect you...
Today's topic has been a recurring theme during the past few weeks.
On the one hand, while I'm in love with my iPhone for email and web access pretty much everywhere, I still really miss having an SSH client on my hip. (On my Treo, the pssh client did the trick. It was simple, talked SSH v2, had a poor random number generator, but could import strong pre-shared keys from a Palm memo.) I'm hoping that a simple, cool new SSH terminal app for the iPhone is just around the corner, now that the iPhone Software Developer Kit is out. I don't need fancy key-mapping, or a variety of terminal emulations. If it can offer to let me talk to Bluetooth devices as well as the Edge/WiFi connections (so I can use my Blue Console devices, it will be worth more to me. (That's a blatant plea/hint to any iPhone developers. Hopefully I've said iPhone enough times for this article to turn up in a web search. :-)
On the other hand, I've found the Service Processor on some SUN Opteron boxes to be a bit vexing. Many hosts have these Service Processors (S.P. from here on) on-board now, and each vendor is implementing it a bit differently. It's an added 'black box' on the communications line between the physical serial port and the serial I/O (tty) port on the host.
When BIOS Redirection was added, this was the first complication making serial ports provide the GETTY console login under a Linux host. Now these processors, supporting ILOM/ALOM/e-i-e-i-o communications are also listening to the line, looking for meta-sequences, waiting to interrupt the conversation and do your bidding. One gotcha is, when you start talking to the S.P., you lose any logging from the Operating System. (Why don't these special purpose computers BUFFER at least SOME amount of host data?)
The odd hang-up with the SUN units I'm wrestling seems to be an inactivity timeout. If I get into the S.P., and let it sit for a while, it seems to go to sleep, and I can't wake it up again. BUT, it never returned the connection back to the OS. Essentially, the O.S. console connection is now bottled-up, and I can't get the S.P. to answer through the serial port, or to relinquish the port back to the O.S., leaving me to reboot the system in order to get the serial console back. There MUST be a better way.
Fortunately, there is a good write-up on the SUN S.P., and I'm going to go over it this weekend. Hopefully I'll find some variables to change the meta characters which trigger the S.P., since they are too close to another sequence I need to send, and I also hope to find some timer settings to get the S.P. to drop out politely if I time out of the session.
-Z-
On the one hand, while I'm in love with my iPhone for email and web access pretty much everywhere, I still really miss having an SSH client on my hip. (On my Treo, the pssh client did the trick. It was simple, talked SSH v2, had a poor random number generator, but could import strong pre-shared keys from a Palm memo.) I'm hoping that a simple, cool new SSH terminal app for the iPhone is just around the corner, now that the iPhone Software Developer Kit is out. I don't need fancy key-mapping, or a variety of terminal emulations. If it can offer to let me talk to Bluetooth devices as well as the Edge/WiFi connections (so I can use my Blue Console devices, it will be worth more to me. (That's a blatant plea/hint to any iPhone developers. Hopefully I've said iPhone enough times for this article to turn up in a web search. :-)
On the other hand, I've found the Service Processor on some SUN Opteron boxes to be a bit vexing. Many hosts have these Service Processors (S.P. from here on) on-board now, and each vendor is implementing it a bit differently. It's an added 'black box' on the communications line between the physical serial port and the serial I/O (tty) port on the host.
When BIOS Redirection was added, this was the first complication making serial ports provide the GETTY console login under a Linux host. Now these processors, supporting ILOM/ALOM/e-i-e-i-o communications are also listening to the line, looking for meta-sequences, waiting to interrupt the conversation and do your bidding. One gotcha is, when you start talking to the S.P., you lose any logging from the Operating System. (Why don't these special purpose computers BUFFER at least SOME amount of host data?)
The odd hang-up with the SUN units I'm wrestling seems to be an inactivity timeout. If I get into the S.P., and let it sit for a while, it seems to go to sleep, and I can't wake it up again. BUT, it never returned the connection back to the OS. Essentially, the O.S. console connection is now bottled-up, and I can't get the S.P. to answer through the serial port, or to relinquish the port back to the O.S., leaving me to reboot the system in order to get the serial console back. There MUST be a better way.
Fortunately, there is a good write-up on the SUN S.P., and I'm going to go over it this weekend. Hopefully I'll find some variables to change the meta characters which trigger the S.P., since they are too close to another sequence I need to send, and I also hope to find some timer settings to get the S.P. to drop out politely if I time out of the session.
-Z-
20080301
Tools and trade-offs...
Recent circumstances gave me the opportunity to switch my phone service from a Palm Treo 650 to an Apple iPhone. I want to ignore the OS/platform discussion, and the monthly service costs, at least for a moment, and look at the move from a functionality standpoint, because it's yet another example of the trade-offs that we need to make while working with the technologies of our choosing.
Before I used the Treo, I'd had a some type of Palm device for many years. It held simple notes in memo files, and larger, illustrated documents were composed and viewed with TealDoc and TealPaint (from TealPoint software). I could 'beam' (InfraRed data transfer) information to other nearby folks with Palm devices to share data, or send the files from a desktop computer by email, or post them to a website to share them with folks farther away (in time or space). But I was happy carrying a separate cell phone, because I hadn't seen a really GOOD marriage of the two technologies yet, and I didn't want over-using one device to drain a battery to disable BOTH functions (PDA and Phone).
I even received a Treo as a gift, but I gave it away. Then, the recipient found a good VPN client, and an SSH tool tat used the data service of the phone...THAT was a feature I would find VERY useful! Later, I found the BlueConsole devices, and the TriConnect software, making the tool even more valuable. I used the Treo for two years.
The iPhone doesn't have a good terminal emulator (so far) that will let me talk to the BlueConsole devices. I can't find a good SSH tool, either. So, you might ask, "why would you want to switch?"
Because, for over a decade, I've wanted a simple, fast way to check email, AND the web. Initially, I wanted this at home, since I usually had fast access at work. But there were sometimes moments when I was away from home when I'd remember I wanted to send email to someone, or I wanted to look up something on the web. When this happened, I wrote an appointment on my Palm, to remind me to send an email, or check something. I'd have to set the time for the next time I thought I'd be near an operating computer, which was sometimes a couple days.
In some of my earlier days, I'd used X-terminals, and other thin-client devices, which booted quickly. But I don't have any of that hardware in the lab. I'd thought about using a simple terminal, and a console server with an "auto-telnet" feature, but that would only give me a quick access to command-line email. All of the options were lacking.
So, the iPhone uses the phone (AT&T Edge network) and WiFi for data connections. It easily remembers WiFi connections where you need to authenticate, and uses the phone when WiFi isn't available. The Safari browser isn't "perfect", but it is so much better than the Treo had been. And the built-in email was now reasonable to us, because I had gigabytes of fast "in-phone" storage. (Many Palm apps didn't play well if you were running them from the SD card.)
So, I have retired the Treo to being a BlueConsole terminal and file repository, since it does those better than the iPhone. But I don't carry the Treo on my belt anymore.
Now that I've used the iPhone, I've decided it was the RIGHT answer for me. I'm REALLY glad the phone doesn't tell folks where I was when I've composed/sent my email. But reading my email has replaced reading my daily CBC/BBC world news (I used to get them via AvantGo, since browsing the web wasn't practical on the Treo.) And, because I'm reading my personal email during the day, it has given me back some time in the evenings. And, being able to send and respond to email more quickly has also sped up some communications exchanges.
The web browsing aspect has also worked out very well, because of that finger-pinching technology that allows you to shrink or expand your view. I need reading glasses now, and some web pages are really fond of the H5 and H6 'fine print' header styles! It's not a problem for me anymore.
I've also put some songs on it, and left my iPod at home. And I also put some pictures on it. Oh, and the built-in camera is pretty good, too. But these weren't really on my radar for making the decision to switch.
I miss having an SSH application in my pocket, and I hope I'll get a terminal emulator that speaks Bluetooth someday, but I've grown too attached to my email and browser to go back to the Treo. The chances are good that applications will come in the future.
We all have to make trade-offs. Here's hoping you are happy with the trade-offs that you need to make.
Before I used the Treo, I'd had a some type of Palm device for many years. It held simple notes in memo files, and larger, illustrated documents were composed and viewed with TealDoc and TealPaint (from TealPoint software). I could 'beam' (InfraRed data transfer) information to other nearby folks with Palm devices to share data, or send the files from a desktop computer by email, or post them to a website to share them with folks farther away (in time or space). But I was happy carrying a separate cell phone, because I hadn't seen a really GOOD marriage of the two technologies yet, and I didn't want over-using one device to drain a battery to disable BOTH functions (PDA and Phone).
I even received a Treo as a gift, but I gave it away. Then, the recipient found a good VPN client, and an SSH tool tat used the data service of the phone...THAT was a feature I would find VERY useful! Later, I found the BlueConsole devices, and the TriConnect software, making the tool even more valuable. I used the Treo for two years.
The iPhone doesn't have a good terminal emulator (so far) that will let me talk to the BlueConsole devices. I can't find a good SSH tool, either. So, you might ask, "why would you want to switch?"
Because, for over a decade, I've wanted a simple, fast way to check email, AND the web. Initially, I wanted this at home, since I usually had fast access at work. But there were sometimes moments when I was away from home when I'd remember I wanted to send email to someone, or I wanted to look up something on the web. When this happened, I wrote an appointment on my Palm, to remind me to send an email, or check something. I'd have to set the time for the next time I thought I'd be near an operating computer, which was sometimes a couple days.
In some of my earlier days, I'd used X-terminals, and other thin-client devices, which booted quickly. But I don't have any of that hardware in the lab. I'd thought about using a simple terminal, and a console server with an "auto-telnet" feature, but that would only give me a quick access to command-line email. All of the options were lacking.
So, the iPhone uses the phone (AT&T Edge network) and WiFi for data connections. It easily remembers WiFi connections where you need to authenticate, and uses the phone when WiFi isn't available. The Safari browser isn't "perfect", but it is so much better than the Treo had been. And the built-in email was now reasonable to us, because I had gigabytes of fast "in-phone" storage. (Many Palm apps didn't play well if you were running them from the SD card.)
So, I have retired the Treo to being a BlueConsole terminal and file repository, since it does those better than the iPhone. But I don't carry the Treo on my belt anymore.
Now that I've used the iPhone, I've decided it was the RIGHT answer for me. I'm REALLY glad the phone doesn't tell folks where I was when I've composed/sent my email. But reading my email has replaced reading my daily CBC/BBC world news (I used to get them via AvantGo, since browsing the web wasn't practical on the Treo.) And, because I'm reading my personal email during the day, it has given me back some time in the evenings. And, being able to send and respond to email more quickly has also sped up some communications exchanges.
The web browsing aspect has also worked out very well, because of that finger-pinching technology that allows you to shrink or expand your view. I need reading glasses now, and some web pages are really fond of the H5 and H6 'fine print' header styles! It's not a problem for me anymore.
I've also put some songs on it, and left my iPod at home. And I also put some pictures on it. Oh, and the built-in camera is pretty good, too. But these weren't really on my radar for making the decision to switch.
I miss having an SSH application in my pocket, and I hope I'll get a terminal emulator that speaks Bluetooth someday, but I've grown too attached to my email and browser to go back to the Treo. The chances are good that applications will come in the future.
We all have to make trade-offs. Here's hoping you are happy with the trade-offs that you need to make.
20080106
Bluetooth console adapters, short time only
I thought I had mentioned these before, but I can't find them by keywords, so here we go.
If you know about the BlueConsole2 bluetooth serial adapters, and you wanted to buy any, order them NOW, since the manufacturer is closing his doors this month (JAN 2008). The ordering page on his site is already closed, but he still has stock, and is trying to make deals to move the last few hundred.
I waited 6 months, trying to get someone else to buy them, so I could try them. OK, now I'm the guy that bought them. They are a bit pricey, but the are clearly the best implementation in this niche product space.
I'm using mine with the TriDone TriConnect software on my Palm Treo 650. The free version of the software throws an extra carriage return (so I'll probably upgrade to the Pro version), but it's a great console with scrollback for watching a device start up, and doing some CLI configuration.)
There are plenty of times where I don't have a laptop handy when I might want one. This tool is an investment in my time. When I need a terminal quickly, unplanned, I'll be able to use the Treo and the BlueConsole.
If you want to save your time, then don't waste any time. Get yours today, and save yourself the prie of trying to buy one on ebay later.
-Z-
If you know about the BlueConsole2 bluetooth serial adapters, and you wanted to buy any, order them NOW, since the manufacturer is closing his doors this month (JAN 2008). The ordering page on his site is already closed, but he still has stock, and is trying to make deals to move the last few hundred.
I waited 6 months, trying to get someone else to buy them, so I could try them. OK, now I'm the guy that bought them. They are a bit pricey, but the are clearly the best implementation in this niche product space.
I'm using mine with the TriDone TriConnect software on my Palm Treo 650. The free version of the software throws an extra carriage return (so I'll probably upgrade to the Pro version), but it's a great console with scrollback for watching a device start up, and doing some CLI configuration.)
There are plenty of times where I don't have a laptop handy when I might want one. This tool is an investment in my time. When I need a terminal quickly, unplanned, I'll be able to use the Treo and the BlueConsole.
If you want to save your time, then don't waste any time. Get yours today, and save yourself the prie of trying to buy one on ebay later.
-Z-
20080103
It's OK to talk back...
With the new year came some new toys, but I still have the same old Doctor's Bag, so I needed to find out what needed to stay, and what could be combined or made smaller.
One of the relatively large things were a couple of DB25 Loopback modules, from Cyclades (one included with each server, so I have a bunch). To use this useful tool, I also have a collection of DB25 adapters for a few of the console servers I often work with.
I'd tried using some simple loopback plugs...just short wires into an RJ45 plug (sometimes called an 'ice cube') like the one you also get with a new Cyclades console server...but these get lost in places, sometimes even lost in console server ports if bigger gear is mounted above and below the console server, so I gave up on them.
My next attempt was a 'pigtail'...a 4"-6" section of cable into the 'ice cube'. This was a 'flag', to help me find it sticking out of the console server, but I still had the trouble getting these out of console servers which had been mounted in tight corners.
All the while, I still needed a way to check the signal at the other end of the cable. My real concern was usually "do I have good connectivity up to the back of the machine?", and this required a loopback at the device in question, so I started packing the loopback modules.
I'd thought about taking an RJ45 socket out of a patch-panel, and looping wires around in that. It would be smaller and lighter, but how do I label them. I thought about using color, like the hood colors on the console server adapters. (As baby boomers get older, color coding lets one get by without reading glasses, at least some of the time.) I'd tried a variety of discarded sockets, with some success, but they all had flaws.
I finally stumbled on a part that I like for the purpose! Panduit's Minicom line of CAT-3 patch panel parts. You can get them in the common resistor color-code colors, they are compact, rugged, you can buy them in small quantites, and they have a flat spot you can use for labeling the jack. I'll try to get a couple pictures on Flickr, so I can link them here (and update my Doctor's Bag page). The only real modification to make is to round the sharp corners with some sandpaper, so they don't cause problems in the bag.
-Z-
One of the relatively large things were a couple of DB25 Loopback modules, from Cyclades (one included with each server, so I have a bunch). To use this useful tool, I also have a collection of DB25 adapters for a few of the console servers I often work with.
I'd tried using some simple loopback plugs...just short wires into an RJ45 plug (sometimes called an 'ice cube') like the one you also get with a new Cyclades console server...but these get lost in places, sometimes even lost in console server ports if bigger gear is mounted above and below the console server, so I gave up on them.
My next attempt was a 'pigtail'...a 4"-6" section of cable into the 'ice cube'. This was a 'flag', to help me find it sticking out of the console server, but I still had the trouble getting these out of console servers which had been mounted in tight corners.
All the while, I still needed a way to check the signal at the other end of the cable. My real concern was usually "do I have good connectivity up to the back of the machine?", and this required a loopback at the device in question, so I started packing the loopback modules.
I'd thought about taking an RJ45 socket out of a patch-panel, and looping wires around in that. It would be smaller and lighter, but how do I label them. I thought about using color, like the hood colors on the console server adapters. (As baby boomers get older, color coding lets one get by without reading glasses, at least some of the time.) I'd tried a variety of discarded sockets, with some success, but they all had flaws.
I finally stumbled on a part that I like for the purpose! Panduit's Minicom line of CAT-3 patch panel parts. You can get them in the common resistor color-code colors, they are compact, rugged, you can buy them in small quantites, and they have a flat spot you can use for labeling the jack. I'll try to get a couple pictures on Flickr, so I can link them here (and update my Doctor's Bag page). The only real modification to make is to round the sharp corners with some sandpaper, so they don't cause problems in the bag.
-Z-
20071230
Happy New Year! It's about time...
Time for a New Years Resolution! Establish an NTP server at every site I work on, and point as many devices to it as I can. (I'll do this to save everyone some time.)
Consider taking a look around your installation. How long would it take to check the uptimes of the devices, and also check their on-board Real Time Clocks. How many do you think are correctly showing the current time, within +/- 10 minutes? For your network gear, how many are showing log timestamps as the "from startup" uptime stamp, instead of a date/time notation?
Now, consider that, in the time it would take to EVALUATE your clocks and timestamp settings, that you could probably FIX THEM, not just record the results! If you plan to go and touch all the gear anyway, why not save some time, and SET the clocks, instead of checking them?
But, when was the last time anyone bothered to do this, in your installation? Maybe the clocks on your devices only get set when each device is installed. (If you ship devices between data centers, across timezones, do you make a point of changing the device clocks when you receive them into your installation?) Let's face it, these clocks are cheap, and they ALL have some drift in them. Even if the drift is minimal, the error is cumulative, and can be significant across 3 or more years. You complicate the drift when you don't use a stable starting point. (What time reference does everyone use when they set device clocks? Probably their independent wrist watch?)
In the same way that you need to spend some money to make more money, you need to spend time to make more time! NTP is a great way to set clocks, and keep them close to the current time. (Even a daily time sync will all but eliminate the effect of drift on the cheap clocks.) Most devices have a no-cost client available, and many servers and some network gear have a no-cost NTP Server option available. If you have a complicated network, with multiple customers or DMZs, you can invest in an appliance that uses GPS or the cellular phne service to provide a drop-in connection on an isolated network. You just need to take some time to set up a server, and then set your devices to be clients.
Technically, all of this work (and invested time setting up NTP) doesn't save time, per se. What it will do is reduce the time you spend tracking down problems. Calculating time-math is hard. Correlating time offsets in log files between devices with unsynchronized clocks takes a LOT of time. But, when you have an "event" in your installation, and you need to find the Root Cause, you need to compare a bunch of logs, to figure out what happend first, then second, and etc.
If you take the time to set up an NTP server, you make it quick and easy to 'set and forget' the clocks on new devices coming into your installation. When you take the time to point all of your existing devices to the NTP server, you'll save time comparing the logs on all future 'events'. It's hard to guess how much time, or how soon you will recover that time, but it WILL happen. Just give it time.
-Zonker-
Some NTP appliance resources I've found to be interesting:
VMware NTP appliance VM info
Symmetricom S200
J-Time (Meinberg USA) Lantime M300
Brandywine Communications NTV-100RG
Consider taking a look around your installation. How long would it take to check the uptimes of the devices, and also check their on-board Real Time Clocks. How many do you think are correctly showing the current time, within +/- 10 minutes? For your network gear, how many are showing log timestamps as the "from startup" uptime stamp, instead of a date/time notation?
Now, consider that, in the time it would take to EVALUATE your clocks and timestamp settings, that you could probably FIX THEM, not just record the results! If you plan to go and touch all the gear anyway, why not save some time, and SET the clocks, instead of checking them?
But, when was the last time anyone bothered to do this, in your installation? Maybe the clocks on your devices only get set when each device is installed. (If you ship devices between data centers, across timezones, do you make a point of changing the device clocks when you receive them into your installation?) Let's face it, these clocks are cheap, and they ALL have some drift in them. Even if the drift is minimal, the error is cumulative, and can be significant across 3 or more years. You complicate the drift when you don't use a stable starting point. (What time reference does everyone use when they set device clocks? Probably their independent wrist watch?)
In the same way that you need to spend some money to make more money, you need to spend time to make more time! NTP is a great way to set clocks, and keep them close to the current time. (Even a daily time sync will all but eliminate the effect of drift on the cheap clocks.) Most devices have a no-cost client available, and many servers and some network gear have a no-cost NTP Server option available. If you have a complicated network, with multiple customers or DMZs, you can invest in an appliance that uses GPS or the cellular phne service to provide a drop-in connection on an isolated network. You just need to take some time to set up a server, and then set your devices to be clients.
Technically, all of this work (and invested time setting up NTP) doesn't save time, per se. What it will do is reduce the time you spend tracking down problems. Calculating time-math is hard. Correlating time offsets in log files between devices with unsynchronized clocks takes a LOT of time. But, when you have an "event" in your installation, and you need to find the Root Cause, you need to compare a bunch of logs, to figure out what happend first, then second, and etc.
If you take the time to set up an NTP server, you make it quick and easy to 'set and forget' the clocks on new devices coming into your installation. When you take the time to point all of your existing devices to the NTP server, you'll save time comparing the logs on all future 'events'. It's hard to guess how much time, or how soon you will recover that time, but it WILL happen. Just give it time.
-Zonker-
Some NTP appliance resources I've found to be interesting:
VMware NTP appliance VM info
Symmetricom S200
J-Time (Meinberg USA) Lantime M300
Brandywine Communications NTV-100RG
Subscribe to:
Posts (Atom)