I'm very much in favor of a symmetrical RJ45 wiring schema for console servers, because it could minimize how many adapters I would need to stock. The advantages of a symmetrical pinout include using a simple 'rolled' cable to make a 'null-modem' connection between devices and adapters.
Now, I'm not picking any particular vendors wiring scheme as 'the best'. I'm espousing symmetry as the 'best way'. The problem is, this means having two Signal Ground pins, which usually means that DCD won't have a pin. (On an 8-wire RJ45 interface, you would have two ground leads, two data leads, two hardware flow control leads, and two hardware handshaking leads.)
Remember that today's Console Server has its roots in yesteryears Terminal Servers. The old TS used modems or terminals, to let remote users connect to hosts across the network. The hardware handshaking was used by the terminals to indicate if they were connected. With modems, they could be connected and powered, but it was DCD signals from the modem to the TS which told the TS whether an interface was 'on-line' with a remote user.
But, these days in console service, modems are a pretty rare thing. Still, I'll be the first to recognize that modems could be used to dial into a remote site and try to connect when the main WAN links are down. However, you can configure most modems to toggle the DSR lead to reflect the status of DCD, and you can configure most console servers to recognize the DSR lead as a signal that the device isn't ready. (So, I don't think we really need DCD in console service. I'll leave Dial-up Networking for later article.)
If you look at my Signals Page, in the RJ45 section, you'll notice that there are already a couple of symmetrical pinout schemes. But you can also see the wide variety of combinations that I've found so far. While pin 4 seems to be the most common pin for signal ground, its clear that not everyone uses pin for for the signal ground. That makes it harder to make an RJ45 signal tracer that works in all cases, so there is less incentive for a vendor to take on that project.
So, for the existing symmetrical RJ45 pinouts on my signals page, you'll notice that they use pins 4 and 5 as ground pins. Data pins are on pins 3 and 6. Handshaking is on pins 2 and 7, and flow control are on pins 1 and 8.
If you're asking "why can't we use pin 5 for DCD?", then you may not have thought about the issue deeply. The first problem comes when you use a rolled cable as a null modem, and you now connect two devices signal leads, but not their signal ground pins! You'll be replying on the chassis ground connections to provide the signal reference. Using these console servers in large data centers means that you may be connecting devices on different phases of the AC circuit, or even across two different Power Distrubtion Units (PDUs) or Uninterruptible Power Supplies (UPSs). The result could be burned out serial ports on your console server and the attached devices, or more sever damage to the units.
So, to summarize the vendors I know about today;
Symmetrical, with 2 grounds (9+ of 26):
Cisco, Logical Solutions SCS, Lantronics SCS-1600, Digi PortServer II, Xyplex 1600/1800, iTouch/MRV in-Reach lines, Juniper Networks (J-series devices, and many NetScreen devices), Server Technology CDU consoles, and Sun Microsystems console/TTY ports
Symmetrical, except for 1 ground and DCD (3 of 26):
Digi CM, Digi STS-1610, Opengear CM
Of the 27 console connections I've documented, 15 use pin 4 for ground, and 9 use pin 6 for ground. (Keep that in mind if you are making your own RJ45 signal tracers.) And of the 9 symmetrical pinouts, eight of them use pins 4 and 5 for ground, and one uses pin6 3 and 6.
It seems to me that other vendors (who are not making console servers) have been choosing the Cisco wiring scheme (Juniper, Sun, Server Technology). Lantronix picked the null-modem compliment to the Cisco pinout. Those choices have been good for me, since it simplifies some of the deployments that I work on. It's interesting to note that the Xyplex/iTouch/MRV pinouts are almost identical, except that pins 1 and 8 (the flow control leads) are inverted from the Cisco schema. (So, if you don't care about flow control, you can mix-and-match these devices with the Cisco-schema cousins.)
I'm under the impression that some vendors are considering adopting a different wiring scheme for their future product series. I'd strongly urge that they consider adopting a symmetrical schema, and I'd also strongly encourage them to consider using an existing schema, rather than trying to come up with a unique schema. And I'm always interested in discussing it in person, though an email on the topic is also welcome.
20070920
Sometimes you want it raw...
Your console connection, that is!
When you are using a console server for remote access to serial consoles, you're defaults are usually a telnet-type listerner on the console server. That means a 7-bit session, with the high-order bit normally being "0". (That's why you sometimes have a session that displays unusual characters...something trips the high-order bit to "1", and you need to reset the session/port to clear the bit.)
But, what if you NEED to pass 8-bit data? Sam Crooks recently had the need to upgrade the OS on a router, but didn't have a the ability to use the network to get the code to the unit. He could, however, use XMODEM-1K from a locally-attached laptop serial port to the console of the unit. Except that his laptop was across the network. The gotcha here was the 8th-bit was set to "1" in much of the data, and certainly in the file transfer sequence numbers. This botches up the connection, and you need to start over again. (I'll explain about the modes first, but I'll also share notes from Sam's solution towards the end of this article.)
Fortunately, most console servers today have legacy roots in "Terminal Server" service, as in dial-in modem pools, and binary file transfers. So, besides their 'default' 7-bit session mode, most have an "8-bit, escapable" mode, and some even have a full "raw 8-bit, non-escapable" mode. You just may have to dig around a bit to find out how to access them. To demonstrate the issues, we'll use Cisco access servers as the example.
With the Cisco devices, to reach across the network to a particular serial port, you normally use a telnet client, to the IP address of the access server, at the TCP port 2000 + n, where n equals the (async) line number. That is, line 21, would be at TCP 2021. This is for the normal, 7-bit session. So far, so good?
If you need to use the 8th bit regularly (specialty keys, alternate characters, etc.), you can get the "escapable" 8-bit mode by using the TCP port 4000 + n. (The same n as before.) The 'escapable' pat means that you still have a couple of the 254 possible characters that you cannot use as data, because these will be used for the client to escape from sending data, so it can send control info (like "quit this session"). Most folks could use the 7-bit mode or the 8-bit mode interchangeably.
But, if you need to send ALL 256 possible characters (think the block sequence numbers, if you think that your data wouldn't have all the characters), then you need to use a "raw 8-bit" mode. But, if you can't escape from the data session (because your escape characters are "just data" now...), how do you close the session? You need to get the device connected to the async port to toggle it's handshake line. (Think of it as a modem call ending, and the modem drops the DCD line, or it cycles the DSR line...because that where that code has it's roots.)
In the Cisco access servers, you can invoke the "raw 8-bit" mode at TCP port 6000 + n.
Here are the notes from Sam's solution, once he discovered these different modes;
The serial speed of 9,600 bps is too slow for big file transfers, so he increases the port speeds to 115,200 bps. (In his case, he uses autobaud on one side, and manually sets the speed on the other. I presume this is so the speeds will resort to 9,600 after everything's finished.)
Next, he sets the port for the data transfer;
Switch#terminal download
Finally, when the transfer completes, he needs to break the session from the far end, so he uses the command clear line n (where n is the line number of the serial port you are using for the transfer).
Cisco also has a good, older page about the various commands related to configuring access servers for modem usage, which has some useful clues as well, including transport in and out, and changing your escape characters. You'll find references to these binary modes on this page, in the section relating to 'rotary'.
You can also find useful information searching for "configuring lines for modem support". For example, you can find a brief summary related to contacting the AUX port on a Cisco DSLAM using these three modes here.
Related Cisco commands (Anyone can search the public Documentation part of the Cisco website, but Cisco support contract holders can find more information if they are logged in.)
-Z-
When you are using a console server for remote access to serial consoles, you're defaults are usually a telnet-type listerner on the console server. That means a 7-bit session, with the high-order bit normally being "0". (That's why you sometimes have a session that displays unusual characters...something trips the high-order bit to "1", and you need to reset the session/port to clear the bit.)
But, what if you NEED to pass 8-bit data? Sam Crooks recently had the need to upgrade the OS on a router, but didn't have a the ability to use the network to get the code to the unit. He could, however, use XMODEM-1K from a locally-attached laptop serial port to the console of the unit. Except that his laptop was across the network. The gotcha here was the 8th-bit was set to "1" in much of the data, and certainly in the file transfer sequence numbers. This botches up the connection, and you need to start over again. (I'll explain about the modes first, but I'll also share notes from Sam's solution towards the end of this article.)
Fortunately, most console servers today have legacy roots in "Terminal Server" service, as in dial-in modem pools, and binary file transfers. So, besides their 'default' 7-bit session mode, most have an "8-bit, escapable" mode, and some even have a full "raw 8-bit, non-escapable" mode. You just may have to dig around a bit to find out how to access them. To demonstrate the issues, we'll use Cisco access servers as the example.
With the Cisco devices, to reach across the network to a particular serial port, you normally use a telnet client, to the IP address of the access server, at the TCP port 2000 + n, where n equals the (async) line number. That is, line 21, would be at TCP 2021. This is for the normal, 7-bit session. So far, so good?
If you need to use the 8th bit regularly (specialty keys, alternate characters, etc.), you can get the "escapable" 8-bit mode by using the TCP port 4000 + n. (The same n as before.) The 'escapable' pat means that you still have a couple of the 254 possible characters that you cannot use as data, because these will be used for the client to escape from sending data, so it can send control info (like "quit this session"). Most folks could use the 7-bit mode or the 8-bit mode interchangeably.
But, if you need to send ALL 256 possible characters (think the block sequence numbers, if you think that your data wouldn't have all the characters), then you need to use a "raw 8-bit" mode. But, if you can't escape from the data session (because your escape characters are "just data" now...), how do you close the session? You need to get the device connected to the async port to toggle it's handshake line. (Think of it as a modem call ending, and the modem drops the DCD line, or it cycles the DSR line...because that where that code has it's roots.)
In the Cisco access servers, you can invoke the "raw 8-bit" mode at TCP port 6000 + n.
Here are the notes from Sam's solution, once he discovered these different modes;
The serial speed of 9,600 bps is too slow for big file transfers, so he increases the port speeds to 115,200 bps. (In his case, he uses autobaud on one side, and manually sets the speed on the other. I presume this is so the speeds will resort to 9,600 after everything's finished.)
Next, he sets the port for the data transfer;
Switch#terminal download
Switch#
Switch#copy xmodem: flash:/c3560-advipservicesk9-mz.122-37.SE1.bin
Destination filename [c3560-advipservicesk9-mz.122-37.SE1.bin]?
Begin the Xmodem or Xmodem-1K transfer now...
CCCCCCCCC
now here, we go to TermTerm's, menus....
File../Transfer->/XMODEM->Send...
File../Transfer->/XMODEM->Send
Pick the file, select 1K at the bottom of the window (vs CRC or checksum)...
and off the file transfer goes....hopefully to completion, and in a reasonable time...
Finally, when the transfer completes, he needs to break the session from the far end, so he uses the command clear line n (where n is the line number of the serial port you are using for the transfer).
Cisco also has a good, older page about the various commands related to configuring access servers for modem usage, which has some useful clues as well, including transport in and out, and changing your escape characters. You'll find references to these binary modes on this page, in the section relating to 'rotary'.
You can also find useful information searching for "configuring lines for modem support". For example, you can find a brief summary related to contacting the AUX port on a Cisco DSLAM using these three modes here.
Related Cisco commands (Anyone can search the public Documentation part of the Cisco website, but Cisco support contract holders can find more information if they are logged in.)
- reverse telnet
- configuring modem line rotary
- telnet break-on-ip
- exec (look for telnet-related BREAK info)
-Z-
20070919
Insert Wire A into Slot B (specialty cables and adapters)
The other title for this article could be "Pocket Notes (you really CAN take it with you)". Here's how I 'remember' how to wire a serial crossover cable when I'm in a data center, without easy web access.
I used to carry a small pocket notebook with notes and adapter diagrams, but those got worn and ratty and dog-eared. So I kept them in a computer, printed the notes and drawings, and carried a small (5" x 8.5") binder. This way, I could mark-up the notes when I was away from the computer, and make changes on the computer, and print more 'clean' pages. It was also easy to give a page away, since I knew I could easily replace it. Eventually, the binder got bigger, thicker, and heavier, until I finally started carrying the computer around.
Now, ten years later, I keep my console information pages on the web, but I also keep critical notes on my Treo (Palm OS device). But the OS isn't important! If you have a pocket/palmtop computer, you can use the memo functions to carry useful cable info in the note files.
ASCII art isn't easy, and it isn't pretty unless you are using a mono-spaced font. But recording the wiring for specialty cables is easy. In an earlier posting, I described how to determine the wiring for specialty cables. And most of us could easily spout off the wiring sequence for an AT&T 468B-wired CAT-5 cable, right? So, you start with that color sequence on one end...but what colors do you need on the other end? THIS is easily described with text, and proportional fonts do not get in the way.
I usually show the cable, wired from both ends, since I can use a simple RJ-45 cable continuity tester...but, I add the Cat-5 color code in between the pin numbers. I use the AT&T 468B sequence on the first end, since it's a good visual starting place. Then, I show the sequence on the 'other' end, with the resulting color sequence. It's this other sequence that it hard to translate in my mind, so I keep a 'note' that won't get dog-eared in my phone, since it's almost always going to be with me.
Here's an example of a crossover cable between Cyclades ACS Console Servers;
Cyclades crossover
one end...
1 whi-orn 5
2 orn-whi 8
3 whi-grn 6
4 blu-whi 4
5 whi-blu 1
6 grn-whi 3
7 whi-brn 7
8 brn-whi 2
other end...
1 whi-blu 5
2 brn-whi 8
3 grn-whi 6
4 blu-whi 4
5 whi-orn 1
6 whi-grn 3
7 whi-brn 7
8 orn-whi 2
Using the PalmOS memopad feature, I have note files for a variety of console servers. It's easy to combine wiring notes (crossover, loopback, signal inputs/outputs) for a single vendor into a separate file, for quick reference. You should be able to do something similar with other PalmTop devices.
I do still keep post-its handy at a few sites, with the "other end" wiring sequence, in case the memos aren't available. (The battery died, or I can't take a camera-phone into certain data centers...) When they start to get worn, I make new ones, using the records on my phone, or my web pages.
I hope this note is helpful to some of you. I know that it saves me a LOT of time.
I used to carry a small pocket notebook with notes and adapter diagrams, but those got worn and ratty and dog-eared. So I kept them in a computer, printed the notes and drawings, and carried a small (5" x 8.5") binder. This way, I could mark-up the notes when I was away from the computer, and make changes on the computer, and print more 'clean' pages. It was also easy to give a page away, since I knew I could easily replace it. Eventually, the binder got bigger, thicker, and heavier, until I finally started carrying the computer around.
Now, ten years later, I keep my console information pages on the web, but I also keep critical notes on my Treo (Palm OS device). But the OS isn't important! If you have a pocket/palmtop computer, you can use the memo functions to carry useful cable info in the note files.
ASCII art isn't easy, and it isn't pretty unless you are using a mono-spaced font. But recording the wiring for specialty cables is easy. In an earlier posting, I described how to determine the wiring for specialty cables. And most of us could easily spout off the wiring sequence for an AT&T 468B-wired CAT-5 cable, right? So, you start with that color sequence on one end...but what colors do you need on the other end? THIS is easily described with text, and proportional fonts do not get in the way.
I usually show the cable, wired from both ends, since I can use a simple RJ-45 cable continuity tester...but, I add the Cat-5 color code in between the pin numbers. I use the AT&T 468B sequence on the first end, since it's a good visual starting place. Then, I show the sequence on the 'other' end, with the resulting color sequence. It's this other sequence that it hard to translate in my mind, so I keep a 'note' that won't get dog-eared in my phone, since it's almost always going to be with me.
Here's an example of a crossover cable between Cyclades ACS Console Servers;
Cyclades crossover
one end...
1 whi-orn 5
2 orn-whi 8
3 whi-grn 6
4 blu-whi 4
5 whi-blu 1
6 grn-whi 3
7 whi-brn 7
8 brn-whi 2
other end...
1 whi-blu 5
2 brn-whi 8
3 grn-whi 6
4 blu-whi 4
5 whi-orn 1
6 whi-grn 3
7 whi-brn 7
8 orn-whi 2
Using the PalmOS memopad feature, I have note files for a variety of console servers. It's easy to combine wiring notes (crossover, loopback, signal inputs/outputs) for a single vendor into a separate file, for quick reference. You should be able to do something similar with other PalmTop devices.
I do still keep post-its handy at a few sites, with the "other end" wiring sequence, in case the memos aren't available. (The battery died, or I can't take a camera-phone into certain data centers...) When they start to get worn, I make new ones, using the records on my phone, or my web pages.
I hope this note is helpful to some of you. I know that it saves me a LOT of time.
20070831
When did my console service become Critical Infrastructure?
Many Conserver instances were started as an experiment. It was something to add to an existing console server deployment. It's usually installed on a seldom-used machine, probably older hardware, and maybe without support. But, it was certainly "enough to do the job, for now."
Once it was working, a few other folks started to use it. Then you offered to add more ports into the config for other system and network administrators. Soon, you've pulled in most of the original console server ports in the shop, and you're buying more console servers, and you're starting to look for more RAM and bigger hard drives. You're wondering if you are backing this system up, since the logs have become useful data. Then, one day, some data retention policy comes along, and you realize your console data just became Vital Records, and needs to be protected. It's time to think about an upgrade, and a service contract, and a way to write directly to archival media.
Why didn't you think about those things sooner? That's the topic for today's blog. What started as a demonstration just became critical infrastructure at one site I support. Here's what we are considering as we are looking for making this Conserver deployment a "Production Service".
Supported Hardware. Rather than a hand-me-down or a "Frankenstein machine", consider getting newer hardware. This will give you a longer product life (read that as "you can get tech support and replacement parts from the vendor" for 3-4 years), and you should consider the support contract, since this will likely become Critical Infrastructure if it hasn't already.
If you are trying to do this "on the cheap", you could try using a hand-me-down machine. Make sure you get a spare chassis (with power supply and motherboard), and and many drives and RAM! Remember, older drives and RAM get to be more expensive when they are no longer the new stuff! You'll also need to be able to service your own gear, on your own time.
Redundant Power Supplies. Unless your data center has fancy power distribution units that source two circuits to a since power cable, you should consider using a chassis that has dual power supplies. Make sure that the chassis can run fine (fully configured) on just one power supply! You should make sure that you are sourcing the power supplies from two different circuits. Also, find out if the power supplies have to be on the same PHASE of power, and find out BEFORE you plug them in. (Have I mentioned the value of a support contract for your hardware?)
I need RAM. Lots and lots of RAM. But how much is "lots". This depends on the number of consoles you plan to support, including your "someday" scenarios. Remember that Conserver starts with it's own process, and then spawns children for (generally) every 16 ports. Your OS will want some. And any other tools and scripts will need some. (If you are going to be editing large log files, searching large data sets, or processing many large log files, you're going to want a LOT of RAM, so that you avoid swapping memory to the hard drives.) Don't skimp on RAM.
Dedicated Log Data Drive(s). You don't want your logs on the same drive a your main OS. (If your console logs fill the disk over a holiday weekend, and your system can't write it's own system logs, your Monday morning is going to be a LONG morning!) How much space do you need? This is the hardest thing to estimate, since it really depends on how much you use your consoles. But, here is a good ballpark to start - estimate 20 MBytes for every console you plan to support, plus 1-2 GBytes as a buffer for when some logs start filling up faster than you expected.
Since Conserver can auto-rotate the logfiles when they reach a certain size, you want to consider how large a file you can open with the editing tools you want to use. I rotate my logs between 10-20 MB, and I use grep and PERL to find things. But if you use other tools that can't open files larger than, say, 5 MB, then you should adjust your log rotation size accordingly.
Use RAID to help protect your data. Even basic mirroring of your log drive will help protect the data set (your vital log files). If you are capturing a lot of very busy log files, you might want to consider striping the data. You may also want to consider hardware RAID support. (This can save you system some CPU overhead, but it also adds some hardware complication. Still, many sites are successfully using hardware RAID. If this is your first attempt to use hardware RAID, you should consider getting the hardware support contract for your machine.)
Consider using a RAID pair to protect your OS, scripts and tools as well! Many newer machines can host five 2.5-inch drives in a 1-Rack-Unit chassis. That gives you a pair for the OS, and a pair for log data.
Backing up your critical log files to archival media. CD media is cheap, but so is DVD media. (Heck, DVD drives are cheap, too!) You can get 5+ GBytes on a single-layer DVD, and must laptops and other workstations can read them. It's an ideal media for the day when the auditor comes and asks you to produce the log files for certain machines across a range of dates.
Remember, too, that most log data is going to be ASCII data. It's VERY compressible, and you can use PERL and cron jobs to compress the latest newly-rotated log files to a gzip version. This will let you store more log data longer, and lets you delete some of the older non-zipped versions. (Compressing the log files means you are going to be backing up the compressed logs, so you can store more of them on the DVD media...this means some savings in the number of discs you need to write over time.)
So, what would I recommend? Let me start by giving you the information that I'm using to base the decision, and then I'll tell you.
Given:
650 console ports today, could grow to 1024
(20 MB x 1,024 = 20 GB of drive, plus overhead, 25 GB minimum)
I want to store large amounts of compressed log files.
(3 yr retention needs for some files, 60 GB minimum)
1024 consoles means 65 processes, minimum, but more if my processes are really busy due to verbose logging. Could be 128+ Conserver processes later.
I want to run Splunk for log checking (RAM and drive implications.)
I also need drive space for backup, and log report manipulation.
What I'm proposing;
Dell 2950, with two 1.8 GHz Quad-core processors (Energy Smart)
Pick your OS (I'll pick Suse Linux, 3 yr license for the OS support)
8 GB of RAM (four 2 GB Dual Ranked DIMMs, Energy Smart)
Hardware RAID
76 GB, 15k-RPM drive threesome for the OS
146 GB, 10k-RPM drive threesome for the logs
Dual PS
This gives me a pair of drives for the OS, plus an on-site warm-spare, plus a pair for the logs with a warm spare. The drives are hot-swappable, so if I have a failed drive that RAID cannot recover, I'll yank the failed drive and swap in the warm spare, then let RAID rebuild it, and then I'll call Dell support for a next-day replacement, for the next three years. If I were really worried about hardware failure, I might opt for the 4-hour on-site contract.
Total cost comes in at $8.5k(US) today, from Dell, though the price may be better through other channels. (That works out to about $2,900 per year, or about $240 per month, to support 1,000 vital ports in the shop.)
Can you get by for less? Certainly. But, what is the cost to you if you lose log files you need to be retaining? What will the business impact be if the server is down for a day or more, and your administrators have to scramble to manage their servers and network gear 'the old fashioned way'? The cost isn't too bad, for Critical Infrastructure, now that it's proven itself to be useful and reliable. Shouldn't you put it on reliable hardware as well?
-Z-
Once it was working, a few other folks started to use it. Then you offered to add more ports into the config for other system and network administrators. Soon, you've pulled in most of the original console server ports in the shop, and you're buying more console servers, and you're starting to look for more RAM and bigger hard drives. You're wondering if you are backing this system up, since the logs have become useful data. Then, one day, some data retention policy comes along, and you realize your console data just became Vital Records, and needs to be protected. It's time to think about an upgrade, and a service contract, and a way to write directly to archival media.
Why didn't you think about those things sooner? That's the topic for today's blog. What started as a demonstration just became critical infrastructure at one site I support. Here's what we are considering as we are looking for making this Conserver deployment a "Production Service".
Supported Hardware. Rather than a hand-me-down or a "Frankenstein machine", consider getting newer hardware. This will give you a longer product life (read that as "you can get tech support and replacement parts from the vendor" for 3-4 years), and you should consider the support contract, since this will likely become Critical Infrastructure if it hasn't already.
If you are trying to do this "on the cheap", you could try using a hand-me-down machine. Make sure you get a spare chassis (with power supply and motherboard), and and many drives and RAM! Remember, older drives and RAM get to be more expensive when they are no longer the new stuff! You'll also need to be able to service your own gear, on your own time.
Redundant Power Supplies. Unless your data center has fancy power distribution units that source two circuits to a since power cable, you should consider using a chassis that has dual power supplies. Make sure that the chassis can run fine (fully configured) on just one power supply! You should make sure that you are sourcing the power supplies from two different circuits. Also, find out if the power supplies have to be on the same PHASE of power, and find out BEFORE you plug them in. (Have I mentioned the value of a support contract for your hardware?)
I need RAM. Lots and lots of RAM. But how much is "lots". This depends on the number of consoles you plan to support, including your "someday" scenarios. Remember that Conserver starts with it's own process, and then spawns children for (generally) every 16 ports. Your OS will want some. And any other tools and scripts will need some. (If you are going to be editing large log files, searching large data sets, or processing many large log files, you're going to want a LOT of RAM, so that you avoid swapping memory to the hard drives.) Don't skimp on RAM.
Dedicated Log Data Drive(s). You don't want your logs on the same drive a your main OS. (If your console logs fill the disk over a holiday weekend, and your system can't write it's own system logs, your Monday morning is going to be a LONG morning!) How much space do you need? This is the hardest thing to estimate, since it really depends on how much you use your consoles. But, here is a good ballpark to start - estimate 20 MBytes for every console you plan to support, plus 1-2 GBytes as a buffer for when some logs start filling up faster than you expected.
Since Conserver can auto-rotate the logfiles when they reach a certain size, you want to consider how large a file you can open with the editing tools you want to use. I rotate my logs between 10-20 MB, and I use grep and PERL to find things. But if you use other tools that can't open files larger than, say, 5 MB, then you should adjust your log rotation size accordingly.
Use RAID to help protect your data. Even basic mirroring of your log drive will help protect the data set (your vital log files). If you are capturing a lot of very busy log files, you might want to consider striping the data. You may also want to consider hardware RAID support. (This can save you system some CPU overhead, but it also adds some hardware complication. Still, many sites are successfully using hardware RAID. If this is your first attempt to use hardware RAID, you should consider getting the hardware support contract for your machine.)
Consider using a RAID pair to protect your OS, scripts and tools as well! Many newer machines can host five 2.5-inch drives in a 1-Rack-Unit chassis. That gives you a pair for the OS, and a pair for log data.
Backing up your critical log files to archival media. CD media is cheap, but so is DVD media. (Heck, DVD drives are cheap, too!) You can get 5+ GBytes on a single-layer DVD, and must laptops and other workstations can read them. It's an ideal media for the day when the auditor comes and asks you to produce the log files for certain machines across a range of dates.
Remember, too, that most log data is going to be ASCII data. It's VERY compressible, and you can use PERL and cron jobs to compress the latest newly-rotated log files to a gzip version. This will let you store more log data longer, and lets you delete some of the older non-zipped versions. (Compressing the log files means you are going to be backing up the compressed logs, so you can store more of them on the DVD media...this means some savings in the number of discs you need to write over time.)
So, what would I recommend? Let me start by giving you the information that I'm using to base the decision, and then I'll tell you.
Given:
650 console ports today, could grow to 1024
(20 MB x 1,024 = 20 GB of drive, plus overhead, 25 GB minimum)
I want to store large amounts of compressed log files.
(3 yr retention needs for some files, 60 GB minimum)
1024 consoles means 65 processes, minimum, but more if my processes are really busy due to verbose logging. Could be 128+ Conserver processes later.
I want to run Splunk for log checking (RAM and drive implications.)
I also need drive space for backup, and log report manipulation.
What I'm proposing;
Dell 2950, with two 1.8 GHz Quad-core processors (Energy Smart)
Pick your OS (I'll pick Suse Linux, 3 yr license for the OS support)
8 GB of RAM (four 2 GB Dual Ranked DIMMs, Energy Smart)
Hardware RAID
76 GB, 15k-RPM drive threesome for the OS
146 GB, 10k-RPM drive threesome for the logs
Dual PS
This gives me a pair of drives for the OS, plus an on-site warm-spare, plus a pair for the logs with a warm spare. The drives are hot-swappable, so if I have a failed drive that RAID cannot recover, I'll yank the failed drive and swap in the warm spare, then let RAID rebuild it, and then I'll call Dell support for a next-day replacement, for the next three years. If I were really worried about hardware failure, I might opt for the 4-hour on-site contract.
Total cost comes in at $8.5k(US) today, from Dell, though the price may be better through other channels. (That works out to about $2,900 per year, or about $240 per month, to support 1,000 vital ports in the shop.)
Can you get by for less? Certainly. But, what is the cost to you if you lose log files you need to be retaining? What will the business impact be if the server is down for a day or more, and your administrators have to scramble to manage their servers and network gear 'the old fashioned way'? The cost isn't too bad, for Critical Infrastructure, now that it's proven itself to be useful and reliable. Shouldn't you put it on reliable hardware as well?
-Z-
20070714
Making progress, leaving footprints.
It's a bit depressing to see that I'm down to about one blog per month. At least I'm making a bit of effort hre, as well as making progress on the other projects that compete for my time.
As I've changed jobs, I appreciate the 'Oral History Project' that most companies use to store and share knowledge about network infrastructure, legacy reasons for doing things, even network architecture. But, I only like them a little, mainly because they aren't a stable source. People leave companies, or even just transfer to other departments. Stories change with time, and details tend to degrade very rapidly. I like documents better. (That's one large reason I make my web pages, and keep this blog.)
I learned a long time ago, from two examples, that the time it takes to make some basic documentation is largely paid ack when you use the documentation 3-4 times. Whether it's time saved when you refer to it yourself at a later date, or whether it's time saved when you give the document to someone else, and they can do a task, without you needing to give them the information via storytelling and sitting with them while they do the task. (I won't put the example stories in this article, to save space. Maybe I'll post them in a separate article, if there is some demand for them.)
When I'm debugging a problem, I keep notes, usually in a 'notepad' or 'teach text' file. (Save the file with a useful name, and save it often during the project!) If I'm working on a device configuration, I keep a diary of the changes I made, in case I need to revert anything. I also paste in any error messages I get during the process.
BUT, here's the message for this blog... During the process, I make notes about some of the useful options for some of the commands I'm using. (I needed to look through the syntax for this effort, and I learned something...so I save it for a future effort!) When I'm done, and I've figured out the proper sequence of commands, I now have a basic document that shows me the correct steps, and a few added notes for related commands. If I take an extra 30-60 mintes, I can add some comments for the next time... things to be aware of, pitfalls to avoid, information to gather before you start, how long it might take... these all get into the document.
At the end of the effort, I now save a copy of the file in a "Clues" directory. This is my 'canonical' place for the most current version of my clues, for anything I've worked on. I've written my stories down, and everyone on the team knows where to look for them if they need to work on a project. If I'm unavailable, my team will still have access to useful knowledge on an array of tasks. (And, if it's easy to hand off a project, it's more likely I'll be able to hand it off and do something more interesting the next time.)
Make a clues repository, start gathering simple process docs, and get the other folks in your department to share in filling the repository.
-Z-
As I've changed jobs, I appreciate the 'Oral History Project' that most companies use to store and share knowledge about network infrastructure, legacy reasons for doing things, even network architecture. But, I only like them a little, mainly because they aren't a stable source. People leave companies, or even just transfer to other departments. Stories change with time, and details tend to degrade very rapidly. I like documents better. (That's one large reason I make my web pages, and keep this blog.)
I learned a long time ago, from two examples, that the time it takes to make some basic documentation is largely paid ack when you use the documentation 3-4 times. Whether it's time saved when you refer to it yourself at a later date, or whether it's time saved when you give the document to someone else, and they can do a task, without you needing to give them the information via storytelling and sitting with them while they do the task. (I won't put the example stories in this article, to save space. Maybe I'll post them in a separate article, if there is some demand for them.)
When I'm debugging a problem, I keep notes, usually in a 'notepad' or 'teach text' file. (Save the file with a useful name, and save it often during the project!) If I'm working on a device configuration, I keep a diary of the changes I made, in case I need to revert anything. I also paste in any error messages I get during the process.
BUT, here's the message for this blog... During the process, I make notes about some of the useful options for some of the commands I'm using. (I needed to look through the syntax for this effort, and I learned something...so I save it for a future effort!) When I'm done, and I've figured out the proper sequence of commands, I now have a basic document that shows me the correct steps, and a few added notes for related commands. If I take an extra 30-60 mintes, I can add some comments for the next time... things to be aware of, pitfalls to avoid, information to gather before you start, how long it might take... these all get into the document.
At the end of the effort, I now save a copy of the file in a "Clues" directory. This is my 'canonical' place for the most current version of my clues, for anything I've worked on. I've written my stories down, and everyone on the team knows where to look for them if they need to work on a project. If I'm unavailable, my team will still have access to useful knowledge on an array of tasks. (And, if it's easy to hand off a project, it's more likely I'll be able to hand it off and do something more interesting the next time.)
Make a clues repository, start gathering simple process docs, and get the other folks in your department to share in filling the repository.
-Z-
20070602
The Emergency Kit
I've put together a small kit, which can give me quick access to the serial consoles of various machines, for those little emergency trips to the colocation facility, or a customer site. It's been so successful and handy that colleagues are borrowing it for small cluster jumpstarts, and even for a Disaster Recovery exercise. So, I'd also describe it here.
The gist is, I started with a small 8-port console server, so I could use my laptop, a pre-configured telnet client, a crossover-Ethernet cable, and a handful of adapters to connect to various devices. I found that I often needed to be connected to the machine's network as well, so I added a small hub to the kit, as well as a small power outlet strip to my laptop bag.
I picked the Cyclades TS-800 8-port console server because it was small, was BREAK-safe, and ran from a 'wall wart' (power pack). I really wanted one that could run on 12-volts DC, so I could run the kit on car battery power, but couldn't find one. My laptop can also run from 12-volts DC (which could be handy in an emergency). Sadly, the TS-800 units are no longer sold by Avocent, but there are still a variety of small console servers on the market.
The small console server fit in the large pocket of my laptop bag, but limited how many cables I could carry, along with all the adapters. I finally fell to using a small, extra bag for the console bits.
I added a small 10-Mbps hub initially, so I could also do some packet sniffing, since most small switches don't have the ability to 'span' or 'mirror' ports. But, that bottleneck also throttled network access to the attached device, and I wasn't doing so much packet sniffing, so I 'traded up' to a Allied Telesyn AT-FS708LE 8-port 10/100 Mbps switch. This is a 12-volt DC device. It's small, lightweight, sturdy, and still available cheaply.
The kit now includes the switch, the console server, two crossover-Ethernet cables (1'- and 5'-long), an assortment of lengths of regular 8-wire CAT-5 cables, and an assortment of 24 Cyclades serial adapters. I also have a few 1'-long Cyclades null-modem (DCE-to-DTE crossed RJ-45 cables, and a few CAT-5e-rated female RF45 couplers. Add in the wall warts, and the kit lives in a 6-quart Sterilite plastic container about the size of a shoebox. Easy for security folks to see what is inside, but you don't want to put this in the belly of an airplane.
The Console Server is really the key to this kit, so here's what I've done to make it work;
First, configured the device as a console server, then copied the config file so I have an on-board copy, in case someone feels the need to change settings while they borrow it.
cp /etc/portslave/pslave.conf /etc/portslave/pslave.bkup
Make sure the backup config is saved into the unit's NVRAM. On the Cyclades, this means adding the file into /etc/config_files. (This is one reason I strip most of the comments out of my console server config files, since I'm basically saving two copies of the configuration file.) Now, if someone needs to change the config file, it's quick and easy to get back on the console, and copy the config back over the pslave.conf file, then 'saveconf' and reboot.
I also add an entry for my laptop into the /etc/hosts file on the console server, to reduce delays when making your reverse-TCP connections to the console server.
I had originally assigned different port speeds (19.2k, 38.8, 57.6, and 115.2) to ports 5 through 8, to make it quick and easy to access a high-speed port. But, I found that most ports are STILL at 9.6kbps as a default, and sometimes I needed more than 4 ports. In those cases when I needed a higher speed, I often needed more than one port at the higher speed at the same time. Since I found myself making many configuration changes I decided to revert the base configuration to 9.6k all 'round, but that's just explaining my reasons. You may want to do it your own way, and that's up to you.
I also pointed all the "extra" services (Radius server, syslog, etc) that the console server might want to use to the laptop IP address. I already use a small but of RFC-1918 address space, to reduce the chances of a conflict with the network that I may attach to. Pointing just to my laptop reduces the chance of the TS tripping any Intrusion Detection System (IDS) that might be watching the network. (I don't use the Syslog much anymore, but if I want to try it, that's one less part I need to configure.)
The current laptop is a PC, using PuTTY as the client tool. This lets me save the basic settings for "ts-1" through "ts-8" (configured to connect to the IP address of the console server, at ports 7001-7008, respectively). I use a distinctive window size and text/background colors for these windows, and each connection is configured with a unique 'window name', and a unique log-file name. The distinctive window settings make it easy to tell which open windows are serial connections, versus network sessions when I'm debugging.
An advantage to using PuTTY for this, is I can also 'export' a copy the Windows Registry settings for these sessions onto my FLASH fob, and easily give the settings to someone who is going to borrow the kit, and VIOLA! They are ready to connect to the ports quickly and easily. PuTTY is small, and runs without needing to be 'installed', so I have a copy of that on the FLASH fob as well.
The final bit that makes the kit as easy to loan as it is for me to use it, is a 1-page README document (also on the FLASH fob...with a canned Conserver config snippet for this device, just in case), which outlines the default IP settings, serial speed settings, adapter and cabling clues, and basics about how to open a reverse-TCP session to a serial port. I also include clues and the password for configuring the device. (Add a few useful labels to the hub and console server, mark the wall-wart connectors with a color-code at the device-end, and things go pretty smoothly.)
I don't use it very often, but when I need it, it's ready to go! When your time is worth money, having a kit on the shelf is money well spent!
-Z-
The gist is, I started with a small 8-port console server, so I could use my laptop, a pre-configured telnet client, a crossover-Ethernet cable, and a handful of adapters to connect to various devices. I found that I often needed to be connected to the machine's network as well, so I added a small hub to the kit, as well as a small power outlet strip to my laptop bag.
I picked the Cyclades TS-800 8-port console server because it was small, was BREAK-safe, and ran from a 'wall wart' (power pack). I really wanted one that could run on 12-volts DC, so I could run the kit on car battery power, but couldn't find one. My laptop can also run from 12-volts DC (which could be handy in an emergency). Sadly, the TS-800 units are no longer sold by Avocent, but there are still a variety of small console servers on the market.
The small console server fit in the large pocket of my laptop bag, but limited how many cables I could carry, along with all the adapters. I finally fell to using a small, extra bag for the console bits.
I added a small 10-Mbps hub initially, so I could also do some packet sniffing, since most small switches don't have the ability to 'span' or 'mirror' ports. But, that bottleneck also throttled network access to the attached device, and I wasn't doing so much packet sniffing, so I 'traded up' to a Allied Telesyn AT-FS708LE 8-port 10/100 Mbps switch. This is a 12-volt DC device. It's small, lightweight, sturdy, and still available cheaply.
The kit now includes the switch, the console server, two crossover-Ethernet cables (1'- and 5'-long), an assortment of lengths of regular 8-wire CAT-5 cables, and an assortment of 24 Cyclades serial adapters. I also have a few 1'-long Cyclades null-modem (DCE-to-DTE crossed RJ-45 cables, and a few CAT-5e-rated female RF45 couplers. Add in the wall warts, and the kit lives in a 6-quart Sterilite plastic container about the size of a shoebox. Easy for security folks to see what is inside, but you don't want to put this in the belly of an airplane.
The Console Server is really the key to this kit, so here's what I've done to make it work;
First, configured the device as a console server, then copied the config file so I have an on-board copy, in case someone feels the need to change settings while they borrow it.
cp /etc/portslave/pslave.conf /etc/portslave/pslave.bkup
Make sure the backup config is saved into the unit's NVRAM. On the Cyclades, this means adding the file into /etc/config_files. (This is one reason I strip most of the comments out of my console server config files, since I'm basically saving two copies of the configuration file.) Now, if someone needs to change the config file, it's quick and easy to get back on the console, and copy the config back over the pslave.conf file, then 'saveconf' and reboot.
I also add an entry for my laptop into the /etc/hosts file on the console server, to reduce delays when making your reverse-TCP connections to the console server.
I had originally assigned different port speeds (19.2k, 38.8, 57.6, and 115.2) to ports 5 through 8, to make it quick and easy to access a high-speed port. But, I found that most ports are STILL at 9.6kbps as a default, and sometimes I needed more than 4 ports. In those cases when I needed a higher speed, I often needed more than one port at the higher speed at the same time. Since I found myself making many configuration changes I decided to revert the base configuration to 9.6k all 'round, but that's just explaining my reasons. You may want to do it your own way, and that's up to you.
I also pointed all the "extra" services (Radius server, syslog, etc) that the console server might want to use to the laptop IP address. I already use a small but of RFC-1918 address space, to reduce the chances of a conflict with the network that I may attach to. Pointing just to my laptop reduces the chance of the TS tripping any Intrusion Detection System (IDS) that might be watching the network. (I don't use the Syslog much anymore, but if I want to try it, that's one less part I need to configure.)
The current laptop is a PC, using PuTTY as the client tool. This lets me save the basic settings for "ts-1" through "ts-8" (configured to connect to the IP address of the console server, at ports 7001-7008, respectively). I use a distinctive window size and text/background colors for these windows, and each connection is configured with a unique 'window name', and a unique log-file name. The distinctive window settings make it easy to tell which open windows are serial connections, versus network sessions when I'm debugging.
An advantage to using PuTTY for this, is I can also 'export' a copy the Windows Registry settings for these sessions onto my FLASH fob, and easily give the settings to someone who is going to borrow the kit, and VIOLA! They are ready to connect to the ports quickly and easily. PuTTY is small, and runs without needing to be 'installed', so I have a copy of that on the FLASH fob as well.
The final bit that makes the kit as easy to loan as it is for me to use it, is a 1-page README document (also on the FLASH fob...with a canned Conserver config snippet for this device, just in case), which outlines the default IP settings, serial speed settings, adapter and cabling clues, and basics about how to open a reverse-TCP session to a serial port. I also include clues and the password for configuring the device. (Add a few useful labels to the hub and console server, mark the wall-wart connectors with a color-code at the device-end, and things go pretty smoothly.)
I don't use it very often, but when I need it, it's ready to go! When your time is worth money, having a kit on the shelf is money well spent!
-Z-
20070422
Relieve pressure in your shop!
We've got plenty of pressure with deadlines looming, new projects, and other 'people issues'. But the pressure I'm writing about today is torque.
For serial adapters that screw on (DB25, DE9, etc.), you shouldn't need to tighten them more than finger-tight. The screws are meant to keep the connector from being pulled out if the cable is tugged. Most of the small threaded nuts and screw-end on the adapters are not made for serious torque, and will likely break, or strip, if you apply too much pressure.
Finger-tight for you may be tighter than finger-tight for someone else that might need to work on the gear later. I don't want to make someone have to go get tools to remove serial adapters.
This goes for the screws for mounting your gear in cabinets or relay racks. You can tighten these hand-tight, but don't set your screw-gun to be an impact wrench! The lowest settings will probably be fine. (If it's not, you should make sure that you haven't cross-threaded the screw to begin with, rather than cutting new threads in the rails...because aluminum rails will strip easily, and steel rails will chew your screws easily. Let's try to not damage any parts today.)
The other damage that occurs from too much pressure is damaging the screwdriver bits and the screw-tops. It starts when the screw stops, and the drill keeps going, and the screwdriver bit starts to round over the screw head, but it's starting to damage the screwdriver bit as well! This makes it easier for the screwdriver bit to round over the next screwheads. (Take a good look at the philips bit in your screwgun today. If you don't have good, straight edges, it's time to turn down the torque, and either get out the file to renew your bit, or use a new replacement bit.)
The pressure you will save, will be the pressure on you in your next emergency. You'll be able to quickly remove an adapter without needing to go get tools. Your tools will easily remove and install screws, and your screws and rack rails won't be damaged by 'the last guy that worked on them'.
Once you set the torque on your screwgun to only 1 or 2, keep checking it, since there are probably other folks that borrow it. They'll likely change the setting, and not put it back. When you find it has been changed, you may be able to share this note with the culprit, and make things just a bit nicer in your shop.
For serial adapters that screw on (DB25, DE9, etc.), you shouldn't need to tighten them more than finger-tight. The screws are meant to keep the connector from being pulled out if the cable is tugged. Most of the small threaded nuts and screw-end on the adapters are not made for serious torque, and will likely break, or strip, if you apply too much pressure.
Finger-tight for you may be tighter than finger-tight for someone else that might need to work on the gear later. I don't want to make someone have to go get tools to remove serial adapters.
This goes for the screws for mounting your gear in cabinets or relay racks. You can tighten these hand-tight, but don't set your screw-gun to be an impact wrench! The lowest settings will probably be fine. (If it's not, you should make sure that you haven't cross-threaded the screw to begin with, rather than cutting new threads in the rails...because aluminum rails will strip easily, and steel rails will chew your screws easily. Let's try to not damage any parts today.)
The other damage that occurs from too much pressure is damaging the screwdriver bits and the screw-tops. It starts when the screw stops, and the drill keeps going, and the screwdriver bit starts to round over the screw head, but it's starting to damage the screwdriver bit as well! This makes it easier for the screwdriver bit to round over the next screwheads. (Take a good look at the philips bit in your screwgun today. If you don't have good, straight edges, it's time to turn down the torque, and either get out the file to renew your bit, or use a new replacement bit.)
The pressure you will save, will be the pressure on you in your next emergency. You'll be able to quickly remove an adapter without needing to go get tools. Your tools will easily remove and install screws, and your screws and rack rails won't be damaged by 'the last guy that worked on them'.
Once you set the torque on your screwgun to only 1 or 2, keep checking it, since there are probably other folks that borrow it. They'll likely change the setting, and not put it back. When you find it has been changed, you may be able to share this note with the culprit, and make things just a bit nicer in your shop.
Subscribe to:
Posts (Atom)