Best Practices-System Changes (Small environments)

[George Toft]

George rambling on . . .

Making Sure Servers Are Identical
=================================
I have a file called /root/changes that is actually a shell script of 
every command I issue to make a change on a box.  If I need to make a 
change to httpd.conf, for example, I script it with sed or vi.

Here's an example of how it worked: I built a web server in development, 
and tested it.  Once we were happy with it, we deployed the production 
server and executed the commands in /root/changes by scraping and 
pasting the commands.  It took 15 minutes to configure the prod server 
and 30 minutes to upload content.  I could have executed the file, but 
chose to carefully view the results of each command instead.

When we had to add mail functionality, I built qmail in development, 
added the instructions to /root/changes, and paste/scrape in prod. 
Easy-peasy.  Then I updated /root/changes on the prod server :)

Needless to say, I absolutely backup /root as my DR plan for that server 
is stored there :)


Managing Multiple Servers Simultaneously
========================================
If you need to make small changes to many machines, consider using an 
expect script called multixterm that allows you to spawn multiple xterms 
with one command window.  This way, all you need to do is enter the 
command once in the command window, and it goes to every box you have an 
xterm on.

DANGER *** DANGER *** DANGER *** DANGER *** DANGER *** DANGER
You better type the command perfectly.  If you make a mistake, you are 
magnifying your mistake exponentially.

I used this method at IBM for the 100+ servers I admin'd.  It drove 
management nuts, but I was 4x as efficient as most other SA's (by their 
metrics).  Since the script was standard and well tested by the OS 
Engineers, I could update the servers as fast as I could log in and 
paste the commands.

Ever try to manage 100+ xterms?  I used icewm with 12 desktops on two 
monitors to logically group the xterms and position them so I could see 
the results of the commands.  (It was actually icewm under cygwin on 
Windows XP.)


Different Method
================
When I was at a web hosting company in L.A., I developed a system where 
I could push out updates in the form of a script.  Every hour, the 
recipient boxes would check for the update script and execute it.  Yes, 
I made a mistake on one push and had to engineer a fix and push it out 
in under an hour.  There is much to be said for *thoroughly* testing 
your changes as well as scrape and paste :)


Cheers!

George Toft, CISSP, MSIS
623-203-1760




Dazed_75 wrote:
> I find myself making the same alterations to a small number of systems 
> but some of them repeatedly as I try different distros.  I try to keep 
> some notes but find they are inadequate and inconsistent.  It is also 
> painful to always make the changes manually.  I know that in large 
> environments there are tools such as source management systems, network 
> installers, etc. but they are serious overkill to the small network or 
> individual user such as I.
> 
> It occurs to me that this is a situation many of you have faced and 
> perhaps conquered.  My hope is that there are some best practices people 
> might be willing to discuss here and that the discussion might be useful 
> to others besides me.  So if you have favored techniques or tools, lets 
> hear about them.  For example, a tool/script/etc to scan for and 
> document changes from a base install, a format/method for noting changes 
> made (and when/how?), or a reasonable means for re-asserting those 
> changes after a system re-install or upgrade.  Yes, it is a lot to ask 
> but not asking helps no one.
> 
> -- 
> Be who you are and say what you feel, because those who mind don't 
> matter and those who matter don't mind.  - Dr. Seuss
> 
> 
> ------------------------------------------------------------------------
> 
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change  you mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss