load balanced configuration

[Alex Dean]

On May 19, 2010, at 2:47 PM, keith smith wrote:

>
>
> Hi Plug,
>
> I am considering combining the sites on two servers and creating a  
> load balanced configuration.  One server will be in one data center  
> and another will be in a different data center.  This is to ensure  
> if one data center goes off line that these site will still be  
> available via the Internet.  This is also a consideration for  
> backing up our data and content.  Each server is configured with a  
> RAID 1 disk set.
>
> I have never taken on such a responsibility so I am not sure of any  
> gotchas or anything I should be considering.
>
> Our data center guy will be doing most of the work, which will help.
>
> Any suggestions or concerns I should have with two servers in a load  
> balanced arrangement with each server in a different location?

You're entering a world of pain. :)

HA is cool, but is no panacea.  If you haven't actually experienced  
downtime due to your server crashing or your datacenter losing  
connectivity, I recommend thinking long and hard about it.  Don't  
solve a problem you don't have.  The downtime created from unneeded  
failovers will likely exceed the actual/real downtime caused by either  
a server or datacenter being offline.  Managing the cluster itself (as  
distinct from the services provided by the cluster) needs to be  
accounted for as an expense/responsibility.

I don't want to sound overly pessimistic.  I've set up quite a few HA  
clusters, and actually enjoy it most of the time.  But it WILL cause  
you headaches in the middle of the night which you wouldn't have had  
if you only had a single server.

Leave yourself lots of time to set up a development/test cluster, and  
abuse it in many ways.  Pull out network cables, kill the switch, yank  
out power cables, etc.  Do this with real hardware, not VMs.

When the cluster nodes lose contact with each other, both will decide  
to become primary.  This is a split brain.  This can happen when the  
switch in-between them gets busy and starts dropping pings.  Now, you  
can always recover from such things.  I'm just recommending you become  
very familiar with these issues before going live with this setup.

http://clusterlabs.org/wiki/Main_Page
http://people.linbit.com/~florian/heartbeat-users-guide/

Let me/us know if you have specific questions once you start setting  
things up.  Good luck!

alex