load balanced configuration

Wed May 19 18:09:16 MST 2010

One thing struck me here with your description...

³and a change to the DNS and we are off and running²

While your DNS records might be changed relatively quickly during an
incident, the change
Itself can take quite a while to trickle down to the end users/clients out
in the cloud.
Any client¹s DNS resolution that has not expired in the cache nor manually
refreshed will
still fail to properly resolve/connect.  It doesn¹t usually, but I tell
clients to plan for 48 hours
Estimated time for the change to completely propagate.

I would hate for you to get blindsided with a person hovering over you
asking how much longer
It is going to take before the site is back up and operational.  It is
frustrating when you have
Fixed the issue [  problem :-)   ] but have to just sit and wait for it to
complete.

There are certainly strategies to mitigate this risk and I do not know if
you maintain your
Own DNS servers or do you work through a hosting provider/domain registrar.

I hope this helps a bit.

Ed

On 5/19/10 2:07 PM, "keith smith" <klsmith2020 at yahoo.com> wrote:

> 
> 
> Currently we have two servers in our main data center.  One serves our
> shopping cart.  The other contains quite a bit of content that is data driven
> (reads).  The content site is very active.  The orders on the shopping cart
> are spread apart by one or two minutes during the busiest part of the day.  We
> store a lot of data with each order so most of this is writing. The shopping
> cart is backed up to the server in the other data center.  Supposedly if there
> is a problem, a few things need to be done to the backup server in preparation
> to make it live, and a change to the DNS and we are off and running.
> 
> The problem I am trying to solve is the other server (content site) is not
> currently backed up automatically.
> 
> Another layer of this is these are managed servers.  We have an excellent
> relationship with the data center owner and have 24/7 access to him and his
> staff.  He manages all three servers and has always done a good job.
> 
> I am the one tasked with keeping our sites online 24/7.
> 
> I was hoping by configuring two servers, each in a different location, that,
> in the event of one of the data centers being completely severed from the
> Internet that the other server would automatically, without any human
> intervention, take over the full load of the other server and those visiting
> either of our sites would not know there had been an issue.
> 
> In a nutshell I am trying to create an automated backup that is a automated
> fail over solution.
> 
> I appreciate all your feedback!
> 
> ------------------------
> Keith Smith
> 
> --- On Wed, 5/19/10, Dan Dubovik <dandubo at gmail.com> wrote:
>> 
>> From: Dan Dubovik <dandubo at gmail.com>
>> Subject: Re: load balanced configuration
>> To: "Main PLUG discussion list" <plug-discuss at lists.plug.phoenix.az.us>
>> Date: Wednesday, May 19, 2010, 1:45 PM
>> 
>> The question I have, are you trying to actually load balance things? Or just
>> have a remote location that you can fire up with live data at a moments
>> notice?  Basically, are you wanting an active/active configuration, or
>> active/passive? 
>> active/active across DC's can get kind of hairy depending on what the network
>> looks like.  active/passive won't give you any performance gains, but can
>> simplify the configuration, while providing the HA you seem to be after.  As
>> Kaia pointed out, what the traffic looks like (reads vs writes) is a
>> consideration.  If it is something that users don't write to, and data
>> doesn't have to be replicated across DCs frequently, this further simplifies
>> things.
>> 
>> Ultimately, the configuration will depend on what the application and network
>> looks like currently, and what level of redundancy you want / need.
>> 
>> -- Dan.
>> 
>> On Wed, May 19, 2010 at 1:40 PM, Matt Iavarone <matt.iavarone at gmail.com
>> </mc/compose?to=matt.iavarone at gmail.com> > wrote:
>>> I think the original question was around stateless load balancing, not
>>> clustering.  Cross DC clustering is a headache, but HA web sites aren't
>>> exactly terchnical challenges these days.
>>>> On May 19, 2010 4:33 PM, "Alex Dean" <alex at crackpot.org
>>>> </mc/compose?to=alex at crackpot.org> > wrote:
>>>> 
>>>> 
>>>> On May 19, 2010, at 2:47 PM, keith smith wrote:
>>>> 
>>>>> >
>>>>> >
>>>>> > Hi Plug,
>>>>> >
>>>>> > I am considering combining the ... You're entering a world of pain. :)
>>>> 
>>>> HA is cool, but is no panacea.  If you haven't actually experienced
>>>> downtime due to your server crashing or your datacenter losing
>>>> connectivity, I recommend thinking long and hard about it.  Don't solve a
>>>> problem you don't have.  The downtime created from unneeded failovers will
>>>> likely exceed the actual/real downtime caused by either a server or
>>>> datacenter being offline.  Managing the cluster itself (as distinct from
>>>> the services provided by the cluster) needs to be accounted for as an
>>>> expense/responsibility.
>>>> 
>>>> I don't want to sound overly pessimistic.  I've set up quite a few HA
>>>> clusters, and actually enjoy it most of the time.  But it WILL cause you
>>>> headaches in the middle of the night which you wouldn't have had if you
>>>> only had a single server.
>>>> 
>>>> Leave yourself lots of time to set up a development/test cluster, and abuse
>>>> it in many ways.  Pull out network cables, kill the switch, yank out power
>>>> cables, etc.  Do this with real hardware, not VMs.
>>>> 
>>>> When the cluster nodes lose contact with each other, both will decide to
>>>> become primary.  This is a split brain.  This can happen when the switch
>>>> in-between them gets busy and starts dropping pings.  Now, you can always
>>>> recover from such things.  I'm just recommending you become very familiar
>>>> with these issues before going live with this setup.
>>>> 
>>>> http://clusterlabs.org/wiki/Main_Page
>>>> http://people.linbit.com/~florian/heartbeat-users-guide/
>>>> <http://people.linbit.com/%7Eflorian/heartbeat-users-guide/>
>>>> 
>>>> Let me/us know if you have specific questions once you start setting things
>>>> up.  Good luck!
>>>> 
>>>> alex
>>>> ---------------------------------------------------
>>>> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>>>> </mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us>
>>>> To subscribe, unsubscribe, or to change your mail settings:
>>>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>> 
>>> ---------------------------------------------------
>>> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>>> </mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us>
>>> To subscribe, unsubscribe, or to change your mail settings:
>>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>> 
>> 
>> -----Inline Attachment Follows-----
>> 
>> ---------------------------------------------------
>> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>> </mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us>
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
> 
> 
> 
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.PLUG.phoenix.az.us/pipermail/plug-discuss/attachments/20100519/381feaad/attachment.htm>