load balanced configuration

Wed May 19 19:57:59 MST 2010

This is kind of fizzy to me.  I'm glad you brought it up.  I did experience this 6 to 9 month ago when the data center chanced the NIC card.  I think they had to flush some buffers in their routers so the new MAC address could be found and cached if I recall correctly.

We are in a data center and use their DNS.  So I'm thinking the request goes to the root server then to the data center's DNS and it tells the client what the IP address is.  So if the Data Center's DNS is changed to point to a new IP for our domain then that would be instantaneous or would the client and everyone along the way cache the IP?   

------------------------

Keith Smith

--- On Wed, 5/19/10, Ed Knapp <catbertek at hotmail.com> wrote:

From: Ed Knapp <catbertek at hotmail.com>
Subject: Re: load balanced configuration
To: "Main PLUG discussion list" <plug-discuss at lists.plug.phoenix.az.us>
Date: Wednesday, May 19, 2010, 6:09 PM

Re: load balanced configuration
One thing struck me here with your description...

“and a change to the DNS and we are off and running”

While your DNS records might be changed relatively quickly during an incident, the change

Itself can take quite a while to trickle down to the end users/clients out in the cloud.

Any client’s DNS resolution that has not expired in the cache nor manually refreshed will

still fail to properly resolve/connect.  It doesn’t usually, but I tell clients to plan for 48 hours

Estimated time for the change to completely propagate.

I would hate for you to get blindsided with a person hovering over you asking how much longer

It is going to take before the site is back up and operational.  It is frustrating when you have

Fixed the issue [  problem :-)   ] but have to just sit and wait for it to complete.

There are certainly strategies to mitigate this risk and I do not know if you maintain your

Own DNS servers or do you work through a hosting provider/domain registrar.

I hope this helps a bit.

Ed

On 5/19/10 2:07 PM, "keith smith" <klsmith2020 at yahoo.com> wrote:

Currently we have two servers in our main data center.  One serves our shopping cart.  The other contains quite a bit of content that is data driven (reads).  The content site is very active.  The orders on the shopping cart are spread apart by one or two minutes during the busiest part of the day.  We store a lot of data with each order so most of this is writing. The shopping cart is backed up to the server in the other data center.  Supposedly if there is a problem, a few things need to be done to the backup server in preparation to make it live, and a change to the DNS and we are off and running. 

The problem I am trying to solve is the other server (content site) is not currently backed up automatically.

Another layer of this is these are managed servers.  We have an excellent relationship with the data center owner and have 24/7 access to him and his staff.  He manages all three servers and has always done a good job.  

I am the one tasked with keeping our sites online 24/7.

I was hoping by configuring two servers, each in a different location, that, in the event of one of the data centers being completely severed from the Internet that the other server would automatically, without any human intervention, take over the full load of the other server and those visiting either of our sites would not know there had been an issue.

In a nutshell I am trying to create an automated backup that is a automated fail over solution.

I appreciate all your feedback!

------------------------

Keith Smith

--- On Wed, 5/19/10, Dan Dubovik <dandubo at gmail.com> wrote:

From: Dan Dubovik <dandubo at gmail.com>

Subject: Re: load balanced configuration

To: "Main PLUG discussion list" <plug-discuss at lists.plug.phoenix.az.us>

Date: Wednesday, May 19, 2010, 1:45 PM

The question I have, are you trying to actually load balance things? Or just have a remote location that you can fire up with live data at a moments notice?  Basically, are you wanting an active/active configuration, or active/passive? 

active/active across DC's can get kind of hairy depending on what the network looks like.  active/passive won't give you any performance gains, but can simplify the configuration, while providing the HA you seem to be after.  As Kaia pointed out, what the traffic looks like (reads vs writes) is a consideration.  If it is something that users don't write to, and data doesn't have to be replicated across DCs frequently, this further simplifies things.

Ultimately, the configuration will depend on what the application and network looks like currently, and what level of redundancy you want / need.

-- Dan.

On Wed, May 19, 2010 at 1:40 PM, Matt Iavarone <matt.iavarone at gmail.com </mc/compose?to=matt.iavarone at gmail.com> > wrote:

I think the original question was around stateless load balancing, not clustering.  Cross DC clustering is a headache, but HA web sites aren't exactly terchnical challenges these days. 

On May 19, 2010 4:33 PM, "Alex Dean" <alex at crackpot.org </mc/compose?to=alex at crackpot.org> > wrote:

On May 19, 2010, at 2:47 PM, keith smith wrote:

>

>

> Hi Plug,

>

> I am considering combining the ... You're entering a world of pain. :)

HA is cool, but is no panacea.  If you haven't actually experienced downtime due to your server crashing or your datacenter losing connectivity, I recommend thinking long and hard about it.  Don't solve a problem you don't have.  The downtime created from unneeded failovers will likely exceed the actual/real downtime caused by either a server or datacenter being offline.  Managing the cluster itself (as distinct from the services provided by the cluster) needs to be accounted for as an expense/responsibility.

I don't want to sound overly pessimistic.  I've set up quite a few HA clusters, and actually enjoy it most of the time.  But it WILL cause you headaches in the middle of the night which you wouldn't have had if you only had a single server.

Leave yourself lots of time to set up a development/test cluster, and abuse it in many ways.  Pull out network cables, kill the switch, yank out power cables, etc.  Do this with real hardware, not VMs.

When the cluster nodes lose contact with each other, both will decide to become primary.  This is a split brain.  This can happen when the switch in-between them gets busy and starts dropping pings.  Now, you can always recover from such things.  I'm just recommending you become very familiar with these issues before going live with this setup.

http://clusterlabs.org/wiki/Main_Page

http://people.linbit.com/~florian/heartbeat-users-guide/ <http://people.linbit.com/%7Eflorian/heartbeat-users-guide/> 

Let me/us know if you have specific questions once you start setting things up.  Good luck!

alex

---------------------------------------------------

PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us </mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us> 

To subscribe, unsubscribe, or to change your mail settings:

http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss

---------------------------------------------------

PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us </mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us> 

To subscribe, unsubscribe, or to change your mail settings:

http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss

-----Inline Attachment Follows-----

---------------------------------------------------

PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us </mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us> 

To subscribe, unsubscribe, or to change your mail settings:

http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss

---------------------------------------------------

PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us

To subscribe, unsubscribe, or to change your mail settings:

http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss

-----Inline Attachment Follows-----

---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
To subscribe, unsubscribe, or to change your mail settings:
http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.PLUG.phoenix.az.us/pipermail/plug-discuss/attachments/20100519/0743dc97/attachment.htm>