load balanced configuration

Wed May 19 20:09:39 MST 2010

On Wed, May 19, 2010 at 7:57 PM, keith smith <klsmith2020 at yahoo.com> wrote:

>
> This is kind of fizzy to me.  I'm glad you brought it up.  I did experience
> this 6 to 9 month ago when the data center chanced the NIC card.  I think
> they had to flush some buffers in their routers so the new MAC address could
> be found and cached if I recall correctly.
>

They likely had to clear the ARP cache.

>
> We are in a data center and use their DNS.  So I'm thinking the request
> goes to the root server then to the data center's DNS and it tells the
> client what the IP address is.  So if the Data Center's DNS is changed to
> point to a new IP for our domain then that would be instantaneous or would
> the client and everyone along the way cache the IP?
>
>
The A Record for your site can still be cached for up to the TTL time by
local dns services.  Thus, the following can occur:

1) Client visits site
2) ISP cache's IP address after it does the initial look up
3) Something bad happens to the data center
4) IP address is updated
5) Client goes to visit site again
6) Client's ISP realizes they have visited that site recently, and provides
the cached answer for the DNS response.
7) Clients browser / application is forwarded to the old IP address, and
encounters an error.

>
> ------------------------
> Keith Smith
>
> --- On *Wed, 5/19/10, Ed Knapp <catbertek at hotmail.com>* wrote:
>
>
> From: Ed Knapp <catbertek at hotmail.com>
>
> Subject: Re: load balanced configuration
> To: "Main PLUG discussion list" <plug-discuss at lists.plug.phoenix.az.us>
> Date: Wednesday, May 19, 2010, 6:09 PM
>
>
> One thing struck me here with your description...
>
> “and a change to the DNS and we are off and running”
>
> While your DNS records might be changed relatively quickly during an
> incident, the change
> Itself can take quite a while to trickle down to the end users/clients out
> in the cloud.
> Any client’s DNS resolution that has not expired in the cache nor manually
> refreshed will
> still fail to properly resolve/connect.  It doesn’t usually, but I tell
> clients to plan for 48 hours
> Estimated time for the change to completely propagate.
>
> I would hate for you to get blindsided with a person hovering over you
> asking how much longer
> It is going to take before the site is back up and operational.  It is
> frustrating when you have
> Fixed the issue [  problem :-)   ] but have to just sit and wait for it to
> complete.
>
> There are certainly strategies to mitigate this risk and I do not know if
> you maintain your
> Own DNS servers or do you work through a hosting provider/domain registrar.
>
> I hope this helps a bit.
>
> Ed
>
>
> On 5/19/10 2:07 PM, "keith smith" <klsmith2020 at yahoo.com> wrote:
>
>
>
> Currently we have two servers in our main data center.  One serves our
> shopping cart.  The other contains quite a bit of content that is data
> driven (reads).  The content site is very active.  The orders on the
> shopping cart are spread apart by one or two minutes during the busiest part
> of the day.  We store a lot of data with each order so most of this is
> writing. The shopping cart is backed up to the server in the other data
> center.  Supposedly if there is a problem, a few things need to be done to
> the backup server in preparation to make it live, and a change to the DNS
> and we are off and running.
>
> The problem I am trying to solve is the other server (content site) is not
> currently backed up automatically.
>
> Another layer of this is these are managed servers.  We have an excellent
> relationship with the data center owner and have 24/7 access to him and his
> staff.  He manages all three servers and has always done a good job.
>
> I am the one tasked with keeping our sites online 24/7.
>
> I was hoping by configuring two servers, each in a different location,
> that, in the event of one of the data centers being completely severed from
> the Internet that the other server would automatically, without any human
> intervention, take over the full load of the other server and those visiting
> either of our sites would not know there had been an issue.
>
> In a nutshell I am trying to create an automated backup that is a automated
> fail over solution.
>
> I appreciate all your feedback!
>
> ------------------------
> Keith Smith
>
> --- On *Wed, 5/19/10, Dan Dubovik <dandubo at gmail.com>* wrote:
>
>
> From: Dan Dubovik <dandubo at gmail.com>
> Subject: Re: load balanced configuration
> To: "Main PLUG discussion list" <plug-discuss at lists.plug.phoenix.az.us>
> Date: Wednesday, May 19, 2010, 1:45 PM
>
> The question I have, are you trying to actually load balance things? Or
> just have a remote location that you can fire up with live data at a moments
> notice?  Basically, are you wanting an active/active configuration, or
> active/passive?
> active/active across DC's can get kind of hairy depending on what the
> network looks like.  active/passive won't give you any performance gains,
> but can simplify the configuration, while providing the HA you seem to be
> after.  As Kaia pointed out, what the traffic looks like (reads vs writes)
> is a consideration.  If it is something that users don't write to, and data
> doesn't have to be replicated across DCs frequently, this further simplifies
> things.
>
> Ultimately, the configuration will depend on what the application and
> network looks like currently, and what level of redundancy you want / need.
>
> -- Dan.
>
> On Wed, May 19, 2010 at 1:40 PM, Matt Iavarone <matt.iavarone at gmail.com <
> /mc/compose?to=matt.iavarone at gmail.com> > wrote:
>
> I think the original question was around stateless load balancing, not
> clustering.  Cross DC clustering is a headache, but HA web sites aren't
> exactly terchnical challenges these days.
>
> On May 19, 2010 4:33 PM, "Alex Dean" <alex at crackpot.org <
> /mc/compose?to=alex at crackpot.org> > wrote:
>
>
> On May 19, 2010, at 2:47 PM, keith smith wrote:
>
> >
> >
> > Hi Plug,
> >
> > I am considering combining the ... You're entering a world of pain. :)
>
> HA is cool, but is no panacea.  If you haven't actually experienced
> downtime due to your server crashing or your datacenter losing connectivity,
> I recommend thinking long and hard about it.  Don't solve a problem you
> don't have.  The downtime created from unneeded failovers will likely exceed
> the actual/real downtime caused by either a server or datacenter being
> offline.  Managing the cluster itself (as distinct from the services
> provided by the cluster) needs to be accounted for as an
> expense/responsibility.
>
> I don't want to sound overly pessimistic.  I've set up quite a few HA
> clusters, and actually enjoy it most of the time.  But it WILL cause you
> headaches in the middle of the night which you wouldn't have had if you only
> had a single server.
>
> Leave yourself lots of time to set up a development/test cluster, and abuse
> it in many ways.  Pull out network cables, kill the switch, yank out power
> cables, etc.  Do this with real hardware, not VMs.
>
> When the cluster nodes lose contact with each other, both will decide to
> become primary.  This is a split brain.  This can happen when the switch
> in-between them gets busy and starts dropping pings.  Now, you can always
> recover from such things.  I'm just recommending you become very familiar
> with these issues before going live with this setup.
>
> http://clusterlabs.org/wiki/Main_Page
> http://people.linbit.com/~florian/heartbeat-users-guide/<http://people.linbit.com/%7Eflorian/heartbeat-users-guide/><
> http://people.linbit.com/%7Eflorian/heartbeat-users-guide/>
>
> Let me/us know if you have specific questions once you start setting things
> up.  Good luck!
>
> alex
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us <
> /mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us>
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us <
> /mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us>
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
>
>
> -----Inline Attachment Follows-----
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us <
> /mc/compose?to=PLUG-discuss at lists.plug.phoenix.az.us>
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
>
>
> ------------------------------
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
>
> -----Inline Attachment Follows-----
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us<http://mc/compose?to=PLUG-discuss@lists.plug.phoenix.az.us>
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
>
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.PLUG.phoenix.az.us/pipermail/plug-discuss/attachments/20100519/35691a24/attachment.htm>