Re: load balanced configuration

Top Page
This message is part of the following thread:
Mthe complete thread tree sorted by date
MDan Dubovik at <-
MEd Knapp at ->
Attachments:
Message as email
+ (text/plain)
+ (text/html)
+ (text/plain)
Delete this message
Reply to this message
Author: Bryan O'Neal
Date:  
To: Main PLUG discussion list
Subject: Re: load balanced configuration
This is very true - but in practice this is not as common as you think. Our
studies using DynDNS showed that Cox, Time Warner, Verizon, Qwest, Yahoo
(Via dns change not ISP hosting), and MegaPath did not cache, at least not
more then a few min. I was unable to find anyone actually on AOL or EathLink
or Comcast to test with. I did have a person on some small ISP like
RoadRunner and they did cache.

Mind you our studies were not very scientific - call person we know on
network have them hit site, change dns, have them hit site again. For some
providers we only had one or two people using them. And for all the Yahoo
based ISP's we just changed our primary DNS and tested ourselves. But, what
we found supported our experience in that it was not a big issue.

On Wed, May 19, 2010 at 8:09 PM, Dan Dubovik <> wrote:

>
>
>
>
> On Wed, May 19, 2010 at 7:57 PM, keith smith <>wrote:
>
>>
>> This is kind of fizzy to me. I'm glad you brought it up. I did
>> experience this 6 to 9 month ago when the data center chanced the NIC card.
>> I think they had to flush some buffers in their routers so the new MAC
>> address could be found and cached if I recall correctly.
>>
>
> They likely had to clear the ARP cache.
>
>
>>
>> We are in a data center and use their DNS. So I'm thinking the request
>> goes to the root server then to the data center's DNS and it tells the
>> client what the IP address is. So if the Data Center's DNS is changed to
>> point to a new IP for our domain then that would be instantaneous or would
>> the client and everyone along the way cache the IP?
>>
>>
> The A Record for your site can still be cached for up to the TTL time by
> local dns services. Thus, the following can occur:
>
> 1) Client visits site
> 2) ISP cache's IP address after it does the initial look up
> 3) Something bad happens to the data center
> 4) IP address is updated
> 5) Client goes to visit site again
> 6) Client's ISP realizes they have visited that site recently, and provides
> the cached answer for the DNS response.
> 7) Clients browser / application is forwarded to the old IP address, and
> encounters an error.
>
>
>
>
>>
>> ------------------------
>> Keith Smith
>>
>> --- On *Wed, 5/19/10, Ed Knapp <>* wrote:
>>
>>
>> From: Ed Knapp <>
>>
>> Subject: Re: load balanced configuration
>> To: "Main PLUG discussion list" <>
>> Date: Wednesday, May 19, 2010, 6:09 PM
>>
>>
>> One thing struck me here with your description...
>>
>> “and a change to the DNS and we are off and running”
>>
>> While your DNS records might be changed relatively quickly during an
>> incident, the change
>> Itself can take quite a while to trickle down to the end users/clients out
>> in the cloud.
>> Any client’s DNS resolution that has not expired in the cache nor manually
>> refreshed will
>> still fail to properly resolve/connect. It doesn’t usually, but I tell
>> clients to plan for 48 hours
>> Estimated time for the change to completely propagate.
>>
>> I would hate for you to get blindsided with a person hovering over you
>> asking how much longer
>> It is going to take before the site is back up and operational. It is
>> frustrating when you have
>> Fixed the issue [ problem :-) ] but have to just sit and wait for it to
>> complete.
>>
>> There are certainly strategies to mitigate this risk and I do not know if
>> you maintain your
>> Own DNS servers or do you work through a hosting provider/domain
>> registrar.
>>
>> I hope this helps a bit.
>>
>> Ed
>>
>>
>> On 5/19/10 2:07 PM, "keith smith" <> wrote:
>>
>>
>>
>> Currently we have two servers in our main data center. One serves our
>> shopping cart. The other contains quite a bit of content that is data
>> driven (reads). The content site is very active. The orders on the
>> shopping cart are spread apart by one or two minutes during the busiest part
>> of the day. We store a lot of data with each order so most of this is
>> writing. The shopping cart is backed up to the server in the other data
>> center. Supposedly if there is a problem, a few things need to be done to
>> the backup server in preparation to make it live, and a change to the DNS
>> and we are off and running.
>>
>> The problem I am trying to solve is the other server (content site) is not
>> currently backed up automatically.
>>
>> Another layer of this is these are managed servers. We have an excellent
>> relationship with the data center owner and have 24/7 access to him and his
>> staff. He manages all three servers and has always done a good job.
>>
>> I am the one tasked with keeping our sites online 24/7.
>>
>> I was hoping by configuring two servers, each in a different location,
>> that, in the event of one of the data centers being completely severed from
>> the Internet that the other server would automatically, without any human
>> intervention, take over the full load of the other server and those visiting
>> either of our sites would not know there had been an issue.
>>
>> In a nutshell I am trying to create an automated backup that is a
>> automated fail over solution.
>>
>> I appreciate all your feedback!
>>
>> ------------------------
>> Keith Smith
>>
>> --- On *Wed, 5/19/10, Dan Dubovik <>* wrote:
>>
>>
>> From: Dan Dubovik <>
>> Subject: Re: load balanced configuration
>> To: "Main PLUG discussion list" <>
>> Date: Wednesday, May 19, 2010, 1:45 PM
>>
>> The question I have, are you trying to actually load balance things? Or
>> just have a remote location that you can fire up with live data at a moments
>> notice? Basically, are you wanting an active/active configuration, or
>> active/passive?
>> active/active across DC's can get kind of hairy depending on what the
>> network looks like. active/passive won't give you any performance gains,
>> but can simplify the configuration, while providing the HA you seem to be
>> after. As Kaia pointed out, what the traffic looks like (reads vs writes)
>> is a consideration. If it is something that users don't write to, and data
>> doesn't have to be replicated across DCs frequently, this further simplifies
>> things.
>>
>> Ultimately, the configuration will depend on what the application and
>> network looks like currently, and what level of redundancy you want / need.
>>
>> -- Dan.
>>
>> On Wed, May 19, 2010 at 1:40 PM, Matt Iavarone < <
>> /mc/compose?to=> > wrote:
>>
>> I think the original question was around stateless load balancing, not
>> clustering. Cross DC clustering is a headache, but HA web sites aren't
>> exactly terchnical challenges these days.
>>
>> On May 19, 2010 4:33 PM, "Alex Dean" < <
>> /mc/compose?to=> > wrote:
>>
>>
>> On May 19, 2010, at 2:47 PM, keith smith wrote:
>>
>> >
>> >
>> > Hi Plug,
>> >
>> > I am considering combining the ... You're entering a world of pain. :)
>>
>> HA is cool, but is no panacea. If you haven't actually experienced
>> downtime due to your server crashing or your datacenter losing connectivity,
>> I recommend thinking long and hard about it. Don't solve a problem you
>> don't have. The downtime created from unneeded failovers will likely exceed
>> the actual/real downtime caused by either a server or datacenter being
>> offline. Managing the cluster itself (as distinct from the services
>> provided by the cluster) needs to be accounted for as an
>> expense/responsibility.
>>
>> I don't want to sound overly pessimistic. I've set up quite a few HA
>> clusters, and actually enjoy it most of the time. But it WILL cause you
>> headaches in the middle of the night which you wouldn't have had if you only
>> had a single server.
>>
>> Leave yourself lots of time to set up a development/test cluster, and
>> abuse it in many ways. Pull out network cables, kill the switch, yank out
>> power cables, etc. Do this with real hardware, not VMs.
>>
>> When the cluster nodes lose contact with each other, both will decide to
>> become primary. This is a split brain. This can happen when the switch
>> in-between them gets busy and starts dropping pings. Now, you can always
>> recover from such things. I'm just recommending you become very familiar
>> with these issues before going live with this setup.
>>
>> http://clusterlabs.org/wiki/Main_Page
>> http://people.linbit.com/~florian/heartbeat-users-guide/<http://people.linbit.com/%7Eflorian/heartbeat-users-guide/><
>> http://people.linbit.com/%7Eflorian/heartbeat-users-guide/>
>>
>> Let me/us know if you have specific questions once you start setting
>> things up. Good luck!
>>
>> alex
>> ---------------------------------------------------
>> PLUG-discuss mailing list - <
>> /mc/compose?to=>
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>>
>> ---------------------------------------------------
>> PLUG-discuss mailing list - <
>> /mc/compose?to=>
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>>
>>
>> -----Inline Attachment Follows-----
>>
>> ---------------------------------------------------
>> PLUG-discuss mailing list - <
>> /mc/compose?to=>
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>>
>>
>> ------------------------------
>> ---------------------------------------------------
>> PLUG-discuss mailing list -
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>>
>> -----Inline Attachment Follows-----
>>
>> ---------------------------------------------------
>> PLUG-discuss mailing list - <http://mc/compose?to=PLUG-discuss@lists.plug.phoenix.az.us>
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>>
>>
>> ---------------------------------------------------
>> PLUG-discuss mailing list -
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>
>
> ---------------------------------------------------
> PLUG-discuss mailing list -
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
---------------------------------------------------
PLUG-discuss mailing list -
To subscribe, unsubscribe, or to change your mail settings:
http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
This message was posted to the following mailing lists:
Plug Discuss
Mailing List Info | Nearby Messages		<-Re: load balanced configurationRe: load balanced configuration->
Some Mailing List Archive administrated by UnconfiguredLurker (version 2.3)