Clustering

[Stephen]

SuperMicro has a number of servers built with infiniband on-baord in
1u, twin 1u and the like... not sureif they have onbaord infiniband
with 4 pci-e 16x slots...

On Tue, Jun 30, 2009 at 1:14 AM, Michael Butash<michael at butash.net> wrote:
> Exactly as Nadeem said, start local, and if you need to expand, look at
> infiniband.  Buying IB gear directly from the vendor, supported,
> warranted, etc, will get expensive quick.  You'll probably find
> programming info fairly obscure, as most of this is very proprietary, as
> supercomputing is a cash cow to certain vars.
>
> Obscurity seems to make it cheap on the secondary market, as most app
> dev's won't/don't take it to that level, and not many really need it.
> Tesla's change things as they provide an extensible way to increase
> generic crunching local, and can scale significantly with pcie bandwith
> and hardware with enough slots.
>
> For me IB was interesting as I was investigating upping my house to
> gige, fiber-channel, and other things to geek out on.  IB actually
> looked fairly promising as it's device stack allows for IP/FC
> emululation layer, as well as it's own native socket stack where
> parallel processing plays into.  Then I found out it was cheaper than
> buying decent 4g fiber channel gear off ebay, and got even more
> interesting.  Might provide some ideal scalability if you need to expand
> the processing outside of one box eventually.
>
> -mb
>
>
> On Tue, 2009-06-30 at 00:35 -0700, Stephen wrote:
>> Are you looking at the 1u Tesla or the x58 personal supercomputer they
>> are talking about...
>>
>> I would look into RH mostly for compatability fo ahrdware as it is
>> tested by them, and for most packages... but the distro is really
>> dependant on your app.
>>
>> Man they have really changed tesla, it used to be a bunch of gpu cards
>> in a box chained to a header and then 16 of those can be tied together
>> to a single machine.
>>
>> but now you can buy a preconfigured clsuter
>>
>> http://www.penguincomputing.com/products/linux/server/web_promotion/tesla_workgroup_cluster
>>
>> man i stop following for just 1 year...
>>
>>
>> On Mon, Jun 29, 2009 at 6:29 PM, Eric Shubert<ejs at shubes.net> wrote:
>> > I'm looking at clustering together a handful of hosts, each running dual
>> > nvidia tesla cards. Modeling applications of some sort. I honestly don't
>> > know much more than that.
>> >
>> > Michael Butash wrote:
>> >> You're probably talking infiniband switching, infiniband hba's,
>> >> pci-e/htx interfaces, fiber channel disk arrays, etc.  Linux seems to
>> >> support infiniband hba's reasonably well, and 10g 4x infiniband hba's
>> >> tend to be cheap these days on ebay.  We're talking $100 used hba's for
>> >> the nodes, and ~$1200 for a 12 port Cisco/Topspin switch.  I thought
>> >> about buying some to play with, as it ends up cheaper than IP or Fiber
>> >> channel technologies, yet can replace them all to some extents.
>> >>
>> >> IB is quite versatile, emulating fiber-channel, IP network, or raw
>> >> interrupt switching to a cpu and memory via different driver socket
>> >> interface api's.  Cray always used a similar means to make theirs with
>> >> proprietary north bridge and software, but IB is more of a standard now,
>> >> enabling (relatively) cheap supercomputing on the fly with commodity
>> >> hardware.  Well, hardware-wise at least...
>> >>
>> >> So yeah, what apps are you talking about utilizing it?
>> >>
>> >> -mb
>> >>
>> >>
>> >> On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote:
>> >>> From: Eric Shubert <ejs at shubes.net>
>> >>>> Has anyone here implemented any clusters?
>> >>> I've only set one up, but I maintain the ones that my predecessors
>> >>> set up.  It's not rocket science.
>> >>>
>> >>>> Is any particular distro better or worse at clustering?
>> >>> Not really.  Every distro has heartbeat/DRBD/LVS available.
>> >>>
>> >>>> Any pointers regarding clustering you'd like to share?
>> >>> Define the problem you're trying to solve more rigorously than just
>> >>> "clustering" first.  Do you want flailover between 2 boxes?  Do you
>> >>> want a frontend box with N service-providing boxes behind it?  The
>> >>> answers to that greatly affect what you will end up doing, as does
>> >>> the question "What services is this cluster going to provide?"
>> >>>
>> >>> Basically, all I can do is handwave without answers to "what services?"
>> >>> and "how many machines?".
>> >>>
>> >>
>> >
>> >
>> > --
>> > -Eric 'shubes'
>> >
>> > ---------------------------------------------------
>> > PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>> > To subscribe, unsubscribe, or to change your mail settings:
>> > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>> >
>>
>>
>>
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>



-- 
A mouse trap, placed on top of your alarm clock, will prevent you from
rolling over and going back to sleep after you hit the snooze button.

Stephen