One thing I tell clients - if you need more than a gig, get a 10gbe interface. That comes with it's own challenges too, see if you can get it to use all of that 10gbe...

The issue you face with using a 802.3ad bond is the flow-hashing. You're using a l2 policy, which I presume to mean "mac source/dst flow-hashing". Basically, if all your communications is outside the subnet, you're hashing to one mac, your default gateway, that doesn't work well for distributing traffic. Use a L3-based policy, including source/dst ports in tuples, that is what makes switches effective.

Sadly, you also have about the worst server switch ever created, namely a legacy cat4k. Those if you look into the architecture, they're 6gbe backplane, spread among 8 port groups (6x8=48 ports), that every 1 gbe ports share 1x 1gbe asic interconnect to the "fabric" (if you can call it that in 2015). At least get something modern with big buffers on 1gbe ports, like an Arista 7048T, or if you must stay cisco, nexus 3k/5k and up.

Those 4k's suck because for the oversubscription, make sure your two server ports are spread between two asic groups if you want max performance. Putting 8 hosts trying to talk a gig each will just slam the asic into 8:1 oversubscription, they'll all get some fraction of 1gbe shared per 8 ports (iperf this if you don't believe me, I have). Also, same as your server, make sure you're using "etherchannel load-balance src-dst-mixed-ip-port" for the most entropy for distribution of flows at a L3 level using ip, port, src/dst as tuples for distribution among your downstream (and upstream) paths.

Your network engineer should do that anyways, if not, spank him to buy something outside a ccna-level book or look up the command.

If you have one source, to one dest, or large "elephant herding" flows like single filer connections, flow-distribution does little to nothing to help. Goto first paragraph, get a 10gbe nic. :)

-mb

On 04/22/2015 02:42 PM, Stephen Partington wrote:

I am working on a ProxmoxVE cluster i have set up.

I am needing a bit better network performance as i am also running CEPH for
the stoage layer

This is what i have for network configuration is the following. it seems to
be working. the nodes i have configured appear to be running with better
throughput.

root@computername:~# cat /etc/network/interfaces
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto bond0
iface bond0 inet static
        address 10.40.216.235
        netmask 255.255.255.0
        slaves eth0 eth1
        bond_miimon 100
        bond_mode 802.3ad
        bond_xmit_hash_policy layer2

auto vmbr0
iface vmbr0 inet static
        address 10.40.216.235
        netmask 255.255.255.0
        gateway 10.40.216.1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0

root@computername:~#

The backend network these servers are connected to are a Cisco Catalyst
4705R loaded with WS-X4524-GB-RJ45V modules.

All my research says to use a network bond running LCAP for best
compatability/performance with this hardware. it all seems to be running,
but it is kind of weird that Proxmox would want me to create the bridge for
the VMs to run on kind of makes sense just feels weird to run a bond
inside bridge.

If anyone who has worked with proxmox has a better suggestion please let me
know.

Thanks for your time.

--
A mouse trap, placed on top of your alarm clock, will prevent you from
rolling over and going back to sleep after you hit the snooze button.

Stephen
---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
http://lists.phxlinux.org/mailman/listinfo/plug-discuss