Don't get me wrong, I'd love to fund good HA module for FS, if not for any other reason that I could benefit from that. But having done a few installations of systems that were supposed to be "HA" and seen them fail when real problems came I know it ain't easy. Redundant networks are fine but following scenarios usually lead to both machines reply to ARPs for virtual IP and whole HA falls apart: 1) FS stops responding (e.g. due to heavy swapping or disk full), yet kernel manages to reply to ARPs 2) the HA module fails (like in crashes) but FS manages to work 3) some firewall rule is activated that stops multicast traffic (all or some) STONITH based on separate technology (like USB-USB connection connected to some KVM-over-IP with control over power) is indispensable in such scenarios. 2013/2/10 Eliot Gable <egable+freeswitch@gmail.com>:
On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
Steve,
As Avi pointed out, I mentioned having multiple physical networks as a guard against a network split / partition. If one network is split such that A and B can talk to each other over it and C and D can talk to each other over it, you would indeed have an issue if you only had one network. However, with two or more networks, all four nodes will still be able to talk to each other over the other network(s).
Now, granted, if you have a network split in all networks, then you are still screwed. Pacemaker and other solutions deal with this, as you mentioned, using something called "quorum" where you need a majority of nodes to be able to see each other, and they fence the remaining nodes. As I documented on my wiki page for the module, I do have plans to eventually support such functionality. However, that is a bit further down the road as it will take some time to develop STONITH interfaces to various hardware or even to reuse the STONITH modules from Pacemaker or another project. In any case, I feel it is more important to get the base functionality developed and debugged as utilizing multiple networks is a good way to prevent network splits from being an issue.
That being said, there are other issues to contend with when discussing network splits. For example, if A and B can see the Internet but C and D cannot, but C is a Master and B is a slave, you still have an issue to address. In this case, mod_ha_cluster must be able to determine that C and D cannot see the Internet. They need to perform very fast pings to some IP address, or have some external host sending them data in some way that they can detect when traffic to/from the Internet has stopped. I can place a media bug on the audio streams to make this determination fairly accurately. I can also rely on a ping mechanism to make the determination. Once the determination is made, mod_ha_cluster then has to promote B to a master to take over C.
So, there are certainly still other issues to address when a network split occurs, but split-brain is easily avoided by simply adding redundant networks.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik