[Freeswitch-users] High Availability Cluster Module for FreeSWITCH

[Steven Ayre]

>
> One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair.
> That way if any single node fails, there's something to pick up the slack.


Corosync clusters are not limited to active/passive pairs. It's just a very
common setup.

For example you could have resource agents 1) to keep FS running on all
nodes 2) for virtual IPs 3) for IP:port Sofia profiles. You can then define
dependancies between them. That should let you keep FS running at all times
and move an IP and the associated Sofia profiles to a new node that's
already running FS when the original node fails. For maintance you can
simply trigger that from the CRM.

Secondly, in order to recover live calls, you need a list of the calls.
> That currently requires some sort of odbc (or postgres) with replication.
> Again, that's abstracted as part of mod_ha_cluster.
> Third: The docs mention a similar of pooling for registration, that you
> can register to one server and you're regged on them all without needing a
> DB to sync everything.


Which can also be done using Corosync's IPC messaging API.

(Personally I prefer using MySQL Cluster via ODBC - which is
HA, synchronous and offloads all load off of the FS nodes, but that's
offtopic).

Fourth, according to the docs: single configuration for all FS instances,
> rather than manually ensuring each one has the same config.


This could be achieved with the Corosync API.

Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs
> to be rewritten for that, perhaps...


That's not going to happen without a storage API added to the FS core -
you're always going to need some 3rd party such NFS, CIFS, DRBD etc.
mod_voicemail is hardcoded to use the ODBC interface but that's only for
the index not the recordings.nc

Corosync would allow you to make FS depend on the NFS/whatever service
running so if the storage backend has failed FS would move to a node where
it is available - not necessarily possibly from within mod_ha_cluster
itself.

There's certainly *something *special possible with mod_ha_cluster that
> can't be done with existing solutions cleanly, if at all...


Don't misunderstand me, I think it's a great idea to have a module aimed at
HA and sharing state across a cluster while being able to detect new
failure conditions.

I just think that in the specific area of node monitoring and messaging
across the cluster it would be better to use a well tested and proven
solution such as Corosync which is based on a large number of papers,
algorithms, and generally decades of work. Every time I've seen/heard of an
attempt to redo that from scratch it's been unreliable especially in
unexpected failure conditions. Simply because it's a dependency on another
program isn't a good enough reason not to use it - you're already depending
on various other programs (Linux, sysvinit, monit, cron, syslog etc)
anyway. By not using it you're just adding extra work, adding unnecessarily
complexity and increasing the risk of bugs. Corosync would also have
advantages because the tools to migrate services to another node
for maintenance, detect and restart resources on failure etc already exist.

-Steve





On 10 February 2013 20:40, Avi Marcus <avi at avimarcus.net> wrote:

> Steve:
> Several reasons.
> One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair.
> That way if any single node fails, there's something to pick up the slack.
>
> Secondly, in order to recover live calls, you need a list of the calls.
> That currently requires some sort of odbc (or postgres) with replication.
> Again, that's abstracted as part of mod_ha_cluster.
>
> Third: The docs mention a similar of pooling for registration, that you
> can register to one server and you're regged on them all without needing a
> DB to sync everything.
>
> Fourth, according to the docs: single configuration for all FS instances,
> rather than manually ensuring each one has the same config.
>
> Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's
> APIs to be rewritten for that, perhaps...
>
> There's certainly *something *special possible with mod_ha_cluster that
> can't be done with existing solutions cleanly, if at all...
>
> -Avi
>
> On Sun, Feb 10, 2013 at 10:16 PM, Steven Ayre <steveayre at gmail.com> wrote:
>
>> Fair enough.
>>
>> I still don't see the need to reinvent the wheel just to avoid depending
>> on a 3rd party well developed and extensively tested piece of software.
>> Corosync provides an API that can be used for passing messages between
>> nodes in a cluster.
>>
>> -Steve
>>
>>
>>
>> On 10 February 2013 18:14, Eliot Gable <egable+freeswitch at gmail.com>wrote:
>>
>>> On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre at gmail.com>
>>> wrote:
>>> > That covers redundancy in case of a network card or cable failure, but
>>> isn't
>>> > what partitioning is about. Multiple NICs cannot prevent partitioning.
>>> >
>>> > As an example, partitioning might happen when a network switch between
>>> two
>>> > network segments fails so you have nodes A+B in segment 1 able to talk
>>> to
>>> > each other but unable to talk to nodes C+D in segment 2, while C+D can
>>> talk
>>> > to each other but not A+B.
>>> >
>>> > Pacemaker/corosync contain a lot of algorithms to fence off partitions
>>> > without quorum and can resort to things like STONITH if required to
>>> force a
>>> > node to shutdown rather than risk it causing disruption to the cluster
>>> (for
>>> > example if it tries to take over traffic to a virtual IP you could end
>>> up in
>>> > a case where you have two servers sending ARP responses for the same
>>> IP).
>>> >
>>>
>>> Steve,
>>>
>>> As Avi pointed out, I mentioned having multiple physical networks as a
>>> guard against a network split / partition. If one network is split
>>> such that A and B can talk to each other over it and C and D can talk
>>> to each other over it, you would indeed have an issue if you only had
>>> one network. However, with two or more networks, all four nodes will
>>> still be able to talk to each other over the other network(s).
>>>
>>> Now, granted, if you have a network split in all networks, then you
>>> are still screwed. Pacemaker and other solutions deal with this, as
>>> you mentioned, using something called "quorum" where you need a
>>> majority of nodes to be able to see each other, and they fence the
>>> remaining nodes. As I documented on my wiki page for the module, I do
>>> have plans to eventually support such functionality. However, that is
>>> a bit further down the road as it will take some time to develop
>>> STONITH interfaces to various hardware or even to reuse the STONITH
>>> modules from Pacemaker or another project. In any case, I feel it is
>>> more important to get the base functionality developed and debugged as
>>> utilizing multiple networks is a good way to prevent network splits
>>> from being an issue.
>>>
>>> That being said, there are other issues to contend with when
>>> discussing network splits. For example, if A and B can see the
>>> Internet but C and D cannot, but C is a Master and B is a slave, you
>>> still have an issue to address. In this case, mod_ha_cluster must be
>>> able to determine that C and D cannot see the Internet. They need to
>>> perform very fast pings to some IP address, or have some external host
>>> sending them data in some way that they can detect when traffic
>>> to/from the Internet has stopped. I can place a media bug on the audio
>>> streams to make this determination fairly accurately. I can also rely
>>> on a ping mechanism to make the determination. Once the determination
>>> is made, mod_ha_cluster then has to promote B to a master to take over
>>> C.
>>>
>>> So, there are certainly still other issues to address when a network
>>> split occurs, but split-brain is easily avoided by simply adding
>>> redundant networks.
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130210/71d7bb26/attachment.html