[Freeswitch-users] High Availability Cluster Module for FreeSWITCH

[Marcin Gozdalik]

+1

I do not doubt mod_ha is necessary inside of FS  and it may be
better/simpler than writing Pacemaker resource agent, but writing
yet-another-cluster-communication-engine is IMHO the wrong way to go
and using Corosync for communication will give a lot of value from
mature codebase.

2013/2/10 Steven Ayre <steveayre at gmail.com>:
>> One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair.
>> That way if any single node fails, there's something to pick up the slack.
>
>
> Corosync clusters are not limited to active/passive pairs. It's just a very
> common setup.
>
> For example you could have resource agents 1) to keep FS running on all
> nodes 2) for virtual IPs 3) for IP:port Sofia profiles. You can then define
> dependancies between them. That should let you keep FS running at all times
> and move an IP and the associated Sofia profiles to a new node that's
> already running FS when the original node fails. For maintance you can
> simply trigger that from the CRM.
>
>> Secondly, in order to recover live calls, you need a list of the calls.
>> That currently requires some sort of odbc (or postgres) with replication.
>> Again, that's abstracted as part of mod_ha_cluster.
>> Third: The docs mention a similar of pooling for registration, that you
>> can register to one server and you're regged on them all without needing a
>> DB to sync everything.
>
>
> Which can also be done using Corosync's IPC messaging API.
>
> (Personally I prefer using MySQL Cluster via ODBC - which is HA, synchronous
> and offloads all load off of the FS nodes, but that's offtopic).
>
>> Fourth, according to the docs: single configuration for all FS instances,
>> rather than manually ensuring each one has the same config.
>
>
> This could be achieved with the Corosync API.
>
>> Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's
>> APIs to be rewritten for that, perhaps...
>
>
> That's not going to happen without a storage API added to the FS core -
> you're always going to need some 3rd party such NFS, CIFS, DRBD etc.
> mod_voicemail is hardcoded to use the ODBC interface but that's only for the
> index not the recordings.nc
>
> Corosync would allow you to make FS depend on the NFS/whatever service
> running so if the storage backend has failed FS would move to a node where
> it is available - not necessarily possibly from within mod_ha_cluster
> itself.
>
>> There's certainly something special possible with mod_ha_cluster that
>> can't be done with existing solutions cleanly, if at all...
>
>
> Don't misunderstand me, I think it's a great idea to have a module aimed at
> HA and sharing state across a cluster while being able to detect new failure
> conditions.
>
> I just think that in the specific area of node monitoring and messaging
> across the cluster it would be better to use a well tested and proven
> solution such as Corosync which is based on a large number of papers,
> algorithms, and generally decades of work. Every time I've seen/heard of an
> attempt to redo that from scratch it's been unreliable especially in
> unexpected failure conditions. Simply because it's a dependency on another
> program isn't a good enough reason not to use it - you're already depending
> on various other programs (Linux, sysvinit, monit, cron, syslog etc) anyway.
> By not using it you're just adding extra work, adding unnecessarily
> complexity and increasing the risk of bugs. Corosync would also have
> advantages because the tools to migrate services to another node for
> maintenance, detect and restart resources on failure etc already exist.
>
> -Steve
>
>
>
>
>
> On 10 February 2013 20:40, Avi Marcus <avi at avimarcus.net> wrote:
>>
>> Steve:
>> Several reasons.
>> One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair.
>> That way if any single node fails, there's something to pick up the slack.
>>
>> Secondly, in order to recover live calls, you need a list of the calls.
>> That currently requires some sort of odbc (or postgres) with replication.
>> Again, that's abstracted as part of mod_ha_cluster.
>>
>> Third: The docs mention a similar of pooling for registration, that you
>> can register to one server and you're regged on them all without needing a
>> DB to sync everything.
>>
>> Fourth, according to the docs: single configuration for all FS instances,
>> rather than manually ensuring each one has the same config.
>>
>> Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's
>> APIs to be rewritten for that, perhaps...
>>
>> There's certainly something special possible with mod_ha_cluster that
>> can't be done with existing solutions cleanly, if at all...
>>
>> -Avi
>>
>> On Sun, Feb 10, 2013 at 10:16 PM, Steven Ayre <steveayre at gmail.com> wrote:
>>>
>>> Fair enough.
>>>
>>> I still don't see the need to reinvent the wheel just to avoid depending
>>> on a 3rd party well developed and extensively tested piece of software.
>>> Corosync provides an API that can be used for passing messages between nodes
>>> in a cluster.
>>>
>>> -Steve
>>>
>>>
>>>
>>> On 10 February 2013 18:14, Eliot Gable <egable+freeswitch at gmail.com>
>>> wrote:
>>>>
>>>> On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre at gmail.com>
>>>> wrote:
>>>> > That covers redundancy in case of a network card or cable failure, but
>>>> > isn't
>>>> > what partitioning is about. Multiple NICs cannot prevent partitioning.
>>>> >
>>>> > As an example, partitioning might happen when a network switch between
>>>> > two
>>>> > network segments fails so you have nodes A+B in segment 1 able to talk
>>>> > to
>>>> > each other but unable to talk to nodes C+D in segment 2, while C+D can
>>>> > talk
>>>> > to each other but not A+B.
>>>> >
>>>> > Pacemaker/corosync contain a lot of algorithms to fence off partitions
>>>> > without quorum and can resort to things like STONITH if required to
>>>> > force a
>>>> > node to shutdown rather than risk it causing disruption to the cluster
>>>> > (for
>>>> > example if it tries to take over traffic to a virtual IP you could end
>>>> > up in
>>>> > a case where you have two servers sending ARP responses for the same
>>>> > IP).
>>>> >
>>>>
>>>> Steve,
>>>>
>>>> As Avi pointed out, I mentioned having multiple physical networks as a
>>>> guard against a network split / partition. If one network is split
>>>> such that A and B can talk to each other over it and C and D can talk
>>>> to each other over it, you would indeed have an issue if you only had
>>>> one network. However, with two or more networks, all four nodes will
>>>> still be able to talk to each other over the other network(s).
>>>>
>>>> Now, granted, if you have a network split in all networks, then you
>>>> are still screwed. Pacemaker and other solutions deal with this, as
>>>> you mentioned, using something called "quorum" where you need a
>>>> majority of nodes to be able to see each other, and they fence the
>>>> remaining nodes. As I documented on my wiki page for the module, I do
>>>> have plans to eventually support such functionality. However, that is
>>>> a bit further down the road as it will take some time to develop
>>>> STONITH interfaces to various hardware or even to reuse the STONITH
>>>> modules from Pacemaker or another project. In any case, I feel it is
>>>> more important to get the base functionality developed and debugged as
>>>> utilizing multiple networks is a good way to prevent network splits
>>>> from being an issue.
>>>>
>>>> That being said, there are other issues to contend with when
>>>> discussing network splits. For example, if A and B can see the
>>>> Internet but C and D cannot, but C is a Master and B is a slave, you
>>>> still have an issue to address. In this case, mod_ha_cluster must be
>>>> able to determine that C and D cannot see the Internet. They need to
>>>> perform very fast pings to some IP address, or have some external host
>>>> sending them data in some way that they can detect when traffic
>>>> to/from the Internet has stopped. I can place a media bug on the audio
>>>> streams to make this determination fairly accurately. I can also rely
>>>> on a ping mechanism to make the determination. Once the determination
>>>> is made, mod_ha_cluster then has to promote B to a master to take over
>>>> C.
>>>>
>>>> So, there are certainly still other issues to address when a network
>>>> split occurs, but split-brain is easily avoided by simply adding
>>>> redundant networks.
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>



--
Marcin Gozdalik