<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack.</blockquote>
<div><br></div><div>Corosync clusters are not limited to active/passive pairs. It's just a very common setup.</div><div><br></div>For example you could have resource agents 1) to keep FS running on all nodes 2) for virtual IPs 3) for IP:port Sofia profiles. You can then define dependancies between them. That should let you keep FS running at all times and move an IP and the associated Sofia profiles to a new node that's already running FS when the original node fails. For maintance you can simply trigger that from the CRM.<div>
<br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Secondly, in order to recover live calls, you need a list of the calls. That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster.<br>
Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.</blockquote></div><div><br></div><div>Which can also be done using Corosync's IPC messaging API.</div>
<div><br></div><div>(Personally I prefer using MySQL Cluster via ODBC - which is HA, synchronous and offloads all load off of the FS nodes, but that's offtopic).</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Fourth, according to the docs: single configuration for all FS instances, rather than manually ensuring each one has the same config.</blockquote><div><br></div><div>This could be achieved with the Corosync API.</div><div>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs to be rewritten for that, perhaps...</blockquote>
<div><br></div>That's not going to happen without a storage API added to the FS core - you're always going to need some 3rd party such NFS, CIFS, DRBD etc. mod_voicemail is hardcoded to use the ODBC interface but that's only for the index not the <a href="http://recordings.nc" target="_blank">recordings.nc</a><div>
<br></div><div>Corosync would allow you to make FS depend on the NFS/whatever service running so if the storage backend has failed FS would move to a node where it is available - not necessarily possibly from within mod_ha_cluster itself.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">There's certainly <i>something </i>special possible with mod_ha_cluster that can't be done with existing solutions cleanly, if at all...</blockquote>
<div><br></div><div>Don't misunderstand me, I think it's a great idea to have a module aimed at HA and sharing state across a cluster while being able to detect new failure conditions.</div>
<div><br></div><div>I just think that in the specific area of node monitoring and messaging across the cluster it would be better to use a well tested and proven solution such as Corosync which is based on a large number of papers, algorithms, and generally decades of work. Every time I've seen/heard of an attempt to redo that from scratch it's been unreliable especially in unexpected failure conditions. Simply because it's a dependency on another program isn't a good enough reason not to use it - you're already depending on various other programs (Linux, sysvinit, monit, cron, syslog etc) anyway. By not using it you're just adding extra work, adding unnecessarily complexity and increasing the risk of bugs. Corosync would also have advantages because the tools to migrate services to another node for maintenance, detect and restart resources on failure etc already exist.</div>
<div><br></div><div>-Steve</div><div><br></div><div><br></div><div><br></div><div><br> <br><div class="gmail_quote">On 10 February 2013 20:40, Avi Marcus <span dir="ltr"><<a href="mailto:avi@avimarcus.net" target="_blank">avi@avimarcus.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Steve:<div>Several reasons.</div><div>One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack.</div>
<div><br></div>
<div>Secondly, in order to recover live calls, you need a list of the calls. That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster.</div><div><br>
</div><div>Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.</div><div><br></div><div>Fourth, according to the docs: single configuration for all FS instances, rather than manually ensuring each one has the same config.</div>
<div><br></div><div>Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs to be rewritten for that, perhaps...</div><div><br></div><div>There's certainly <i>something </i>special possible with mod_ha_cluster that can't be done with existing solutions cleanly, if at all...</div>
<span><font color="#888888">
<div><br></div><div><span style="font-family:Verdana,Arial,Helvetica,sans-serif">-Avi</span></div></font></span><div><div><div><br><div class="gmail_quote">On Sun, Feb 10, 2013 at 10:16 PM, Steven Ayre <span dir="ltr"><<a href="mailto:steveayre@gmail.com" target="_blank">steveayre@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Fair enough.<div><br></div><div>I still don't see the need to reinvent the wheel just to avoid depending on a 3rd party well developed and extensively tested piece of software. Corosync provides an API that can be used for passing messages between nodes in a cluster.</div>
<div><div><div><br></div><div>-Steve</div><div><div><div><br></div><div><br><br><div class="gmail_quote">On 10 February 2013 18:14, Eliot Gable <span dir="ltr"><<a href="mailto:egable+freeswitch@gmail.com" target="_blank">egable+freeswitch@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <<a href="mailto:steveayre@gmail.com" target="_blank">steveayre@gmail.com</a>> wrote:<br>
> That covers redundancy in case of a network card or cable failure, but isn't<br>
> what partitioning is about. Multiple NICs cannot prevent partitioning.<br>
><br>
> As an example, partitioning might happen when a network switch between two<br>
> network segments fails so you have nodes A+B in segment 1 able to talk to<br>
> each other but unable to talk to nodes C+D in segment 2, while C+D can talk<br>
> to each other but not A+B.<br>
><br>
> Pacemaker/corosync contain a lot of algorithms to fence off partitions<br>
> without quorum and can resort to things like STONITH if required to force a<br>
> node to shutdown rather than risk it causing disruption to the cluster (for<br>
> example if it tries to take over traffic to a virtual IP you could end up in<br>
> a case where you have two servers sending ARP responses for the same IP).<br>
><br>
<br>
</div>Steve,<br>
<br>
As Avi pointed out, I mentioned having multiple physical networks as a<br>
guard against a network split / partition. If one network is split<br>
such that A and B can talk to each other over it and C and D can talk<br>
to each other over it, you would indeed have an issue if you only had<br>
one network. However, with two or more networks, all four nodes will<br>
still be able to talk to each other over the other network(s).<br>
<br>
Now, granted, if you have a network split in all networks, then you<br>
are still screwed. Pacemaker and other solutions deal with this, as<br>
you mentioned, using something called "quorum" where you need a<br>
majority of nodes to be able to see each other, and they fence the<br>
remaining nodes. As I documented on my wiki page for the module, I do<br>
have plans to eventually support such functionality. However, that is<br>
a bit further down the road as it will take some time to develop<br>
STONITH interfaces to various hardware or even to reuse the STONITH<br>
modules from Pacemaker or another project. In any case, I feel it is<br>
more important to get the base functionality developed and debugged as<br>
utilizing multiple networks is a good way to prevent network splits<br>
from being an issue.<br>
<br>
That being said, there are other issues to contend with when<br>
discussing network splits. For example, if A and B can see the<br>
Internet but C and D cannot, but C is a Master and B is a slave, you<br>
still have an issue to address. In this case, mod_ha_cluster must be<br>
able to determine that C and D cannot see the Internet. They need to<br>
perform very fast pings to some IP address, or have some external host<br>
sending them data in some way that they can detect when traffic<br>
to/from the Internet has stopped. I can place a media bug on the audio<br>
streams to make this determination fairly accurately. I can also rely<br>
on a ping mechanism to make the determination. Once the determination<br>
is made, mod_ha_cluster then has to promote B to a master to take over<br>
C.<br>
<br>
So, there are certainly still other issues to address when a network<br>
split occurs, but split-brain is easily avoided by simply adding<br>
redundant networks.<br>
<div><div><br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br></div></div></div></div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div></div></div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div>