High Availability Cluster Module for FreeSWITCH
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it. Some advantages over a 3rd party solution like Pacemaker include: - Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS) These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit: http://wiki.freeswitch.org/wiki/Mod_ha_cluster If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter: http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod... Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support. -- Eliot Gable
Hey Eliot Sounds exciting. Which DB will you be using Postgres? I will try to get my company to pitch in a few dollars. Sent from my mobile device. On Feb 9, 2013, at 12:11 PM, Eliot Gable <egable+freeswitch@gmail.com> wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod...
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The beauty of the module is that it will not require any 3rd party software to operate. I am designing it to run entirely in-memory using multi-cast events to communicate between cluster nodes. No external database will be required to enable the automatic fail over and call recovery. If you have your customers / clients using DNS to point traffic to your system and you run multiple master nodes with DNS performing load balancing, you can also make use of the maintenance mode I have planned which will use a separate IP address and update your DNS records to remove the master which you put into maintenance mode and put in the maintenance IP which is assigned to the slave node which is being promoted as master. This allows calls to drain off the master while new calls come up on the slave, and then you can gracefully shut down the master when all calls are finished. Once you are done with maintenance, you simply bring it back online and it comes up as a slave or a master as needed (according to your configured master / slave ratio). The IP which it was using as a master node becomes the new designated maintenance IP. This is the only case where a 3rd party software package is required, and it is an optional feature. On Sat, Feb 9, 2013 at 6:49 PM, Gilad Abada <ga@steadfasttelecom.com> wrote:
Hey Eliot
Sounds exciting. Which DB will you be using Postgres? I will try to get my company to pitch in a few dollars.
Sent from my mobile device.
On Feb 9, 2013, at 12:11 PM, Eliot Gable <egable+freeswitch@gmail.com> wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod...
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
I wonder how do you handle network partition? Simply "knowing" which nodes are up in a distributed system is hard enough and there is a reason why solutions like pacemaker/corosync have years of development behind them. W dniu niedziela, 10 lutego 2013 użytkownik Eliot Gable napisał:
The beauty of the module is that it will not require any 3rd party software to operate. I am designing it to run entirely in-memory using multi-cast events to communicate between cluster nodes. No external database will be required to enable the automatic fail over and call recovery.
If you have your customers / clients using DNS to point traffic to your system and you run multiple master nodes with DNS performing load balancing, you can also make use of the maintenance mode I have planned which will use a separate IP address and update your DNS records to remove the master which you put into maintenance mode and put in the maintenance IP which is assigned to the slave node which is being promoted as master. This allows calls to drain off the master while new calls come up on the slave, and then you can gracefully shut down the master when all calls are finished. Once you are done with maintenance, you simply bring it back online and it comes up as a slave or a master as needed (according to your configured master / slave ratio). The IP which it was using as a master node becomes the new designated maintenance IP. This is the only case where a 3rd party software package is required, and it is an optional feature.
On Sat, Feb 9, 2013 at 6:49 PM, Gilad Abada <ga@steadfasttelecom.com<javascript:;>> wrote:
Hey Eliot
Sounds exciting. Which DB will you be using Postgres? I will try to get my company to pitch in a few dollars.
Sent from my mobile device.
On Feb 9, 2013, at 12:11 PM, Eliot Gable <egable+freeswitch@gmail.com<javascript:;>> wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod...
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:;> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <javascript:;> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik
You use multiple NICs in the systems and send heartbeats out all of them. There is no other way to do it. Two NICs are required, and 3 are recommended. On Sun, Feb 10, 2013 at 6:54 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
I wonder how do you handle network partition? Simply "knowing" which nodes are up in a distributed system is hard enough and there is a reason why solutions like pacemaker/corosync have years of development behind them.
W dniu niedziela, 10 lutego 2013 użytkownik Eliot Gable napisał:
The beauty of the module is that it will not require any 3rd party software to operate. I am designing it to run entirely in-memory using multi-cast events to communicate between cluster nodes. No external database will be required to enable the automatic fail over and call recovery.
If you have your customers / clients using DNS to point traffic to your system and you run multiple master nodes with DNS performing load balancing, you can also make use of the maintenance mode I have planned which will use a separate IP address and update your DNS records to remove the master which you put into maintenance mode and put in the maintenance IP which is assigned to the slave node which is being promoted as master. This allows calls to drain off the master while new calls come up on the slave, and then you can gracefully shut down the master when all calls are finished. Once you are done with maintenance, you simply bring it back online and it comes up as a slave or a master as needed (according to your configured master / slave ratio). The IP which it was using as a master node becomes the new designated maintenance IP. This is the only case where a 3rd party software package is required, and it is an optional feature.
On Sat, Feb 9, 2013 at 6:49 PM, Gilad Abada <ga@steadfasttelecom.com> wrote:
Hey Eliot
Sounds exciting. Which DB will you be using Postgres? I will try to get my company to pitch in a few dollars.
Sent from my mobile device.
On Feb 9, 2013, at 12:11 PM, Eliot Gable <egable+freeswitch@gmail.com> wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod...
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable "We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower "I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower "Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
On Sun, Feb 10, 2013 at 10:27 AM, Eliot Gable <egable+freeswitch@gmail.com> wrote:
You use multiple NICs in the systems and send heartbeats out all of them. There is no other way to do it. Two NICs are required, and 3 are recommended.
Just to clarify a little, this is something that needs to be handled mostly by the person / organization deploying the HA module. The HA module supports multiple heartbeat NICs. The entity deploying the module needs to design the physical layer of their network to ensure a full network partition can never occur. You do that in general by deploying 2 or more physical networks. This means you need redundant power, battery backups, multiple physical switches, etc. You need to design and deploy your physical network to ensure that no matter what fails (power, wiring, switch ports, switches, NICs, etc), the systems using the module always have an alternative method of communicating with each other. Typically, two physical networks are sufficient for most users. However, carrier deployments who want to offer 99.999% or better uptime might want to go with three physical networks. This means placing three NICs in each system (one on each network) and configuring mod_ha_cluster to send and receive messages on all three NICs. The module keeps several seconds of cached messages in a hash table to eliminate / ignore duplicate messages. The first one received will be used and it's message ID is stored in the hash table. If another copy arrives before that entry is pruned from the cache, the additional copy is ignored. In addition, you can deploy multiple clusters distributed geographically to ensure that if one entire cluster goes offline, your services do not fail entirely. There is nothing magical or "hard" per-se about detecting and preventing a network split from screwing with the cluster. The hardest part is simply educating people on how to design and deploy the physical network and configure the module so it can do the detection. Also, please do not confuse the difficulty of writing a general purpose HA system like Pacemaker with the relative simplicity of writing one for a single specific application. When you write something general purpose like Pacemaker, your task is vastly more difficult due to the multitude of software configurations you need to support. For mod_ha_cluster, there is a very specific design in mind. Namely, you set a master-to-slave ratio and bring nodes online and it is designed to behave a specific way and bring those nodes online in a specific, well-defined way. There is really only "one way" that this module supports bringing up the cluster. This vastly simplifies the code and the entire project.
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning. As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B. Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP). -Steve On 10 February 2013 15:27, Eliot Gable <egable+freeswitch@gmail.com> wrote:
You use multiple NICs in the systems and send heartbeats out all of them. There is no other way to do it. Two NICs are required, and 3 are recommended.
I wonder how do you handle network partition? Simply "knowing" which nodes are up in a distributed system is hard enough and there is a reason why solutions like pacemaker/corosync have years of development behind them.
W dniu niedziela, 10 lutego 2013 użytkownik Eliot Gable napisał:
The beauty of the module is that it will not require any 3rd party software to operate. I am designing it to run entirely in-memory using multi-cast events to communicate between cluster nodes. No external database will be required to enable the automatic fail over and call recovery.
If you have your customers / clients using DNS to point traffic to your system and you run multiple master nodes with DNS performing load balancing, you can also make use of the maintenance mode I have planned which will use a separate IP address and update your DNS records to remove the master which you put into maintenance mode and put in the maintenance IP which is assigned to the slave node which is being promoted as master. This allows calls to drain off the master while new calls come up on the slave, and then you can gracefully shut down the master when all calls are finished. Once you are done with maintenance, you simply bring it back online and it comes up as a slave or a master as needed (according to your configured master / slave ratio). The IP which it was using as a master node becomes the new designated maintenance IP. This is the only case where a 3rd party software package is required, and it is an optional feature.
On Sat, Feb 9, 2013 at 6:49 PM, Gilad Abada <ga@steadfasttelecom.com> wrote:
Hey Eliot
Sounds exciting. Which DB will you be using Postgres? I will try to get my company to pitch in a few dollars.
Sent from my mobile device.
On Feb 9, 2013, at 12:11 PM, Eliot Gable <egable+freeswitch@gmail.com
wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker
On Sun, Feb 10, 2013 at 6:54 AM, Marcin Gozdalik <gozdal@gmail.com> wrote: +
Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod...
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable
"We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower
"I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower
"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Steven: Eliot specifically said 2 (or three!) entire networks: each with their own NIC, cables, and network switches. -Avi On Sun, Feb 10, 2013 at 7:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
-Steve
On 10 February 2013 15:27, Eliot Gable <egable+freeswitch@gmail.com>wrote:
You use multiple NICs in the systems and send heartbeats out all of them. There is no other way to do it. Two NICs are required, and 3 are recommended.
On Sun, Feb 10, 2013 at 6:54 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
I wonder how do you handle network partition? Simply "knowing" which nodes are up in a distributed system is hard enough and there is a reason why solutions like pacemaker/corosync have years of development behind them.
W dniu niedziela, 10 lutego 2013 użytkownik Eliot Gable napisał:
The beauty of the module is that it will not require any 3rd party software to operate. I am designing it to run entirely in-memory using multi-cast events to communicate between cluster nodes. No external database will be required to enable the automatic fail over and call recovery.
If you have your customers / clients using DNS to point traffic to your system and you run multiple master nodes with DNS performing load balancing, you can also make use of the maintenance mode I have planned which will use a separate IP address and update your DNS records to remove the master which you put into maintenance mode and put in the maintenance IP which is assigned to the slave node which is being promoted as master. This allows calls to drain off the master while new calls come up on the slave, and then you can gracefully shut down the master when all calls are finished. Once you are done with maintenance, you simply bring it back online and it comes up as a slave or a master as needed (according to your configured master / slave ratio). The IP which it was using as a master node becomes the new designated maintenance IP. This is the only case where a 3rd party software package is required, and it is an optional feature.
On Sat, Feb 9, 2013 at 6:49 PM, Gilad Abada <ga@steadfasttelecom.com> wrote:
Hey Eliot
Sounds exciting. Which DB will you be using Postgres? I will try to get my company to pitch in a few dollars.
Sent from my mobile device.
On Feb 9, 2013, at 12:11 PM, Eliot Gable <
egable+freeswitch@gmail.com>
wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod...
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable
"We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower
"I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower
"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
Steve, As Avi pointed out, I mentioned having multiple physical networks as a guard against a network split / partition. If one network is split such that A and B can talk to each other over it and C and D can talk to each other over it, you would indeed have an issue if you only had one network. However, with two or more networks, all four nodes will still be able to talk to each other over the other network(s). Now, granted, if you have a network split in all networks, then you are still screwed. Pacemaker and other solutions deal with this, as you mentioned, using something called "quorum" where you need a majority of nodes to be able to see each other, and they fence the remaining nodes. As I documented on my wiki page for the module, I do have plans to eventually support such functionality. However, that is a bit further down the road as it will take some time to develop STONITH interfaces to various hardware or even to reuse the STONITH modules from Pacemaker or another project. In any case, I feel it is more important to get the base functionality developed and debugged as utilizing multiple networks is a good way to prevent network splits from being an issue. That being said, there are other issues to contend with when discussing network splits. For example, if A and B can see the Internet but C and D cannot, but C is a Master and B is a slave, you still have an issue to address. In this case, mod_ha_cluster must be able to determine that C and D cannot see the Internet. They need to perform very fast pings to some IP address, or have some external host sending them data in some way that they can detect when traffic to/from the Internet has stopped. I can place a media bug on the audio streams to make this determination fairly accurately. I can also rely on a ping mechanism to make the determination. Once the determination is made, mod_ha_cluster then has to promote B to a master to take over C. So, there are certainly still other issues to address when a network split occurs, but split-brain is easily avoided by simply adding redundant networks.
Don't get me wrong, I'd love to fund good HA module for FS, if not for any other reason that I could benefit from that. But having done a few installations of systems that were supposed to be "HA" and seen them fail when real problems came I know it ain't easy. Redundant networks are fine but following scenarios usually lead to both machines reply to ARPs for virtual IP and whole HA falls apart: 1) FS stops responding (e.g. due to heavy swapping or disk full), yet kernel manages to reply to ARPs 2) the HA module fails (like in crashes) but FS manages to work 3) some firewall rule is activated that stops multicast traffic (all or some) STONITH based on separate technology (like USB-USB connection connected to some KVM-over-IP with control over power) is indispensable in such scenarios. 2013/2/10 Eliot Gable <egable+freeswitch@gmail.com>:
On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
Steve,
As Avi pointed out, I mentioned having multiple physical networks as a guard against a network split / partition. If one network is split such that A and B can talk to each other over it and C and D can talk to each other over it, you would indeed have an issue if you only had one network. However, with two or more networks, all four nodes will still be able to talk to each other over the other network(s).
Now, granted, if you have a network split in all networks, then you are still screwed. Pacemaker and other solutions deal with this, as you mentioned, using something called "quorum" where you need a majority of nodes to be able to see each other, and they fence the remaining nodes. As I documented on my wiki page for the module, I do have plans to eventually support such functionality. However, that is a bit further down the road as it will take some time to develop STONITH interfaces to various hardware or even to reuse the STONITH modules from Pacemaker or another project. In any case, I feel it is more important to get the base functionality developed and debugged as utilizing multiple networks is a good way to prevent network splits from being an issue.
That being said, there are other issues to contend with when discussing network splits. For example, if A and B can see the Internet but C and D cannot, but C is a Master and B is a slave, you still have an issue to address. In this case, mod_ha_cluster must be able to determine that C and D cannot see the Internet. They need to perform very fast pings to some IP address, or have some external host sending them data in some way that they can detect when traffic to/from the Internet has stopped. I can place a media bug on the audio streams to make this determination fairly accurately. I can also rely on a ping mechanism to make the determination. Once the determination is made, mod_ha_cluster then has to promote B to a master to take over C.
So, there are certainly still other issues to address when a network split occurs, but split-brain is easily avoided by simply adding redundant networks.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik
On Sun, Feb 10, 2013 at 3:11 PM, Marcin Gozdalik <gozdal@gmail.com> wrote:
Don't get me wrong, I'd love to fund good HA module for FS, if not for any other reason that I could benefit from that. But having done a few installations of systems that were supposed to be "HA" and seen them fail when real problems came I know it ain't easy. Redundant networks are fine but following scenarios usually lead to both machines reply to ARPs for virtual IP and whole HA falls apart:
1) FS stops responding (e.g. due to heavy swapping or disk full), yet kernel manages to reply to ARPs 2) the HA module fails (like in crashes) but FS manages to work 3) some firewall rule is activated that stops multicast traffic (all or some)
STONITH based on separate technology (like USB-USB connection connected to some KVM-over-IP with control over power) is indispensable in such scenarios.
I have also seen each of these cases when dealing with HA setups for FreeSWITCH. That is part of why I want to write one specifically for FreeSWITCH. General purpose HA systems cannot catch and properly deal with the sorts of things I see when using one to run FreeSWITCH in a high availability configuration. For #1, it is pretty easy to detect when FS stops responding for whatever reason. A watchdog thread inside FS can shut it down in a lot of those cases, as well as remove the IP from the system. Aside from that, future fencing will be useful to bounce the entire machine if the watchdog thread does not or cannot handle the situation. This is another advantage of doing the module inside FS. For #2, a lot of testing is required to make sure that if the HA system fails, it takes down FS with it, or to ensure that it cannot simply "not work" while FS continues to work. It's very hard for a module inside FS to fail in a way that FS keeps working yet the module doesn't. When using Pacemaker and Corosync, it is easy for the HA system to fail in a way that leaves FS running on a node yet the HA system thinks it is not there. This is one of the advantages of doing a module inside FS. For #3, I would like to eventually have the module scanning the firewall rules for changes and enforcing a specific, pre-determined "known-good" set of firewall rules. That is a ways off, but it is planned. Besides that, this is yet one more reason why having the module in FS is the best option. When running in FS, it is easy to determine if the traffic is being blocked by a firewall rule (you simply do not receive the traffic). Again, a module can more effectively catch and respond to this type of situation compared to a general purpose solution like Pacemaker and Corosync. I do not disagree about STONITH being indispensable in certain cases. However, most companies I have worked with who try to do HA right now using Pacemaker and Corosync completely ignore STONITH entirely. STONITH is another case where the biggest problem is simply getting people to follow best practices. And, interestingly enough, most companies actually get by just fine without deploying a STONITH solution. They simply accept that there is a risk of hitting a case where they need it and will not have it, and they just plan to have someone manually reset something if that case arises. This is why STONITH is not one of my first priorities in development of the module. It is on the list because it is important (and I feel it should be used in any carrier-grade deployment), but most companies are perfectly OK to do without it. This is especially true for companies who want HA on their home-grown PBX systems for their 10-person organization. They couldn't care less whether the reset happens manually or automatically in 1 out of 1,000 failures. As long as those other 999 failures are automatic, they will be happy.
For case #1: How watchdog thread will remove ip address if freeswitch coredumped? I think this should be done by other process. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/High-Availability-Cluster-Modu... Sent from the freeswitch-users mailing list archive at Nabble.com.
On Sun, Feb 10, 2013 at 04:00:21PM -0500, Eliot Gable wrote:
For #3, I would like to eventually have the module scanning the firewall rules for changes and enforcing a specific, pre-determined "known-good" set of firewall rules.
In this scenario, don't forget to consider users who run freeswitch under a dedicated user account, not as root - some method for the "freeswitch" user to be able to run iptables/other will need to be implemented (whether that is sudo, or a seperate (root) process or otherwise). Regards,
Fair enough. I still don't see the need to reinvent the wheel just to avoid depending on a 3rd party well developed and extensively tested piece of software. Corosync provides an API that can be used for passing messages between nodes in a cluster. -Steve On 10 February 2013 18:14, Eliot Gable <egable+freeswitch@gmail.com> wrote:
On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
Steve,
As Avi pointed out, I mentioned having multiple physical networks as a guard against a network split / partition. If one network is split such that A and B can talk to each other over it and C and D can talk to each other over it, you would indeed have an issue if you only had one network. However, with two or more networks, all four nodes will still be able to talk to each other over the other network(s).
Now, granted, if you have a network split in all networks, then you are still screwed. Pacemaker and other solutions deal with this, as you mentioned, using something called "quorum" where you need a majority of nodes to be able to see each other, and they fence the remaining nodes. As I documented on my wiki page for the module, I do have plans to eventually support such functionality. However, that is a bit further down the road as it will take some time to develop STONITH interfaces to various hardware or even to reuse the STONITH modules from Pacemaker or another project. In any case, I feel it is more important to get the base functionality developed and debugged as utilizing multiple networks is a good way to prevent network splits from being an issue.
That being said, there are other issues to contend with when discussing network splits. For example, if A and B can see the Internet but C and D cannot, but C is a Master and B is a slave, you still have an issue to address. In this case, mod_ha_cluster must be able to determine that C and D cannot see the Internet. They need to perform very fast pings to some IP address, or have some external host sending them data in some way that they can detect when traffic to/from the Internet has stopped. I can place a media bug on the audio streams to make this determination fairly accurately. I can also rely on a ping mechanism to make the determination. Once the determination is made, mod_ha_cluster then has to promote B to a master to take over C.
So, there are certainly still other issues to address when a network split occurs, but split-brain is easily avoided by simply adding redundant networks.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Steve: Several reasons. One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack. Secondly, in order to recover live calls, you need a list of the calls. That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster. Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything. Fourth, according to the docs: single configuration for all FS instances, rather than manually ensuring each one has the same config. Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs to be rewritten for that, perhaps... There's certainly *something *special possible with mod_ha_cluster that can't be done with existing solutions cleanly, if at all... -Avi On Sun, Feb 10, 2013 at 10:16 PM, Steven Ayre <steveayre@gmail.com> wrote:
Fair enough.
I still don't see the need to reinvent the wheel just to avoid depending on a 3rd party well developed and extensively tested piece of software. Corosync provides an API that can be used for passing messages between nodes in a cluster.
-Steve
On 10 February 2013 18:14, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
Steve,
As Avi pointed out, I mentioned having multiple physical networks as a guard against a network split / partition. If one network is split such that A and B can talk to each other over it and C and D can talk to each other over it, you would indeed have an issue if you only had one network. However, with two or more networks, all four nodes will still be able to talk to each other over the other network(s).
Now, granted, if you have a network split in all networks, then you are still screwed. Pacemaker and other solutions deal with this, as you mentioned, using something called "quorum" where you need a majority of nodes to be able to see each other, and they fence the remaining nodes. As I documented on my wiki page for the module, I do have plans to eventually support such functionality. However, that is a bit further down the road as it will take some time to develop STONITH interfaces to various hardware or even to reuse the STONITH modules from Pacemaker or another project. In any case, I feel it is more important to get the base functionality developed and debugged as utilizing multiple networks is a good way to prevent network splits from being an issue.
That being said, there are other issues to contend with when discussing network splits. For example, if A and B can see the Internet but C and D cannot, but C is a Master and B is a slave, you still have an issue to address. In this case, mod_ha_cluster must be able to determine that C and D cannot see the Internet. They need to perform very fast pings to some IP address, or have some external host sending them data in some way that they can detect when traffic to/from the Internet has stopped. I can place a media bug on the audio streams to make this determination fairly accurately. I can also rely on a ping mechanism to make the determination. Once the determination is made, mod_ha_cluster then has to promote B to a master to take over C.
So, there are certainly still other issues to address when a network split occurs, but split-brain is easily avoided by simply adding redundant networks.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Sun, Feb 10, 2013 at 3:40 PM, Avi Marcus <avi@avimarcus.net> wrote:
Steve: Several reasons. One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack.
Secondly, in order to recover live calls, you need a list of the calls. That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster.
Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.
Fourth, according to the docs: single configuration for all FS instances, rather than manually ensuring each one has the same config.
Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs to be rewritten for that, perhaps...
There's certainly *something *special possible with mod_ha_cluster that can't be done with existing solutions cleanly, if at all...
-Avi
Thanks, Avi, for the voice of support, and for actually reading my documentation. :) First, however, I need to correct you on one point. I have actually designed mod_ha_cluster as a N + x cluster, meaning N master nodes and x slave nodes. What this means is that you can get better availability than having 1 slave node, while also getting more cost effective scalability than having 1 slave node per master (which is all that is really possible with most other solutions). So, for example, you could run 16 master nodes and 3 slave nodes in the clusters. In addition, you do not need to deal with the cost of a database system to synchronize the calls and registration information between all the nodes. Keep in mind, also, that you would also then have to have that database system set up in a high availability configuration, and it would have to be a pretty big box to handle the load from so many servers. So, bottom line is that mod_ha_cluster can dramatically reduce the cost of deploying a high availability solution for FreeSWITCH, as well as ease the configuration. The reduced deployment complexity not only makes it more approachable for a much larger audience, it can also actually improve availability simply due to fewer components involved and fewer places failures or bugs to cause issues. Also, as you pointed out, this is more than simply a high availability module. It is also a clustering module. This means it shares registration information, call limiting information, call state, and all sorts of other essential information. Using a shared disk resource, you can have voicemail and menus shared between multiple nodes and your entire set of potentially dozens of nodes can act as one single cluster on a single DNS address. Because of the N + x design, you can use smaller nodes in your cluster and scale by adding more nodes vs adding more horsepower to individual nodes. And, as Avi pointed out, this means that if a single node fails, you have other nodes to pick up the slack. Using Pacemaker and Corosync with an active/passive configuration, 16 master nodes requires an additional 16 slave nodes of the same size. That is a lot of unused and expensive gear. Using mod_ha_cluster, you could take those 32 nodes and run 4 as slaves while the other 28 are active. That's a much more efficient and effective design.
One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack.
Corosync clusters are not limited to active/passive pairs. It's just a very common setup. For example you could have resource agents 1) to keep FS running on all nodes 2) for virtual IPs 3) for IP:port Sofia profiles. You can then define dependancies between them. That should let you keep FS running at all times and move an IP and the associated Sofia profiles to a new node that's already running FS when the original node fails. For maintance you can simply trigger that from the CRM. Secondly, in order to recover live calls, you need a list of the calls.
That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster. Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.
Which can also be done using Corosync's IPC messaging API. (Personally I prefer using MySQL Cluster via ODBC - which is HA, synchronous and offloads all load off of the FS nodes, but that's offtopic). Fourth, according to the docs: single configuration for all FS instances,
rather than manually ensuring each one has the same config.
This could be achieved with the Corosync API. Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs
to be rewritten for that, perhaps...
That's not going to happen without a storage API added to the FS core - you're always going to need some 3rd party such NFS, CIFS, DRBD etc. mod_voicemail is hardcoded to use the ODBC interface but that's only for the index not the recordings.nc Corosync would allow you to make FS depend on the NFS/whatever service running so if the storage backend has failed FS would move to a node where it is available - not necessarily possibly from within mod_ha_cluster itself. There's certainly *something *special possible with mod_ha_cluster that
can't be done with existing solutions cleanly, if at all...
Don't misunderstand me, I think it's a great idea to have a module aimed at HA and sharing state across a cluster while being able to detect new failure conditions. I just think that in the specific area of node monitoring and messaging across the cluster it would be better to use a well tested and proven solution such as Corosync which is based on a large number of papers, algorithms, and generally decades of work. Every time I've seen/heard of an attempt to redo that from scratch it's been unreliable especially in unexpected failure conditions. Simply because it's a dependency on another program isn't a good enough reason not to use it - you're already depending on various other programs (Linux, sysvinit, monit, cron, syslog etc) anyway. By not using it you're just adding extra work, adding unnecessarily complexity and increasing the risk of bugs. Corosync would also have advantages because the tools to migrate services to another node for maintenance, detect and restart resources on failure etc already exist. -Steve On 10 February 2013 20:40, Avi Marcus <avi@avimarcus.net> wrote:
Steve: Several reasons. One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack.
Secondly, in order to recover live calls, you need a list of the calls. That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster.
Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.
Fourth, according to the docs: single configuration for all FS instances, rather than manually ensuring each one has the same config.
Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs to be rewritten for that, perhaps...
There's certainly *something *special possible with mod_ha_cluster that can't be done with existing solutions cleanly, if at all...
-Avi
On Sun, Feb 10, 2013 at 10:16 PM, Steven Ayre <steveayre@gmail.com> wrote:
Fair enough.
I still don't see the need to reinvent the wheel just to avoid depending on a 3rd party well developed and extensively tested piece of software. Corosync provides an API that can be used for passing messages between nodes in a cluster.
-Steve
On 10 February 2013 18:14, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
Steve,
As Avi pointed out, I mentioned having multiple physical networks as a guard against a network split / partition. If one network is split such that A and B can talk to each other over it and C and D can talk to each other over it, you would indeed have an issue if you only had one network. However, with two or more networks, all four nodes will still be able to talk to each other over the other network(s).
Now, granted, if you have a network split in all networks, then you are still screwed. Pacemaker and other solutions deal with this, as you mentioned, using something called "quorum" where you need a majority of nodes to be able to see each other, and they fence the remaining nodes. As I documented on my wiki page for the module, I do have plans to eventually support such functionality. However, that is a bit further down the road as it will take some time to develop STONITH interfaces to various hardware or even to reuse the STONITH modules from Pacemaker or another project. In any case, I feel it is more important to get the base functionality developed and debugged as utilizing multiple networks is a good way to prevent network splits from being an issue.
That being said, there are other issues to contend with when discussing network splits. For example, if A and B can see the Internet but C and D cannot, but C is a Master and B is a slave, you still have an issue to address. In this case, mod_ha_cluster must be able to determine that C and D cannot see the Internet. They need to perform very fast pings to some IP address, or have some external host sending them data in some way that they can detect when traffic to/from the Internet has stopped. I can place a media bug on the audio streams to make this determination fairly accurately. I can also rely on a ping mechanism to make the determination. Once the determination is made, mod_ha_cluster then has to promote B to a master to take over C.
So, there are certainly still other issues to address when a network split occurs, but split-brain is easily avoided by simply adding redundant networks.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Sun, Feb 10, 2013 at 4:32 PM, Steven Ayre <steveayre@gmail.com> wrote:
One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair.
That way if any single node fails, there's something to pick up the slack.
Corosync clusters are not limited to active/passive pairs. It's just a very common setup.
For example you could have resource agents 1) to keep FS running on all nodes 2) for virtual IPs 3) for IP:port Sofia profiles. You can then define dependancies between them. That should let you keep FS running at all times and move an IP and the associated Sofia profiles to a new node that's already running FS when the original node fails. For maintance you can simply trigger that from the CRM.
This is true, however it would require a very complicated resource agent to manage FreeSWITCH in a similar configuration to what mod_ha_cluster is designed to do. In addition, the functionality simply does not exist in FreeSWITCH right now to tell it to take over for a specific failed master and recover those specific calls. So, right now, using Pacemaker and Corosync, there is absolutely no way to run a N + x FreeSWITCH cluster. Also, the response time on a Pacemaker + Corosync cluster for failure detection and recovery is measured in seconds, which is not ideal for a real-time communications platform. Obviously, there is nothing at all preventing you from running Pacemaker and Corosync in addition to mod_ha_cluster. In fact, I was even considering providing some CLI arguments to allow FreeSWITCH (with mod_ha_cluster enabled) to be commanded from Pacemaker and act as its own resource agent. If you think that would be an interesting feature, I can look into what it would take to work that out. I previously wrote a resource agent for Broadvox when I worked for them, and I wrote another one after leaving them. Both were intended to manage FreeSWITCH as a master/slave pair, so I have some idea on how to do it, just not necessarily the specifics on making it do N + x the way I intend to have mod_ha_cluster operate.
Secondly, in order to recover live calls, you need a list of the calls.
That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster. Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.
Which can also be done using Corosync's IPC messaging API.
(Personally I prefer using MySQL Cluster via ODBC - which is HA, synchronous and offloads all load off of the FS nodes, but that's offtopic).
I have used MySQL cluster and Postgres with my own replication daemon. Neither are ideal solutions for a multitude of reasons. Also, using something like Corosync's IPC messaging is not ideal either. Besides that, the FreeSWITCH core has a sufficiently robust API for doing network messaging, and also its own event system which happens to be perfectly suited for exactly the kind of messaging I need to accomplish. Using the Corosync IPC messaging API would be like trying to shove a large round peg through a small square hole.
Fourth, according to the docs: single configuration for all FS instances,
rather than manually ensuring each one has the same config.
This could be achieved with the Corosync API.
Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's
APIs to be rewritten for that, perhaps...
That's not going to happen without a storage API added to the FS core - you're always going to need some 3rd party such NFS, CIFS, DRBD etc. mod_voicemail is hardcoded to use the ODBC interface but that's only for the index not the recordings.nc
Corosync would allow you to make FS depend on the NFS/whatever service running so if the storage backend has failed FS would move to a node where it is available - not necessarily possibly from within mod_ha_cluster itself.
Well, I *could* broadcast an entire voicemail to all nodes, but that seems like a bit of overkill, and I don't see any real reason to code a storage system into FreeSWITCH. That really would be reinventing the wheel. I don't think the majority of users would find it all that difficult to set up a shared NFS space somewhere, or even use something like MooseFS. In fact, MooseFS is so easy to use, I went from reading about it for the first time to having a fully functional 6-node shared storage cluster in about 45 minutes. This is one place where a 3rd party solution is really the optimal approach.
There's certainly *something *special possible with mod_ha_cluster that
can't be done with existing solutions cleanly, if at all...
Don't misunderstand me, I think it's a great idea to have a module aimed at HA and sharing state across a cluster while being able to detect new failure conditions.
I just think that in the specific area of node monitoring and messaging across the cluster it would be better to use a well tested and proven solution such as Corosync which is based on a large number of papers, algorithms, and generally decades of work. Every time I've seen/heard of an attempt to redo that from scratch it's been unreliable especially in unexpected failure conditions. Simply because it's a dependency on another program isn't a good enough reason not to use it - you're already depending on various other programs (Linux, sysvinit, monit, cron, syslog etc) anyway. By not using it you're just adding extra work, adding unnecessarily complexity and increasing the risk of bugs. Corosync would also have advantages because the tools to migrate services to another node for maintenance, detect and restart resources on failure etc already exist.
I think you greatly overestimate the complexity of the messaging and node monitoring required to make FS run as a multi-master, multi-slave cluster, detect a node failure, and have it automatically recover calls on a slave system. You seem to be thinking of the task as if it were a general purpose HA solution which needs to work for any random piece of software across every imaginable network configuration and software deployment scenario. The messaging I need to do is simple: 1) I need to send events from the local FS node to all other slave nodes in the cluster so they can synchronize call state, registration information, call limiting information, etc. 2) I need to send heartbeats from all nodes to all nodes out all configured NICs so they can keep track of who is in which state and which is the designated slave in case of failure. These two tasks accomplish everything needed to make this system work. The heartbeats let each node calculate the state of the cluster. There is no "single brain" in the cluster. There is a very specific and well-defined set of rules by which the state of the cluster is calculated by each node, and all nodes will always arrive at the same conclusion so long as you have multiple physical networks to ensure communication is never broken between sets of nodes. The decision making happens on the slave nodes only. The masters are already in their role and will stay that way unless something catastrophic occurs. They have ways to detect internal failures, and I will have a way set up for them to detect and deal with a segfault of FS, as well. If such a failure occurs, their single purpose task is to shut down as cleanly as possible and get themselves back into a stable state. The slaves also have only one goal: to recover the calls of the failed master and become the new master. One slave is always designated as the primary candidate, and all slaves know which it is at all times (once they exchange their first heartbeats). They also know which is the secondary, tertiary, etc. It is all pre-determined up-front and only changes if a failure, reconfiguration, or maintenance event occurs. When the primary sees a master go down, it will perform a sanity check on itself first (to make sure it didn't have an issue) and then take over for the first master it saw go down. Once it has started that process, it broadcasts that it is switching to master and which master it is taking over for. At this point, the secondary immediately becomes primary to all remaining slaves and it is immediately available to take over for any other master nodes which might also have failed simultaneously. The whole process is a very specific and very orchestrated process. The whole thing is barely comparable to the complexity of the failure detection and fail over process which Pacemaker has to deal with. I am not trying to recreate something like Pacemaker. It is a hugely complex system which has to deal with all sorts of random and generalized configurations of endless types of software. Pacemaker is like an interstellar spacecraft while I just need a rocket to put a satellite in orbit.
+1 I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase. 2013/2/10 Steven Ayre <steveayre@gmail.com>:
One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack.
Corosync clusters are not limited to active/passive pairs. It's just a very common setup.
For example you could have resource agents 1) to keep FS running on all nodes 2) for virtual IPs 3) for IP:port Sofia profiles. You can then define dependancies between them. That should let you keep FS running at all times and move an IP and the associated Sofia profiles to a new node that's already running FS when the original node fails. For maintance you can simply trigger that from the CRM.
Secondly, in order to recover live calls, you need a list of the calls. That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster. Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.
Which can also be done using Corosync's IPC messaging API.
(Personally I prefer using MySQL Cluster via ODBC - which is HA, synchronous and offloads all load off of the FS nodes, but that's offtopic).
Fourth, according to the docs: single configuration for all FS instances, rather than manually ensuring each one has the same config.
This could be achieved with the Corosync API.
Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs to be rewritten for that, perhaps...
That's not going to happen without a storage API added to the FS core - you're always going to need some 3rd party such NFS, CIFS, DRBD etc. mod_voicemail is hardcoded to use the ODBC interface but that's only for the index not the recordings.nc
Corosync would allow you to make FS depend on the NFS/whatever service running so if the storage backend has failed FS would move to a node where it is available - not necessarily possibly from within mod_ha_cluster itself.
There's certainly something special possible with mod_ha_cluster that can't be done with existing solutions cleanly, if at all...
Don't misunderstand me, I think it's a great idea to have a module aimed at HA and sharing state across a cluster while being able to detect new failure conditions.
I just think that in the specific area of node monitoring and messaging across the cluster it would be better to use a well tested and proven solution such as Corosync which is based on a large number of papers, algorithms, and generally decades of work. Every time I've seen/heard of an attempt to redo that from scratch it's been unreliable especially in unexpected failure conditions. Simply because it's a dependency on another program isn't a good enough reason not to use it - you're already depending on various other programs (Linux, sysvinit, monit, cron, syslog etc) anyway. By not using it you're just adding extra work, adding unnecessarily complexity and increasing the risk of bugs. Corosync would also have advantages because the tools to migrate services to another node for maintenance, detect and restart resources on failure etc already exist.
-Steve
On 10 February 2013 20:40, Avi Marcus <avi@avimarcus.net> wrote:
Steve: Several reasons. One is the mod_ha_cluster is an N+1 Cluster, not a active/passive pair. That way if any single node fails, there's something to pick up the slack.
Secondly, in order to recover live calls, you need a list of the calls. That currently requires some sort of odbc (or postgres) with replication. Again, that's abstracted as part of mod_ha_cluster.
Third: The docs mention a similar of pooling for registration, that you can register to one server and you're regged on them all without needing a DB to sync everything.
Fourth, according to the docs: single configuration for all FS instances, rather than manually ensuring each one has the same config.
Fifth: Voicemail clustering? Or we'll have to wait for mod_voicemail's APIs to be rewritten for that, perhaps...
There's certainly something special possible with mod_ha_cluster that can't be done with existing solutions cleanly, if at all...
-Avi
On Sun, Feb 10, 2013 at 10:16 PM, Steven Ayre <steveayre@gmail.com> wrote:
Fair enough.
I still don't see the need to reinvent the wheel just to avoid depending on a 3rd party well developed and extensively tested piece of software. Corosync provides an API that can be used for passing messages between nodes in a cluster.
-Steve
On 10 February 2013 18:14, Eliot Gable <egable+freeswitch@gmail.com> wrote:
On Sun, Feb 10, 2013 at 12:08 PM, Steven Ayre <steveayre@gmail.com> wrote:
That covers redundancy in case of a network card or cable failure, but isn't what partitioning is about. Multiple NICs cannot prevent partitioning.
As an example, partitioning might happen when a network switch between two network segments fails so you have nodes A+B in segment 1 able to talk to each other but unable to talk to nodes C+D in segment 2, while C+D can talk to each other but not A+B.
Pacemaker/corosync contain a lot of algorithms to fence off partitions without quorum and can resort to things like STONITH if required to force a node to shutdown rather than risk it causing disruption to the cluster (for example if it tries to take over traffic to a virtual IP you could end up in a case where you have two servers sending ARP responses for the same IP).
Steve,
As Avi pointed out, I mentioned having multiple physical networks as a guard against a network split / partition. If one network is split such that A and B can talk to each other over it and C and D can talk to each other over it, you would indeed have an issue if you only had one network. However, with two or more networks, all four nodes will still be able to talk to each other over the other network(s).
Now, granted, if you have a network split in all networks, then you are still screwed. Pacemaker and other solutions deal with this, as you mentioned, using something called "quorum" where you need a majority of nodes to be able to see each other, and they fence the remaining nodes. As I documented on my wiki page for the module, I do have plans to eventually support such functionality. However, that is a bit further down the road as it will take some time to develop STONITH interfaces to various hardware or even to reuse the STONITH modules from Pacemaker or another project. In any case, I feel it is more important to get the base functionality developed and debugged as utilizing multiple networks is a good way to prevent network splits from being an issue.
That being said, there are other issues to contend with when discussing network splits. For example, if A and B can see the Internet but C and D cannot, but C is a Master and B is a slave, you still have an issue to address. In this case, mod_ha_cluster must be able to determine that C and D cannot see the Internet. They need to perform very fast pings to some IP address, or have some external host sending them data in some way that they can detect when traffic to/from the Internet has stopped. I can place a media bug on the audio streams to make this determination fairly accurately. I can also rely on a ping mechanism to make the determination. Once the determination is made, mod_ha_cluster then has to promote B to a master to take over C.
So, there are certainly still other issues to address when a network split occurs, but split-brain is easily avoided by simply adding redundant networks.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested. I have already written the code to the point that it parses the configuration files and starts sending heartbeats out all of the interfaces configured. I have also already written a lot of the code that deals with the state transitions. All I am talking about doing is implementing a tiny little finite state machine. It's a pretty trivial programming task. In fact, I think it was covered in my first year at Carnegie Mellon University. Of course, I had already figured out how to write such things in high school, I just did not know what it was called at that point. My point is, that this is not yet-another-cluster-communication-engine. It is a very specific and small finite state machine designed solely with the goal in mind of making FS have just enough information to coordinate call fail over internally. If I recall correctly, a lot of people also said writing yet-another-VoIP-server was a waste of time, but now we have FreeSWITCH, and it was obviously worth the effort. And I am not even trying to do something as complex as that. If you think this is yet-another-cluster-communication-engine, you are missing the point. It is not. It never will be. Look at Sonus, Genband, Broadsoft, Veraz, etc. All the big-name carrier-grade telecom providers have a built-in solution for automatic call fail over. The only way FreeSWITCH will ever compete with such solutions is if it also has that feature. Pacemaker and Corosync are overkill just to get FS to handle single node failures and provide call recovery. It took me a full 3 months of working with them every day to really understand how to deploy them properly in conjunction with FreeSWITCH and Postgres to provide a carrier-grade hot-standby solution which was robust enough to handle 99% of the failures I could throw at it. Granted, this was back when the configuration still needed to be written by hand in XML and prior the existence of any resource agent for FreeSWITCH. But, even with those changes, deploying Pacemaker and Corosync is not a simple task. If that is the requirement for FS to have HA, it will never truly stand a chance against commercial offerings.
Is the latest snapshot ofyour module at git://git.freeswitch.org/freeswitch-contrib.git ? 2013/2/11 Eliot Gable <egable+freeswitch@gmail.com>:
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested. I have already written the code to the point that it parses the configuration files and starts sending heartbeats out all of the interfaces configured. I have also already written a lot of the code that deals with the state transitions. All I am talking about doing is implementing a tiny little finite state machine. It's a pretty trivial programming task. In fact, I think it was covered in my first year at Carnegie Mellon University. Of course, I had already figured out how to write such things in high school, I just did not know what it was called at that point. My point is, that this is not yet-another-cluster-communication-engine. It is a very specific and small finite state machine designed solely with the goal in mind of making FS have just enough information to coordinate call fail over internally. If I recall correctly, a lot of people also said writing yet-another-VoIP-server was a waste of time, but now we have FreeSWITCH, and it was obviously worth the effort. And I am not even trying to do something as complex as that. If you think this is yet-another-cluster-communication-engine, you are missing the point. It is not. It never will be.
Look at Sonus, Genband, Broadsoft, Veraz, etc. All the big-name carrier-grade telecom providers have a built-in solution for automatic call fail over. The only way FreeSWITCH will ever compete with such solutions is if it also has that feature. Pacemaker and Corosync are overkill just to get FS to handle single node failures and provide call recovery. It took me a full 3 months of working with them every day to really understand how to deploy them properly in conjunction with FreeSWITCH and Postgres to provide a carrier-grade hot-standby solution which was robust enough to handle 99% of the failures I could throw at it. Granted, this was back when the configuration still needed to be written by hand in XML and prior the existence of any resource agent for FreeSWITCH. But, even with those changes, deploying Pacemaker and Corosync is not a simple task. If that is the requirement for FS to have HA, it will never truly stand a chance against commercial offerings.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Mon, Feb 11, 2013 at 1:06 PM, Luis Daniel Lucio Quiroz < luis.daniel.lucio@gmail.com> wrote:
Is the latest snapshot ofyour module at git://git.freeswitch.org/freeswitch-contrib.git ?
For those who are interested, yes, the latest snapshot is available there. What is there right now is rather old code. I have newer stuff amounting to about 2k lines of changes which has not yet been checked in. I am in the middle of a rather large rearrangement of some things. When I finish with that rearrangement and have it compiling again, I will check in the latest code.
I would choose to go with Elliot's idea of having the HA being build within the FS itself under the condition that it's a easy adaptation comparing to having to use and struggling with Pacemaker and Corosync. They can literally take months if someone is starting from scratch. And it would be nice to have this kind of capability to go against those brand name solutions. On Mon, Feb 11, 2013 at 6:06 AM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested. I have already written the code to the point that it parses the configuration files and starts sending heartbeats out all of the interfaces configured. I have also already written a lot of the code that deals with the state transitions. All I am talking about doing is implementing a tiny little finite state machine. It's a pretty trivial programming task. In fact, I think it was covered in my first year at Carnegie Mellon University. Of course, I had already figured out how to write such things in high school, I just did not know what it was called at that point. My point is, that this is not yet-another-cluster-communication-engine. It is a very specific and small finite state machine designed solely with the goal in mind of making FS have just enough information to coordinate call fail over internally. If I recall correctly, a lot of people also said writing yet-another-VoIP-server was a waste of time, but now we have FreeSWITCH, and it was obviously worth the effort. And I am not even trying to do something as complex as that. If you think this is yet-another-cluster-communication-engine, you are missing the point. It is not. It never will be.
Look at Sonus, Genband, Broadsoft, Veraz, etc. All the big-name carrier-grade telecom providers have a built-in solution for automatic call fail over. The only way FreeSWITCH will ever compete with such solutions is if it also has that feature. Pacemaker and Corosync are overkill just to get FS to handle single node failures and provide call recovery. It took me a full 3 months of working with them every day to really understand how to deploy them properly in conjunction with FreeSWITCH and Postgres to provide a carrier-grade hot-standby solution which was robust enough to handle 99% of the failures I could throw at it. Granted, this was back when the configuration still needed to be written by hand in XML and prior the existence of any resource agent for FreeSWITCH. But, even with those changes, deploying Pacemaker and Corosync is not a simple task. If that is the requirement for FS to have HA, it will never truly stand a chance against commercial offerings.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I believe some more planning and thought needs to go into this before proceeding with code. I recommend a continued discussion. My point of view is we should specifically review the way FS does the message exchange since its the cornerstone of many other possible cluster scenarios besides just fault tolerance. Some are starting to build this externally and I think the real answer is a more clean abstraction between the framework for many FS communicating with each other and the business logic. So some of what people like plivo and 2600hz do externally would be better served as part of FS at the comms level and then still separate the logic. That way everyone can benefit from the low level code. On Mon, Feb 11, 2013 at 12:37 PM, Henry Huang <red.rain.seven@gmail.com>wrote:
I would choose to go with Elliot's idea of having the HA being build within the FS itself under the condition that it's a easy adaptation comparing to having to use and struggling with Pacemaker and Corosync. They can literally take months if someone is starting from scratch.
And it would be nice to have this kind of capability to go against those brand name solutions.
On Mon, Feb 11, 2013 at 6:06 AM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com>wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested. I have already written the code to the point that it parses the configuration files and starts sending heartbeats out all of the interfaces configured. I have also already written a lot of the code that deals with the state transitions. All I am talking about doing is implementing a tiny little finite state machine. It's a pretty trivial programming task. In fact, I think it was covered in my first year at Carnegie Mellon University. Of course, I had already figured out how to write such things in high school, I just did not know what it was called at that point. My point is, that this is not yet-another-cluster-communication-engine. It is a very specific and small finite state machine designed solely with the goal in mind of making FS have just enough information to coordinate call fail over internally. If I recall correctly, a lot of people also said writing yet-another-VoIP-server was a waste of time, but now we have FreeSWITCH, and it was obviously worth the effort. And I am not even trying to do something as complex as that. If you think this is yet-another-cluster-communication-engine, you are missing the point. It is not. It never will be.
Look at Sonus, Genband, Broadsoft, Veraz, etc. All the big-name carrier-grade telecom providers have a built-in solution for automatic call fail over. The only way FreeSWITCH will ever compete with such solutions is if it also has that feature. Pacemaker and Corosync are overkill just to get FS to handle single node failures and provide call recovery. It took me a full 3 months of working with them every day to really understand how to deploy them properly in conjunction with FreeSWITCH and Postgres to provide a carrier-grade hot-standby solution which was robust enough to handle 99% of the failures I could throw at it. Granted, this was back when the configuration still needed to be written by hand in XML and prior the existence of any resource agent for FreeSWITCH. But, even with those changes, deploying Pacemaker and Corosync is not a simple task. If that is the requirement for FS to have HA, it will never truly stand a chance against commercial offerings.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Tony, I'm glad you're thinking this way. I've been looking into this myself, and I think we both got the same idea. I've been playing around with how to hook up the eventing system to external Service Bus implmentations as a starting point. Is there a chance we could all plan a meeting on the FS conference (out of band from the wednsday meetings) to maybe discuss this with the interested ppl? --Dave _____ From: Anthony Minessale [mailto:anthony.minessale@gmail.com] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Mon, 11 Feb 2013 13:30:25 -0800 Subject: Re: [Freeswitch-users] High Availability Cluster Module for FreeSWITCH I believe some more planning and thought needs to go into this before proceeding with code. I recommend a continued discussion. My point of view is we should specifically review the way FS does the message exchange since its the cornerstone of many other possible cluster scenarios besides just fault tolerance. Some are starting to build this externally and I think the real answer is a more clean abstraction between the framework for many FS communicating with each other and the business logic. So some of what people like plivo and 2600hz do externally would be better served as part of FS at the comms level and then still separate the logic. That way everyone can benefit from the low level code. On Mon, Feb 11, 2013 at 12:37 PM, Henry Huang <red.rain.seven@gmail.com> wrote: I would choose to go with Elliot's idea of having the HA being build within the FS itself under the condition that it's a easy adaptation comparing to having to use and struggling with Pacemaker and Corosync. They can literally take months if someone is starting from scratch. And it would be nice to have this kind of capability to go against those brand name solutions. On Mon, Feb 11, 2013 at 6:06 AM, Eliot Gable <egable+freeswitch@gmail.com> wrote: On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com> wrote: +1 I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase. I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested. I have already written the code to the point that it parses the configuration files and starts sending heartbeats out all of the interfaces configured. I have also already written a lot of the code that deals with the state transitions. All I am talking about doing is implementing a tiny little finite state machine. It's a pretty trivial programming task. In fact, I think it was covered in my first year at Carnegie Mellon University. Of course, I had already figured out how to write such things in high school, I just did not know what it was called at that point. My point is, that this is not yet-another-cluster-communication-engine. It is a very specific and small finite state machine designed solely with the goal in mind of making FS have just enough information to coordinate call fail over internally. If I recall correctly, a lot of people also said writing yet-another-VoIP-server was a waste of time, but now we have FreeSWITCH, and it was obviously worth the effort. And I am not even trying to do something as complex as that. If you think this is yet-another-cluster-communication-engine, you are missing the point. It is not. It never will be. Look at Sonus, Genband, Broadsoft, Veraz, etc. All the big-name carrier-grade telecom providers have a built-in solution for automatic call fail over. The only way FreeSWITCH will ever compete with such solutions is if it also has that feature. Pacemaker and Corosync are overkill just to get FS to handle single node failures and provide call recovery. It took me a full 3 months of working with them every day to really understand how to deploy them properly in conjunction with FreeSWITCH and Postgres to provide a carrier-grade hot-standby solution which was robust enough to handle 99% of the failures I could throw at it. Granted, this was back when the configuration still needed to be written by hand in XML and prior the existence of any resource agent for FreeSWITCH. But, even with those changes, deploying Pacemaker and Corosync is not a simple task. If that is the requirement for FS to have HA, it will never truly stand a chance against commercial offerings. _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Yes, on the meeting tomorrow we'll gauge the interest level of the participants and then you can figure out when to meet. Feel free to use the public conf server outside of our normally scheduled times. -MC On Tue, Feb 12, 2013 at 12:21 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** Tony,
I'm glad you're thinking this way. I've been looking into this myself, and I think we both got the same idea. I've been playing around with how to hook up the eventing system to external Service Bus implmentations as a starting point. Is there a chance we could all plan a meeting on the FS conference (out of band from the wednsday meetings) to maybe discuss this with the interested ppl?
--Dave
------------------------------ *From:* Anthony Minessale [mailto:anthony.minessale@gmail.com] *To:* FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] *Sent:* Mon, 11 Feb 2013 13:30:25 -0800 *Subject:* Re: [Freeswitch-users] High Availability Cluster Module for FreeSWITCH
I believe some more planning and thought needs to go into this before proceeding with code. I recommend a continued discussion. My point of view is we should specifically review the way FS does the message exchange since its the cornerstone of many other possible cluster scenarios besides just fault tolerance. Some are starting to build this externally and I think the real answer is a more clean abstraction between the framework for many FS communicating with each other and the business logic. So some of what people like plivo and 2600hz do externally would be better served as part of FS at the comms level and then still separate the logic. That way everyone can benefit from the low level code.
On Mon, Feb 11, 2013 at 12:37 PM, Henry Huang <red.rain.seven@gmail.com>wrote:
I would choose to go with Elliot's idea of having the HA being build within the FS itself under the condition that it's a easy adaptation comparing to having to use and struggling with Pacemaker and Corosync. They can literally take months if someone is starting from scratch.
And it would be nice to have this kind of capability to go against those brand name solutions.
On Mon, Feb 11, 2013 at 6:06 AM, Eliot Gable < egable+freeswitch@gmail.com> wrote:
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com>wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested. I have already written the code to the point that it parses the configuration files and starts sending heartbeats out all of the interfaces configured. I have also already written a lot of the code that deals with the state transitions. All I am talking about doing is implementing a tiny little finite state machine. It's a pretty trivial programming task. In fact, I think it was covered in my first year at Carnegie Mellon University. Of course, I had already figured out how to write such things in high school, I just did not know what it was called at that point. My point is, that this is not yet-another-cluster-communication-engine. It is a very specific and small finite state machine designed solely with the goal in mind of making FS have just enough information to coordinate call fail over internally. If I recall correctly, a lot of people also said writing yet-another-VoIP-server was a waste of time, but now we have FreeSWITCH, and it was obviously worth the effort. And I am not even trying to do something as complex as that. If you think this is yet-another-cluster-communication-engine, you are missing the point. It is not. It never will be.
Look at Sonus, Genband, Broadsoft, Veraz, etc. All the big-name carrier-grade telecom providers have a built-in solution for automatic call fail over. The only way FreeSWITCH will ever compete with such solutions is if it also has that feature. Pacemaker and Corosync are overkill just to get FS to handle single node failures and provide call recovery. It took me a full 3 months of working with them every day to really understand how to deploy them properly in conjunction with FreeSWITCH and Postgres to provide a carrier-grade hot-standby solution which was robust enough to handle 99% of the failures I could throw at it. Granted, this was back when the configuration still needed to be written by hand in XML and prior the existence of any resource agent for FreeSWITCH. But, even with those changes, deploying Pacemaker and Corosync is not a simple task. If that is the requirement for FS to have HA, it will never truly stand a chance against commercial offerings.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Michael S Collins Twitter: @mercutioviz http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org
I applaud your efforts on getting a specialized mechanism in the form of a freeswitch module implemented to correctly perform high availability functions. I see a great potential here for the freeswitch project to attain a whole new level of performance and viability, and I really really hope others agree. I've been struggling with this very issue; how to do correct HA with freeswitch to the point where we aren't dropping calls when something bad happens. It's held me back in a few ways, and while ill never look into a non-open source solution, I've got way to much on my plate right now to play with it now. I'll see if my company can send some $$ your way. Ive got a few clients who would be interested in this as well. People, start donating! Sent from my iPhone On Feb 9, 2013, at 12:00 PM, Eliot Gable <egable+freeswitch@gmail.com> wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-cluster-mod...
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Just my 2 eurocent, i know you said without 3rd party solution. But wouldn't this be a perfect task for Erlangs distribution and fault tolerance capabilities ? -----Ursprüngliche Nachricht----- Von: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] Im Auftrag von Brian Foster Gesendet: Montag, 11. Februar 2013 04:19 An: FreeSWITCH Users Help Betreff: Re: [Freeswitch-users] High Availability Cluster Module for FreeSWITCH I applaud your efforts on getting a specialized mechanism in the form of a freeswitch module implemented to correctly perform high availability functions. I see a great potential here for the freeswitch project to attain a whole new level of performance and viability, and I really really hope others agree. I've been struggling with this very issue; how to do correct HA with freeswitch to the point where we aren't dropping calls when something bad happens. It's held me back in a few ways, and while ill never look into a non-open source solution, I've got way to much on my plate right now to play with it now. I'll see if my company can send some $$ your way. Ive got a few clients who would be interested in this as well. People, start donating! Sent from my iPhone On Feb 9, 2013, at 12:00 PM, Eliot Gable <egable+freeswitch@gmail.com> wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-clust er-module-for-freeswitch
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
-- Eliot Gable
______________________________________________________________________ ___ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-use rs http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I also had considered the erlang way: - born clustered, can run an erlang node besides each FS instance, or run the erlang cluster on some dedicated servers - can share reg, session information with mnesia DB, in memory or on disk - add some functions in FS to recover sessions based on mnesia I wonder how the 2600hz guys are doing HA right now. Don't get me wrong, I'd saw the HA N+M wiki a long time ago and very interested how will it work. And, besides the FS HA, how will 3rd party softwares work with FS, if, say, someone connected via ESL and FS1 crashes and the float IP bind to FS2, will the ESL client fail and re-connect or it can magically connect to FS2 without any addintional work? Thanks -- Seven Du http://www.freeswitch.org.cn http://about.me/dujinfang http://www.dujinfang.com Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Monday, February 11, 2013 at 5:31 PM, Gerald Weber wrote:
Just my 2 eurocent, i know you said without 3rd party solution. But wouldn't this be a perfect task for Erlangs distribution and fault tolerance capabilities ?
-----Ursprüngliche Nachricht----- Von: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] Im Auftrag von Brian Foster Gesendet: Montag, 11. Februar 2013 04:19 An: FreeSWITCH Users Help Betreff: Re: [Freeswitch-users] High Availability Cluster Module for FreeSWITCH
I applaud your efforts on getting a specialized mechanism in the form of a freeswitch module implemented to correctly perform high availability functions. I see a great potential here for the freeswitch project to attain a whole new level of performance and viability, and I really really hope others agree.
I've been struggling with this very issue; how to do correct HA with freeswitch to the point where we aren't dropping calls when something bad happens. It's held me back in a few ways, and while ill never look into a non-open source solution, I've got way to much on my plate right now to play with it now.
I'll see if my company can send some $$ your way. Ive got a few clients who would be interested in this as well. People, start donating!
Sent from my iPhone
On Feb 9, 2013, at 12:00 PM, Eliot Gable <egable+freeswitch@gmail.com (mailto:egable+freeswitch@gmail.com)> wrote:
For the past two years, I have been working on and off on a high availability cluster module for FreeSWITCH. If you would like to see HA built into FreeSWITCH directly, instead of dealing with Pacemaker + Corosync or other 3rd party systems, please check out the module, voice your support, provide feedback and suggestions, and consider supporting it.
Some advantages over a 3rd party solution like Pacemaker include:
- Sub-second failure detection and response (meaning it can start recovering calls less than a second after a failure occurs) - Failure detection is built into FreeSWITCH so it can detect conditions which 3rd party modules cannot detect - One shared configuration file between all nodes (no per-node IP configuration required) - No fixed number of nodes - Graceful migration of calls with a maintenance mode (requires DNS)
These are just some of the benefits of building a FreeSWITCH-specific HA system. For additional details about the project, please visit:
http://wiki.freeswitch.org/wiki/Mod_ha_cluster
If you would like to see this project come to fruition, please consider financially supporting it through Kickstarter:
http://www.kickstarter.com/projects/1175121367/high-availability-clust er-module-for-freeswitch
Thanks in advance for anyone who contributes, either with just a shout-out, with feedback and/or suggestions, and especially to those who pledge some financial support.
-- Eliot Gable
______________________________________________________________________ ___ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org (mailto:consulting@freeswitch.org) http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org (mailto:FreeSWITCH-users@lists.freeswitch.org) http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-use rs http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org (mailto:consulting@freeswitch.org) http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org (mailto:FreeSWITCH-users@lists.freeswitch.org) http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org (mailto:consulting@freeswitch.org) http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org (mailto:FreeSWITCH-users@lists.freeswitch.org) http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Mon, Feb 11, 2013 at 9:03 AM, Seven Du <dujinfang@gmail.com> wrote:
And, besides the FS HA, how will 3rd party softwares work with FS, if, say, someone connected via ESL and FS1 crashes and the float IP bind to FS2, will the ESL client fail and re-connect or it can magically connect to FS2 without any addintional work?
The ESL client would have to reconnect. We could make a little ESL helper daemon you could run on some host which would do connection management for you, but it's probably a better idea to just handle it in your own code. It's not that difficult to notice a failed connection and reconnect.
participants (15)
-
Anthony Minessale
-
Avi Marcus
-
Brian Foster
-
Dave R. Kompel
-
dvl36
-
Eliot Gable
-
Gerald Weber
-
Gilad Abada
-
Henry Huang
-
Luis Daniel Lucio Quiroz
-
Marcin Gozdalik
-
Michael Collins
-
Paul Cupis
-
Seven Du
-
Steven Ayre