Re: [Freeswitch-users] High Availability Cluster Module for FreeSWITCH
2013/2/11 Eliot Gable <egable+freeswitch@gmail.com>:
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested.
I also think I understand what you are saying. It means we have trouble putting thought into writing ;)
From what I understand what you are trying to achieve is that every node in FS "cluster" knows what are the nodes and whether they are down or up. What I am saying is that this simple task is fundamentally hard. Sending and receiving multicast is easy, but keeping distributed state consistent between nodes in cluster is hard (like in really hard, harder than writing VoIP softswitch all over again), especially in case of Byzantine failures (i.e. nodes lying that they are down when they are up or other way round). I am no big expert in the area but seen at least 2 cases (MMM - http://www.xaprb.com/blog/2011/05/04/whats-wrong-with-mmm/ and Chubby in Google - http://www.read.seas.harvard.edu/~kohler/class/08w-dsi/chandra07paxos.pdf) where people were trying to write (MMM) or use (Chubby) some kind of distributed code and failed. That's why whenever I see anything related to distributed state I say that it's way beyond my understanding and best is to use something that works.
I have
already written the code to the point that it parses the configuration files and starts sending heartbeats out all of the interfaces configured. I have also already written a lot of the code that deals with the state transitions. All I am talking about doing is implementing a tiny little finite state machine. It's a pretty trivial programming task. In fact, I think it was covered in my first year at Carnegie Mellon University. Of course, I had already figured out how to write such things in high school, I just did not know what it was called at that point. My point is, that this is not yet-another-cluster-communication-engine. It is a very specific and small finite state machine designed solely with the goal in mind of making FS have just enough information to coordinate call fail over internally. If I recall correctly, a lot of people also said writing yet-another-VoIP-server was a waste of time, but now we have FreeSWITCH, and it was obviously worth the effort. And I am not even trying to do something as complex as that. If you think this is yet-another-cluster-communication-engine, you are missing the point. It is not. It never will be.
See above - if it will never be and you are trying to achieve distributed, consistent state between nodes, IMHO you are going to get it wrong. Frankly I lack the knowledge and time to check if Corosync API is perfect for this task. As Anthony suggested elsewhere maybe it is possible to abstract the communication/keeping distributed state part so that it would be easy to provide Corosync or other (OpenReplica? Zookeeper?) implementations.
Look at Sonus, Genband, Broadsoft, Veraz, etc. All the big-name carrier-grade telecom providers have a built-in solution for automatic call fail over. The only way FreeSWITCH will ever compete with such solutions is if it also has that feature. Pacemaker and Corosync are overkill just to get FS to handle single node failures and provide call recovery. It took me a full 3 months of working with them every day to really understand how to deploy them properly in conjunction with FreeSWITCH and Postgres to provide a carrier-grade hot-standby solution which was robust enough to handle 99% of the failures I could throw at it. Granted, this was back when the configuration still needed to be written by hand in XML and prior the existence of any resource agent for FreeSWITCH. But, even with those changes, deploying Pacemaker and Corosync is not a simple task. If that is the requirement for FS to have HA, it will never truly stand a chance against commercial offerings.
I believe that Clusters from Scratch http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/) allowed me to setup a working Pacemaker/Corosync installation on Debian in less than a day. It is a fair point that FS to compete with big names (at least in marketing buzz-feature checklist) has to have HA. Trouble is HA is always hard. Making it simple and work is the ultimate goal but I'd rather shoot at "work" first and later for "simple". Comparing FS to commercial offerings is hard as the way FS is deployed is usually different than commercial competitors. If you buy Broadsoft you don't get a Debian package or sources to compile - you get a bunch of highly-paid consultants that install and configure everything for you. If HA in FS would work that way it would maybe be even better for building business around FS ;) Jokes aside - if somebody wants to configure FS in HA evironment I don't think it is much of an obstacle for him to configure Corosync as already he has to have some kind of DB failover as well, redundant switches, power, etc. If some commercial vendor is able to manufacture an out-of-the box solutions based on FS with a "HA" checkbox somewhere in Web configuration - well, congratulations to him and he I hope he would be deservedly reaping profits from his product. -- Marcin Gozdalik
building business around FS ;) Jokes aside - if somebody wants to configure FS in HA evironment I don't think it is much of an obstacle for him to configure Corosync as already he has to have some kind of DB failover as well, redundant switches, power, etc.
I would like to see that statement put to the test. I would like to see someone attempt to put Corosync (or whatever existing solution) to use as the HA engine in a FS cluster. I suspect the amount of work involved is way more than has been suggested. If it really were that easy ("easy" being relative compared to what people perceive Eliot is doing) then I suspect it would have been done already. In any case, Tony makes good points about furthering the discussion. If we can improve FreeSWITCH in any way to make this process more feasible (regardless of which solution is used) then that is a win for everyone. We will *briefly* discuss this subject on tomorrow's conference call so that interested parties can decide when and how they'd like to further this discussion. -- Michael S Collins Twitter: @mercutioviz http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org
I would like to see that statement put to the test. I would like to see someone attempt to put Corosync (or whatever existing solution) to use as the HA engine in a FS cluster. I suspect the amount of work involved is way more than has been suggested. If it really were that easy ("easy" being relative compared to what people perceive Eliot is doing) then I suspect it would have been done already.
Hey MC, http://wiki.freeswitch.org/wiki/Enterprise_deployment_with_Corosync ^ I wrote that months ago. Pretty easy :)
On Tue, Feb 12, 2013 at 10:32 AM, Vik Killa <vipkilla@gmail.com> wrote:
I would like to see that statement put to the test. I would like to see someone attempt to put Corosync (or whatever existing solution) to use as the HA engine in a FS cluster. I suspect the amount of work involved is
way
more than has been suggested. If it really were that easy ("easy" being relative compared to what people perceive Eliot is doing) then I suspect it would have been done already.
Hey MC, http://wiki.freeswitch.org/wiki/Enterprise_deployment_with_Corosync ^ I wrote that months ago. Pretty easy :)
How many failure scenarios does that handle? -MC
On Tue, Feb 12, 2013 at 1:32 PM, Vik Killa <vipkilla@gmail.com> wrote:
I would like to see that statement put to the test. I would like to see someone attempt to put Corosync (or whatever existing solution) to use as the HA engine in a FS cluster. I suspect the amount of work involved is
way
more than has been suggested. If it really were that easy ("easy" being relative compared to what people perceive Eliot is doing) then I suspect it would have been done already.
Hey MC, http://wiki.freeswitch.org/wiki/Enterprise_deployment_with_Corosync ^ I wrote that months ago. Pretty easy :)
ip link set <your_interface> arp off Watch your cluster sit there like an idiot while your traffic fails and you spend hours trying to find the cause. ethtool -s <your_interface> speed 10 duplex half Watch your cluster sit there like an idiot while your traffic fails and you spend hours trying to find the cause. Rinse and repeat with literally dozens upon dozens of other things you could do where your cluster just sits there broken and you pull your hair out. Still seem simple?
Hi all, i have a question. i have installed skype on freeswitch. Microsoft changed on march the servers. Is this a problem of old skype version under 4.x? Can i update the 2.X version to current version? What skype versions supported fs? Thanks. --------------- Du kannst mich jederzeit kostenlos per Festnetz erreichen unter: http://www.blindi.net/callback homepage: http://www.blindi.net blinde-misc mailingliste für blinde. anmeldung unter: http://www.blindi.net/mailman/listinfo/blinde-misc
Hello Thomas, just follow the wiki page strictly and you'll be good to go. http://wiki.freeswitch.org/wiki/Skypopen Use the installer.pl and the skype client provided by installer.pl Anything that is not on the wiki page, or that is done differently from the wiki page, is unsupported and will take much of your time. -giovanni On Thu, Feb 14, 2013 at 11:08 PM, Thomas Hoellriegel <admin@blindi.net>wrote:
Hi all, i have a question. i have installed skype on freeswitch. Microsoft changed on march the servers. Is this a problem of old skype version under 4.x? Can i update the 2.X version to current version? What skype versions supported fs? Thanks.
--------------- Du kannst mich jederzeit kostenlos per Festnetz erreichen unter: http://www.blindi.net/callback homepage: http://www.blindi.net blinde-misc mailingliste für blinde. anmeldung unter: http://www.blindi.net/mailman/**listinfo/blinde-misc<http://www.blindi.net/mailman/listinfo/blinde-misc>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely, Giovanni Maruzzelli Cell : +39-347-2665618
Hi all, i have installed skype for linux from the wiki-page. These works fine. My problem: when i place incoming or outboundcalls the sound is very clearlly. The quality is for example a very bad gsm mobile phone line. My Fs works in a datacenter 100mbit network. I have setup a free german voicechat: dorf_telefonchat I demonstrate the problem: Then you call the chat, your heare the bad ton quality. Can your help please? thanks. --------------- Du kannst mich jederzeit kostenlos per Festnetz erreichen unter: http://www.blindi.net/callback homepage: http://www.blindi.net blinde-misc mailingliste für blinde. anmeldung unter: http://www.blindi.net/mailman/listinfo/blinde-misc
be sure to use 16khz sounds etc. Skype is 16khz. Skype call would normally be superior in quality respect to a gsm call. If you are using conference, be sure to be at 16khz. Other than that, follow the wiki strictly, particularly about which distro, and don't use virtual machines (eg real hardware has better timing) or openvz containers. Obviously, don't use weak or overloaded machines, and not 32 bit, or (horror!) machines with a graphic interface :) On 2/16/13, Thomas Hoellriegel <admin@blindi.net> wrote:
Hi all, i have installed skype for linux from the wiki-page. These works fine. My problem: when i place incoming or outboundcalls the sound is very clearlly. The quality is for example a very bad gsm mobile phone line. My Fs works in a datacenter 100mbit network. I have setup a free german voicechat: dorf_telefonchat I demonstrate the problem: Then you call the chat, your heare the bad ton quality.
Can your help please? thanks.
--------------- Du kannst mich jederzeit kostenlos per Festnetz erreichen unter: http://www.blindi.net/callback homepage: http://www.blindi.net blinde-misc mailingliste für blinde. anmeldung unter: http://www.blindi.net/mailman/listinfo/blinde-misc
-- Sincerely, Giovanni Maruzzelli Cell : +39-347-2665618
2013/2/12 Michael Collins <msc@freeswitch.org>:
building business around FS ;) Jokes aside - if somebody wants to configure FS in HA evironment I don't think it is much of an obstacle for him to configure Corosync as already he has to have some kind of DB failover as well, redundant switches, power, etc.
I would like to see that statement put to the test. I would like to see someone attempt to put Corosync (or whatever existing solution) to use as the HA engine in a FS cluster. I suspect the amount of work involved is way more than has been suggested. If it really were that easy ("easy" being relative compared to what people perceive Eliot is doing) then I suspect it would have been done already.
If there was a readily available resource agent for FS it is "easy" to setup Corosync (if you know hat you are doing). I didn't setup Corosync to handle FS, I have OpenSIPS as SBC before FS which handles HA. Of course it is not ideal, as in case of a problem current calls are lost.
If there was a readily available resource agent for FS it is "easy" to setup Corosync (if you know hat you are doing). I didn't setup Corosync to handle FS, I have OpenSIPS as SBC before FS which handles HA. Of course it is not ideal, as in case of a problem current calls are lost.
Which is precisely the point - this is *HIGH* availability we're talking about, not "kinda quickly recovering from a problem but it's okay to lose the calls that were in progress as long as our VoIP comes back up within 5-10 seconds." We're talking about the Holy Grail of VoIP here - a fault-tolerant cluster of VoIP servers that can keep calls up and running even if the primary server goes down. Note that "primary server goes down" includes literally hundreds of scenarios from physical (power, network cable unplugged, HDD crash, etc.) to logical (user misconfiguration of FreeSWITCH or LAN settings, router/firewall changes, service provider goofups, FreeSWITCH core dump, a single sofia profile locked up but other FS is okay, etc.) I'll bet Eliot can name dozens of them off the top of his head. ;) I have a sneaking suspicion that not everyone in this thread is thinking the same thing when we say "HA". Eliot's proposed solution encompasses fault scenarios that some of us haven't even dreamed of. I am open to someone getting P + C to do everything that Eliot's mod_ha_cluster can do but I suspect it isn't realistic. If you can prove me wrong on that then please by all means do so. -MC -- Michael S Collins Twitter: @mercutioviz http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org
2013/2/12 Michael Collins <msc@freeswitch.org>:
If there was a readily available resource agent for FS it is "easy" to setup Corosync (if you know hat you are doing). I didn't setup Corosync to handle FS, I have OpenSIPS as SBC before FS which handles HA. Of course it is not ideal, as in case of a problem current calls are lost.
Which is precisely the point - this is *HIGH* availability we're talking about, not "kinda quickly recovering from a problem but it's okay to lose the calls that were in progress as long as our VoIP comes back up within 5-10 seconds."
We're talking about the Holy Grail of VoIP here - a fault-tolerant cluster of VoIP servers that can keep calls up and running even if the primary server goes down. Note that "primary server goes down" includes literally hundreds of scenarios from physical (power, network cable unplugged, HDD crash, etc.) to logical (user misconfiguration of FreeSWITCH or LAN settings, router/firewall changes, service provider goofups, FreeSWITCH core dump, a single sofia profile locked up but other FS is okay, etc.) I'll bet Eliot can name dozens of them off the top of his head. ;)
I have a sneaking suspicion that not everyone in this thread is thinking the same thing when we say "HA". Eliot's proposed solution encompasses fault scenarios that some of us haven't even dreamed of. I am open to someone getting P + C to do everything that Eliot's mod_ha_cluster can do but I suspect it isn't realistic. If you can prove me wrong on that then please by all means do so.
The Holy Grail you're describing is probably impossible to achieve (even Google sometimes serves you 500, just press reload to be routed to another server :). Maybe this discussion could continue to prepare some kind of a blueprint describing the predicted failure scenarios and how the proposed solution would deal with them? There are several solutions best suited for different classes of availability you're willing to tolerate. -- Marcin Gozdalik
Marcin, have you used commercial platforms that do this kind of live-call failover? They work very well. "impossible", certainly not. Regards, Ben Langfeld On 12 February 2013 21:33, Marcin Gozdalik <gozdal@gmail.com> wrote:
2013/2/12 Michael Collins <msc@freeswitch.org>:
If there was a readily available resource agent for FS it is "easy" to setup Corosync (if you know hat you are doing). I didn't setup Corosync to handle FS, I have OpenSIPS as SBC before FS which handles HA. Of course it is not ideal, as in case of a problem current calls are lost.
Which is precisely the point - this is *HIGH* availability we're talking about, not "kinda quickly recovering from a problem but it's okay to lose the calls that were in progress as long as our VoIP comes back up within 5-10 seconds."
We're talking about the Holy Grail of VoIP here - a fault-tolerant
cluster
of VoIP servers that can keep calls up and running even if the primary server goes down. Note that "primary server goes down" includes literally hundreds of scenarios from physical (power, network cable unplugged, HDD crash, etc.) to logical (user misconfiguration of FreeSWITCH or LAN settings, router/firewall changes, service provider goofups, FreeSWITCH core dump, a single sofia profile locked up but other FS is okay, etc.) I'll bet Eliot can name dozens of them off the top of his head. ;)
I have a sneaking suspicion that not everyone in this thread is thinking the same thing when we say "HA". Eliot's proposed solution encompasses fault scenarios that some of us haven't even dreamed of. I am open to someone getting P + C to do everything that Eliot's mod_ha_cluster can do but I suspect it isn't realistic. If you can prove me wrong on that then please by all means do so.
The Holy Grail you're describing is probably impossible to achieve (even Google sometimes serves you 500, just press reload to be routed to another server :). Maybe this discussion could continue to prepare some kind of a blueprint describing the predicted failure scenarios and how the proposed solution would deal with them? There are several solutions best suited for different classes of availability you're willing to tolerate.
-- Marcin Gozdalik
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I did some, I got some anecdotes from people running them. It certainly is possible to handle *some* failure cases, but there are some *other* cases that will be not handled correctly. Personally I've observed OpenSIPS that run out of some internal memory and anyhow you wanted to monitor it it will reply it is alive. It will even route your simple call scenarios well! Unfortunately the "real" calls were *usually* timing out, but always. Sorry, but I can't imagine how you can automatically handle that, i.e. discover that this node is "bad" and fail-over to some other. Even if you do that correctly your monitoring tool can check that this node is operating perfectly well (because after all the traffic is diverted from the faulty node it begins to work well) and will want to move the traffic back. By "impossible" I mean handling all such gray areas. Certainly if you'll power down the box or unplug Ethernet it is possible to migrate live calls to some other box. I'd just like to have HA that works everytime and everywhere and try to poke some holes in reasoning in hope it'll help to reach better solution. 2013/2/12 Ben Langfeld <ben@langfeld.co.uk>:
Marcin, have you used commercial platforms that do this kind of live-call failover? They work very well. "impossible", certainly not.
Regards, Ben Langfeld
On 12 February 2013 21:33, Marcin Gozdalik <gozdal@gmail.com> wrote:
2013/2/12 Michael Collins <msc@freeswitch.org>:
If there was a readily available resource agent for FS it is "easy" to setup Corosync (if you know hat you are doing). I didn't setup Corosync to handle FS, I have OpenSIPS as SBC before FS which handles HA. Of course it is not ideal, as in case of a problem current calls are lost.
Which is precisely the point - this is *HIGH* availability we're talking about, not "kinda quickly recovering from a problem but it's okay to lose the calls that were in progress as long as our VoIP comes back up within 5-10 seconds."
We're talking about the Holy Grail of VoIP here - a fault-tolerant cluster of VoIP servers that can keep calls up and running even if the primary server goes down. Note that "primary server goes down" includes literally hundreds of scenarios from physical (power, network cable unplugged, HDD crash, etc.) to logical (user misconfiguration of FreeSWITCH or LAN settings, router/firewall changes, service provider goofups, FreeSWITCH core dump, a single sofia profile locked up but other FS is okay, etc.) I'll bet Eliot can name dozens of them off the top of his head. ;)
I have a sneaking suspicion that not everyone in this thread is thinking the same thing when we say "HA". Eliot's proposed solution encompasses fault scenarios that some of us haven't even dreamed of. I am open to someone getting P + C to do everything that Eliot's mod_ha_cluster can do but I suspect it isn't realistic. If you can prove me wrong on that then please by all means do so.
The Holy Grail you're describing is probably impossible to achieve (even Google sometimes serves you 500, just press reload to be routed to another server :). Maybe this discussion could continue to prepare some kind of a blueprint describing the predicted failure scenarios and how the proposed solution would deal with them? There are several solutions best suited for different classes of availability you're willing to tolerate.
-- Marcin Gozdalik
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Marcin Gozdalik
On Tue, Feb 12, 2013 at 5:21 PM, Marcin Gozdalik <gozdal@gmail.com> wrote:
I did some, I got some anecdotes from people running them. It certainly is possible to handle *some* failure cases, but there are some *other* cases that will be not handled correctly.
I have actually used Sonus, Genband, both of which seamlessly and gracefully handle failures. In fact, I cannot think of a single time where either of them failed to migrate calls over to an alternative system when the one handling calls had even one thing go wrong which prevented them from handling calls properly. And, as I mentioned earlier, I have also used P + C to build a multi-city, 6-node FS cluster (three iterations of it, actually) which could do essentially the same thing. While I don't have the resource agent I wrote for that cluster which handled the vast majority of failure conditions (including the two I mentioned earlier where you turn off ARP on the interface or set the speed/duplex wrong), I could fairly easily update my newly written resource agent to cover such scenarios (if someone wanted to pay for my time to do it).
Personally I've observed OpenSIPS that run out of some internal memory and anyhow you wanted to monitor it it will reply it is alive. It will even route your simple call scenarios well! Unfortunately the "real" calls were *usually* timing out, but always. Sorry, but I can't imagine how you can automatically handle that, i.e. discover that this node is "bad" and fail-over to some other. Even if you do that correctly your monitoring tool can check that this node is operating perfectly well (because after all the traffic is diverted from the faulty node it begins to work well) and will want to move the traffic back.
Just because you cannot imagine how it would work, it does not mean everyone else has the same limitations. You are consistently referring to a node (let's call it node A) telling one or more other nodes (B, C, whatever) about its state and ability to handle calls. You are taking the wrong approach entirely. You cannot have a failed node reliably report it's state! What you can do is have Node A broadcast its state many times per second. Nodes B, C, etc, all listen for that state. They keep track of the state and when they *stop seeing* Node A, they know *something* failed. It's that simple. At this point, it is up to those nodes to determine *what* failed. They have to do things like check whether *they* failed, check whether the *network* failed, etc. There are very simple and sane ways they can do each of those things. Eventually, they can conclude that *Node A* failed. They can do this determination very quickly. Think microseconds or maybe 1 - 2 milliseconds quickly. At this point, nobody tells them what to do. They know already what they need to do. They have already pre-negotiated what will happen in the event of a failure, or they have very specific programmatic instructions on what to do, and so they act immediately.
By "impossible" I mean handling all such gray areas. Certainly if you'll power down the box or unplug Ethernet it is possible to migrate live calls to some other box.
There are no such "gray" areas. That is just a fantasy you have. Everything in computing is black or white, true or false. If you don't know, you test and become certain. If you are prevented from accurately testing because (for example) you cannot see the node at all anymore or interact with it in any way, you assume the worst and nuke the box (STONITH). Let's say the FS box runs out of memory. Great! I designed mod_ha_cluster to cause FS to segfault if it runs out of memory. Heartbeats stopped, other node takes over. No gray area. Wait. Did the IP get removed from the box? No? Don't know? STONITH. Did that fail? Seriously? You deployed the system wrong; don't blame me for your mistakes. Did the hard drive go away? Great! I have a test for that and a way to tell the other nodes I need to be STONITH'd if I cannot reboot myself. Did FS deadlock? Great! No more heartbeats. Other node takes over. STONITH. Did a module in FS take an event from the eventing thread and get stuck spinning, never to return? Great! No more heartbeats. Other node takes over. STONITH. Did a module in FS launch 128 threads, all of which want to use 100% CPU? Great! Untimely heartbeat delivery, other node takes over. STONITH. Did your dual-router network have the connection between the two routers go down leaving you with a split network? Great! If you have that secondary network I talked about, it's all properly detected and handled for you! If not, well, don't blame me for your failures. Did someone slap a firewall rule on the box and we suddenly cannot accept SIP messages? Great! One of the other nodes in the cluster will be sending us test SIP traffic on occasion and when we see it doesn't work anymore, we shut down and another node takes over.
I'd just like to have HA that works everytime and everywhere and try
So does everyone else. That is why I want to write mod_ha_cluster. Because what is there right now is overly complex, difficult to configure and test, and does not and cannot catch all of the possible ways in which a FS system can fail.
Eliot, for those of us not running our own network, which datacenters/colos offer multiple networks/nics? e.g. amazon, rackspace, linode, softlayer..? It's not something I recall seeing mentioned... -Avi On Wed, Feb 13, 2013 at 1:18 AM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Tue, Feb 12, 2013 at 5:21 PM, Marcin Gozdalik <gozdal@gmail.com> wrote:
I did some, I got some anecdotes from people running them. It certainly is possible to handle *some* failure cases, but there are some *other* cases that will be not handled correctly.
I have actually used Sonus, Genband, both of which seamlessly and gracefully handle failures. In fact, I cannot think of a single time where either of them failed to migrate calls over to an alternative system when the one handling calls had even one thing go wrong which prevented them from handling calls properly. And, as I mentioned earlier, I have also used P + C to build a multi-city, 6-node FS cluster (three iterations of it, actually) which could do essentially the same thing. While I don't have the resource agent I wrote for that cluster which handled the vast majority of failure conditions (including the two I mentioned earlier where you turn off ARP on the interface or set the speed/duplex wrong), I could fairly easily update my newly written resource agent to cover such scenarios (if someone wanted to pay for my time to do it).
Personally I've observed OpenSIPS that run out of some internal memory and anyhow you wanted to monitor it it will reply it is alive. It will even route your simple call scenarios well! Unfortunately the "real" calls were *usually* timing out, but always. Sorry, but I can't imagine how you can automatically handle that, i.e. discover that this node is "bad" and fail-over to some other. Even if you do that correctly your monitoring tool can check that this node is operating perfectly well (because after all the traffic is diverted from the faulty node it begins to work well) and will want to move the traffic back.
Just because you cannot imagine how it would work, it does not mean everyone else has the same limitations. You are consistently referring to a node (let's call it node A) telling one or more other nodes (B, C, whatever) about its state and ability to handle calls. You are taking the wrong approach entirely. You cannot have a failed node reliably report it's state! What you can do is have Node A broadcast its state many times per second. Nodes B, C, etc, all listen for that state. They keep track of the state and when they *stop seeing* Node A, they know *something* failed. It's that simple.
At this point, it is up to those nodes to determine *what* failed. They have to do things like check whether *they* failed, check whether the *network* failed, etc. There are very simple and sane ways they can do each of those things. Eventually, they can conclude that *Node A* failed. They can do this determination very quickly. Think microseconds or maybe 1 - 2 milliseconds quickly. At this point, nobody tells them what to do. They know already what they need to do. They have already pre-negotiated what will happen in the event of a failure, or they have very specific programmatic instructions on what to do, and so they act immediately.
By "impossible" I mean handling all such gray areas. Certainly if you'll power down the box or unplug Ethernet it is possible to migrate live calls to some other box.
There are no such "gray" areas. That is just a fantasy you have. Everything in computing is black or white, true or false. If you don't know, you test and become certain. If you are prevented from accurately testing because (for example) you cannot see the node at all anymore or interact with it in any way, you assume the worst and nuke the box (STONITH).
Let's say the FS box runs out of memory. Great! I designed mod_ha_cluster to cause FS to segfault if it runs out of memory. Heartbeats stopped, other node takes over. No gray area. Wait. Did the IP get removed from the box? No? Don't know? STONITH. Did that fail? Seriously? You deployed the system wrong; don't blame me for your mistakes.
Did the hard drive go away? Great! I have a test for that and a way to tell the other nodes I need to be STONITH'd if I cannot reboot myself.
Did FS deadlock? Great! No more heartbeats. Other node takes over. STONITH.
Did a module in FS take an event from the eventing thread and get stuck spinning, never to return? Great! No more heartbeats. Other node takes over. STONITH.
Did a module in FS launch 128 threads, all of which want to use 100% CPU? Great! Untimely heartbeat delivery, other node takes over. STONITH.
Did your dual-router network have the connection between the two routers go down leaving you with a split network? Great! If you have that secondary network I talked about, it's all properly detected and handled for you! If not, well, don't blame me for your failures.
Did someone slap a firewall rule on the box and we suddenly cannot accept SIP messages? Great! One of the other nodes in the cluster will be sending us test SIP traffic on occasion and when we see it doesn't work anymore, we shut down and another node takes over.
I'd just like to have HA that works everytime and everywhere and try
So does everyone else. That is why I want to write mod_ha_cluster. Because what is there right now is overly complex, difficult to configure and test, and does not and cannot catch all of the possible ways in which a FS system can fail.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Tue, Feb 12, 2013 at 6:59 PM, Avi Marcus <avi@avimarcus.net> wrote:
Eliot, for those of us not running our own network, which datacenters/colos offer multiple networks/nics? e.g. amazon, rackspace, linode, softlayer..?
It's not something I recall seeing mentioned...
-Avi
Honestly, I have never utilized such services. I have always worked for / with companies which run their own network. Regardless of what datacenter / colo you use, you need to ensure you have redundant communications between your nodes. This is true for *every* HA deployment. The way Sonus and other providers do it is to use a physical chassis which provides the redundant communication mechanism. When you do not have such a chassis, you must use a separate network connection. When you have only two nodes, it's easy to do a crossover cable between them. However, if you have lots of nodes, you need an entirely separate network. That separate network only needs to facilitate communication between nodes. It does not need Internet connectivity.
This is something where you'll need to get a physical rack and install your own setup. It's too specialised for you to find anyone offering it as standard. You're probably going to want them to be pretty close together connected via a dedicated LAN anyway - otherwise packet loss / routing loops tin/ latency / jitter will mean your heartbeats start failing. Similarly you don't want anything stealing your CPU away preventing you from sending heartbeats at the correct time, so VPS options like Amazon AWS are not going to be appropriate either. And you're going to want STONITH support too (eg via power sockets that're controllable via IP) - it's unlikely a dedicated server from someone like rackspace will offer that and shutting down virtual machine instances aren't reliable enough for STONITH (the guest could be down because of a fault on the host which might leave you unable to shutdown the guest). -Steve On 12 February 2013 23:59, Avi Marcus <avi@avimarcus.net> wrote:
Eliot, for those of us not running our own network, which datacenters/colos offer multiple networks/nics? e.g. amazon, rackspace, linode, softlayer..?
It's not something I recall seeing mentioned...
-Avi
On Wed, Feb 13, 2013 at 1:18 AM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Tue, Feb 12, 2013 at 5:21 PM, Marcin Gozdalik <gozdal@gmail.com>wrote:
I did some, I got some anecdotes from people running them. It certainly is possible to handle *some* failure cases, but there are some *other* cases that will be not handled correctly.
I have actually used Sonus, Genband, both of which seamlessly and gracefully handle failures. In fact, I cannot think of a single time where either of them failed to migrate calls over to an alternative system when the one handling calls had even one thing go wrong which prevented them from handling calls properly. And, as I mentioned earlier, I have also used P + C to build a multi-city, 6-node FS cluster (three iterations of it, actually) which could do essentially the same thing. While I don't have the resource agent I wrote for that cluster which handled the vast majority of failure conditions (including the two I mentioned earlier where you turn off ARP on the interface or set the speed/duplex wrong), I could fairly easily update my newly written resource agent to cover such scenarios (if someone wanted to pay for my time to do it).
Personally I've observed OpenSIPS that run out of some internal memory and anyhow you wanted to monitor it it will reply it is alive. It will even route your simple call scenarios well! Unfortunately the "real" calls were *usually* timing out, but always. Sorry, but I can't imagine how you can automatically handle that, i.e. discover that this node is "bad" and fail-over to some other. Even if you do that correctly your monitoring tool can check that this node is operating perfectly well (because after all the traffic is diverted from the faulty node it begins to work well) and will want to move the traffic back.
Just because you cannot imagine how it would work, it does not mean everyone else has the same limitations. You are consistently referring to a node (let's call it node A) telling one or more other nodes (B, C, whatever) about its state and ability to handle calls. You are taking the wrong approach entirely. You cannot have a failed node reliably report it's state! What you can do is have Node A broadcast its state many times per second. Nodes B, C, etc, all listen for that state. They keep track of the state and when they *stop seeing* Node A, they know *something* failed. It's that simple.
At this point, it is up to those nodes to determine *what* failed. They have to do things like check whether *they* failed, check whether the *network* failed, etc. There are very simple and sane ways they can do each of those things. Eventually, they can conclude that *Node A* failed. They can do this determination very quickly. Think microseconds or maybe 1 - 2 milliseconds quickly. At this point, nobody tells them what to do. They know already what they need to do. They have already pre-negotiated what will happen in the event of a failure, or they have very specific programmatic instructions on what to do, and so they act immediately.
By "impossible" I mean handling all such gray areas. Certainly if you'll power down the box or unplug Ethernet it is possible to migrate live calls to some other box.
There are no such "gray" areas. That is just a fantasy you have. Everything in computing is black or white, true or false. If you don't know, you test and become certain. If you are prevented from accurately testing because (for example) you cannot see the node at all anymore or interact with it in any way, you assume the worst and nuke the box (STONITH).
Let's say the FS box runs out of memory. Great! I designed mod_ha_cluster to cause FS to segfault if it runs out of memory. Heartbeats stopped, other node takes over. No gray area. Wait. Did the IP get removed from the box? No? Don't know? STONITH. Did that fail? Seriously? You deployed the system wrong; don't blame me for your mistakes.
Did the hard drive go away? Great! I have a test for that and a way to tell the other nodes I need to be STONITH'd if I cannot reboot myself.
Did FS deadlock? Great! No more heartbeats. Other node takes over. STONITH.
Did a module in FS take an event from the eventing thread and get stuck spinning, never to return? Great! No more heartbeats. Other node takes over. STONITH.
Did a module in FS launch 128 threads, all of which want to use 100% CPU? Great! Untimely heartbeat delivery, other node takes over. STONITH.
Did your dual-router network have the connection between the two routers go down leaving you with a split network? Great! If you have that secondary network I talked about, it's all properly detected and handled for you! If not, well, don't blame me for your failures.
Did someone slap a firewall rule on the box and we suddenly cannot accept SIP messages? Great! One of the other nodes in the cluster will be sending us test SIP traffic on occasion and when we see it doesn't work anymore, we shut down and another node takes over.
I'd just like to have HA that works everytime and everywhere and try
So does everyone else. That is why I want to write mod_ha_cluster. Because what is there right now is overly complex, difficult to configure and test, and does not and cannot catch all of the possible ways in which a FS system can fail.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Tue, Feb 12, 2013 at 3:18 PM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Tue, Feb 12, 2013 at 5:21 PM, Marcin Gozdalik <gozdal@gmail.com> wrote:
I did some, I got some anecdotes from people running them. It certainly is possible to handle *some* failure cases, but there are some *other* cases that will be not handled correctly.
I have actually used Sonus, Genband, both of which seamlessly and gracefully handle failures. In fact, I cannot think of a single time where either of them failed to migrate calls over to an alternative system when the one handling calls had even one thing go wrong which prevented them from handling calls properly. And, as I mentioned earlier, I have also used P + C to build a multi-city, 6-node FS cluster (three iterations of it, actually) which could do essentially the same thing. While I don't have the resource agent I wrote for that cluster which handled the vast majority of failure conditions (including the two I mentioned earlier where you turn off ARP on the interface or set the speed/duplex wrong), I could fairly easily update my newly written resource agent to cover such scenarios (if someone wanted to pay for my time to do it).
Personally I've observed OpenSIPS that run out of some internal memory and anyhow you wanted to monitor it it will reply it is alive. It will even route your simple call scenarios well! Unfortunately the "real" calls were *usually* timing out, but always. Sorry, but I can't imagine how you can automatically handle that, i.e. discover that this node is "bad" and fail-over to some other. Even if you do that correctly your monitoring tool can check that this node is operating perfectly well (because after all the traffic is diverted from the faulty node it begins to work well) and will want to move the traffic back.
Just because you cannot imagine how it would work, it does not mean everyone else has the same limitations. You are consistently referring to a node (let's call it node A) telling one or more other nodes (B, C, whatever) about its state and ability to handle calls. You are taking the wrong approach entirely. You cannot have a failed node reliably report it's state! What you can do is have Node A broadcast its state many times per second. Nodes B, C, etc, all listen for that state. They keep track of the state and when they *stop seeing* Node A, they know *something* failed. It's that simple.
At this point, it is up to those nodes to determine *what* failed. They have to do things like check whether *they* failed, check whether the *network* failed, etc. There are very simple and sane ways they can do each of those things. Eventually, they can conclude that *Node A* failed. They can do this determination very quickly. Think microseconds or maybe 1 - 2 milliseconds quickly. At this point, nobody tells them what to do. They know already what they need to do. They have already pre-negotiated what will happen in the event of a failure, or they have very specific programmatic instructions on what to do, and so they act immediately.
By "impossible" I mean handling all such gray areas. Certainly if you'll power down the box or unplug Ethernet it is possible to migrate live calls to some other box.
There are no such "gray" areas. That is just a fantasy you have. Everything in computing is black or white, true or false. If you don't know, you test and become certain. If you are prevented from accurately testing because (for example) you cannot see the node at all anymore or interact with it in any way, you assume the worst and nuke the box (STONITH).
Let's say the FS box runs out of memory. Great! I designed mod_ha_cluster to cause FS to segfault if it runs out of memory. Heartbeats stopped, other node takes over. No gray area. Wait. Did the IP get removed from the box? No? Don't know? STONITH. Did that fail? Seriously? You deployed the system wrong; don't blame me for your mistakes.
Did the hard drive go away? Great! I have a test for that and a way to tell the other nodes I need to be STONITH'd if I cannot reboot myself.
Did FS deadlock? Great! No more heartbeats. Other node takes over. STONITH.
Did a module in FS take an event from the eventing thread and get stuck spinning, never to return? Great! No more heartbeats. Other node takes over. STONITH.
Did a module in FS launch 128 threads, all of which want to use 100% CPU? Great! Untimely heartbeat delivery, other node takes over. STONITH.
Did your dual-router network have the connection between the two routers go down leaving you with a split network? Great! If you have that secondary network I talked about, it's all properly detected and handled for you! If not, well, don't blame me for your failures.
Did someone slap a firewall rule on the box and we suddenly cannot accept SIP messages? Great! One of the other nodes in the cluster will be sending us test SIP traffic on occasion and when we see it doesn't work anymore, we shut down and another node takes over.
I'd just like to have HA that works everytime and everywhere and try
So does everyone else. That is why I want to write mod_ha_cluster. Because what is there right now is overly complex, difficult to configure and test, and does not and cannot catch all of the possible ways in which a FS system can fail.
+1 again. If you have reservations about what Eliot is doing then great - offer a valid alternative or ask a valid question. However, let's not make assumptions about what can and cannot be accomplished when the discussion is barely 3 days old. Eliot has answered a lot of questions here and has also done a good job of eliminating the FUD while demystifying what happens in a mod_ha_cluster.
Be sure to join us tomorrow on the FreeSWITCH conference call<http://wiki.freeswitch.org/wiki/FS_weekly_2013_02_13>and we'll talk about how to move forward and who wants to be part of the discussion. -MC
On Tue, Feb 12, 2013 at 9:21 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
2013/2/11 Eliot Gable <egable+freeswitch@gmail.com>:
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested.
I also think I understand what you are saying. It means we have trouble putting thought into writing ;)
From what I understand what you are trying to achieve is that every node in FS "cluster" knows what are the nodes and whether they are down or up. What I am saying is that this simple task is fundamentally hard. Sending and receiving multicast is easy, but keeping distributed state consistent between nodes in cluster is hard (like in really hard, harder than writing VoIP softswitch all over again), especially in case of Byzantine failures (i.e. nodes lying that they are down when they are up or other way round). I am no big expert in the area but seen at least 2 cases (MMM - http://www.xaprb.com/blog/2011/05/04/whats-wrong-with-mmm/ and Chubby in Google - http://www.read.seas.harvard.edu/~kohler/class/08w-dsi/chandra07paxos.pdf) where people were trying to write (MMM) or use (Chubby) some kind of distributed code and failed. That's why whenever I see anything related to distributed state I say that it's way beyond my understanding and best is to use something that works.
You were fortunate to have that resource available, as well as (I assume) an already made resource agent available for managing FreeSWITCH. I had to learn it from this: http://clusterlabs.org/doc/en-US/Pacemaker/1.0/pdf/Pacemaker_Explained/Pacem... I also had to craft a resource agent to manage FreeSWITCH (none existed at the time). Then I found out Pacemaker was buggy (it has gotten much better since I started using it) and wouldn't properly honor colocation constraints or grouping correctly in certain failure conditions, so I had to make the resource agent handle managing all the IP addresses for FreeSWITCH (each instance had 12 Sofia profiles with each one running on a different IP). I spent months testing hundreds of different possible failure conditions and fixing dozens if not hundreds of bugs in the configuration and in how the resource agent managed everything and reported on the health of FreeSWITCH. Everything from someone accidentally removing a needed IP from the system to a failed hard drive to a Sofia profile failing to load to firewall rules accidentally blocking needed ports, etc. If you spent only one day setting up such a system, I am certain you failed to account for dozens if not hundreds of possible failure conditions. At the end of those 3 months of hell, I had a single pair of nodes which I could rely on to "do the right thing" under practically any failure condition. However, even then, I still had several dozen ways I could simulate FreeSWITCH failing which the system simply could not detect efficiently. I made attempts at testing some of them, but the load induced on the system to test them frequently enough to matter made the system fall outside the specifications I needed for the project to be profitable and workable. I have years of experience building and deploying FreeSWITCH clusters with Pacemaker and Corosync and hunting down and gracefully handling practically every conceivable way such a system could fail. I understand you think it's hard to do; and that is not without reason. I've lived it; I've done it. I know what's involved in the process. I simply want to take my experience with it and write it down in code in the form of mod_ha_cluster so that other people don't have to waste their time relearning all the things I already know with regard to making FreeSWITCH run in an HA setup. In the absence of Pacemaker and Corosync, my goal is to provide mod_ha_cluster enough awareness that the vast majority of failure cases are handled gracefully and FS can take care of itself for bringing a slave online to take over for a failed node. However, there is no reason I cannot also write it to let Pacemaker and Corosync give it direction as to which slave to turn into a master. So, if it makes you more comfortable, think of it as a glorified resource agent which always happens to know about the "deep state" of the nodes and can test for things that traditional resource agents can never do effectively. Then, when you do a "shallow" poll of the state, you can get back the "deep state" instead, but at the cost of doing a "shallow" test. And, on top of all that, it will handle synchronizing various data between the nodes so you don't need to rely on an external HA database.
On Tue, Feb 12, 2013 at 10:19 AM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
On Tue, Feb 12, 2013 at 9:21 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
2013/2/11 Eliot Gable <egable+freeswitch@gmail.com>:
On Mon, Feb 11, 2013 at 7:36 AM, Marcin Gozdalik <gozdal@gmail.com> wrote:
+1
I do not doubt mod_ha is necessary inside of FS and it may be better/simpler than writing Pacemaker resource agent, but writing yet-another-cluster-communication-engine is IMHO the wrong way to go and using Corosync for communication will give a lot of value from mature codebase.
I understand what you are saying, but what I am trying to get across is that I am not writing yet-another-cluster-communication-engine. All I am really doing is combining a multicast messaging API written by Tony and the event API in FS to broadcast existing state information between multiple FS nodes, as well as adding a tiny amount of logic on top of that to coordinate call fail over and recovery. That's probably a little over-simplified, but it gets the point across. The network communication code is already in FS and well tested. The event system is already in FS and well tested.
I also think I understand what you are saying. It means we have trouble putting thought into writing ;)
From what I understand what you are trying to achieve is that every node in FS "cluster" knows what are the nodes and whether they are down or up. What I am saying is that this simple task is fundamentally hard. Sending and receiving multicast is easy, but keeping distributed state consistent between nodes in cluster is hard (like in really hard, harder than writing VoIP softswitch all over again), especially in case of Byzantine failures (i.e. nodes lying that they are down when they are up or other way round). I am no big expert in the area but seen at least 2 cases (MMM - http://www.xaprb.com/blog/2011/05/04/whats-wrong-with-mmm/ and Chubby in Google - http://www.read.seas.harvard.edu/~kohler/class/08w-dsi/chandra07paxos.pdf ) where people were trying to write (MMM) or use (Chubby) some kind of distributed code and failed. That's why whenever I see anything related to distributed state I say that it's way beyond my understanding and best is to use something that works.
You were fortunate to have that resource available, as well as (I assume) an already made resource agent available for managing FreeSWITCH. I had to learn it from this:
http://clusterlabs.org/doc/en-US/Pacemaker/1.0/pdf/Pacemaker_Explained/Pacem...
I also had to craft a resource agent to manage FreeSWITCH (none existed at the time). Then I found out Pacemaker was buggy (it has gotten much better since I started using it) and wouldn't properly honor colocation constraints or grouping correctly in certain failure conditions, so I had to make the resource agent handle managing all the IP addresses for FreeSWITCH (each instance had 12 Sofia profiles with each one running on a different IP). I spent months testing hundreds of different possible failure conditions and fixing dozens if not hundreds of bugs in the configuration and in how the resource agent managed everything and reported on the health of FreeSWITCH. Everything from someone accidentally removing a needed IP from the system to a failed hard drive to a Sofia profile failing to load to firewall rules accidentally blocking needed ports, etc. If you spent only one day setting up such a system, I am certain you failed to account for dozens if not hundreds of possible failure conditions. At the end of those 3 months of hell, I had a single pair of nodes which I could rely on to "do the right thing" under practically any failure condition. However, even then, I still had several dozen ways I could simulate FreeSWITCH failing which the system simply could not detect efficiently. I made attempts at testing some of them, but the load induced on the system to test them frequently enough to matter made the system fall outside the specifications I needed for the project to be profitable and workable.
I have years of experience building and deploying FreeSWITCH clusters with Pacemaker and Corosync and hunting down and gracefully handling practically every conceivable way such a system could fail. I understand you think it's hard to do; and that is not without reason. I've lived it; I've done it. I know what's involved in the process. I simply want to take my experience with it and write it down in code in the form of mod_ha_cluster so that other people don't have to waste their time relearning all the things I already know with regard to making FreeSWITCH run in an HA setup. In the absence of Pacemaker and Corosync, my goal is to provide mod_ha_cluster enough awareness that the vast majority of failure cases are handled gracefully and FS can take care of itself for bringing a slave online to take over for a failed node. However, there is no reason I cannot also write it to let Pacemaker and Corosync give it direction as to which slave to turn into a master. So, if it makes you more comfortable, think of it as a glorified resource agent which always happens to know about the "deep state" of the nodes and can test for things that traditional resource agents can never do effectively. Then, when you do a "shallow" poll of the state, you can get back the "deep state" instead, but at the cost of doing a "shallow" test. And, on top of all that, it will handle synchronizing various data between the nodes so you don't need to rely on an external HA database.
+100!
In my other post to this thread I postulated that if it were that easy then someone else would have done it already or that it should be "easily" doable. I was not aware of your excruciatingly intimate familiarity with the solutions that the others have been suggesting as alternatives for what you are working on. I think the "glorified resource agent" analogy is particularly descriptive. :) Hopefully others will chime on on the other thread about the message bus/shared key storage<http://lists.freeswitch.org/pipermail/freeswitch-users/2013-February/092185.html>so we can take this feature to the next level! -- Michael S Collins Twitter: @mercutioviz http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org
participants (9)
-
Avi Marcus
-
Ben Langfeld
-
Eliot Gable
-
Giovanni Maruzzelli
-
Marcin Gozdalik
-
Michael Collins
-
Steven Ayre
-
Thomas Hoellriegel
-
Vik Killa