FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc. -- Eliot Gable
I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global. I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering. --Dave _____ From: Eliot Gable [mailto:egable+freeswitch@gmail.com] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Tue, 12 Feb 2013 05:49:13 -0800 Subject: [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc. -- Eliot Gable
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas... João Mesquita FreeSWITCH™ Solutions On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
------------------------------ *From:* Eliot Gable [mailto:egable+freeswitch@gmail.com] *To:* FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] *Sent:* Tue, 12 Feb 2013 05:49:13 -0800 *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility. I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/ There are some drawbacks to it, namely: 1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly. 2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible. 3) They boast about how it can handle up to 8,000 1KB messages per second. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some. Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin. There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster. There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node. If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause. On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org>wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH™ Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
------------------------------ *From:* Eliot Gable [mailto:egable+freeswitch@gmail.com] *To:* FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org ] *Sent:* Tue, 12 Feb 2013 05:49:13 -0800 *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable "We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower "I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower "Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
The Spread License is pretty much a deal killer imo... Unless they are willing to eliminate that Clause for use with FreeSWITCH... That¹s as bad as the AGPL... On 2/17/13 9:50 AM, "Eliot Gable" <egable+freeswitch@gmail.com> wrote:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
There are some drawbacks to it, namely:
1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly.
2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible.
3) They boast about how it can handle up to 8,000 1KB messages per second. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some.
Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin.
There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster.
There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org> wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global. I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering. --Dave
From: Eliot Gable [mailto:egable+freeswitch@gmail.com <mailto:egable%2Bfreeswitch@gmail.com> ] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Tue, 12 Feb 2013 05:49:13 -0800 Subject: [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Restrictive licenses ... Ain't nobody got time fo dat! On Sun, Feb 17, 2013 at 10:32 AM, Ken Rice <krice@freeswitch.org> wrote:
The Spread License is pretty much a deal killer imo... Unless they are willing to eliminate that Clause for use with FreeSWITCH... That’s as bad as the AGPL...
On 2/17/13 9:50 AM, "Eliot Gable" <egable+freeswitch@gmail.com> wrote:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
There are some drawbacks to it, namely:
1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly.
2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible.
3) They boast about how it can handle up to 8,000 1KB messages per second. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some.
Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin.
There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster.
There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org> wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH™ Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
------------------------------ *From:* Eliot Gable [mailto:egable+freeswitch@gmail.com<egable+freeswitch@gmail.com>< mailto:egable%2Bfreeswitch@gmail.com <egable%2Bfreeswitch@gmail.com>> ]
*To:* FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org<freeswitch-users@lists.freeswitch.org> ] *Sent:* Tue, 12 Feb 2013 05:49:13 -0800 *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Please excuse my ignorance, but can you explain again why zmq won't work? Regarding boasting about messaging capability.. it was made to be quite lightweight - supposedly it can easily saturate a 1gigabit link with message size over 150 bytes each: http://www.zeromq.org/results:10gbe-tests If you want to use it for a queue, then it's lack of persistence and assurance that messages get through might not be helpful, but for a broadcast to whoever wants to listen it sounds great for heartbeats, events, etc. -Avi Marcus 1-718-989-9485 (USA) 1-866-202-5850 (USA & Canada Toll Free) 02-372-1570 (Israel) 020-3298-2875 (UK) On Sun, Feb 17, 2013 at 5:50 PM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
There are some drawbacks to it, namely:
1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH.. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly.
2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible.
3) They boast about how it can handle up to 8,000 1KB messages per second.. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some.
Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin.
There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster.
There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org>wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH™ Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
------------------------------ *From:* Eliot Gable [mailto:egable+freeswitch@gmail.com] *To:* FreeSWITCH Users Help [mailto: freeswitch-users@lists.freeswitch.org] *Sent:* Tue, 12 Feb 2013 05:49:13 -0800 *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable
"We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower
"I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower
"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
It has a strict philosophy on not using locks or mutexes and only doing com over socket. When you create an instance of it, it uses its own threads internally and that model gets messed up severely if you call fork from the process (even when you are doing it for something else like system command). Its basically an engine waiting for a host program rather than something you can integrated into a more complex application. Also the multicast has issues, there is no way to specificy host local scope for hops. Finally, their philosophy is to ignore the dilemma of sub/pub. There are no subscriptions to events, there is only send all events to everyone and let them filter locally. Its easy to believe this is adequate when doing pre-determined test scenarios but we know in reality that in a busy FS box we, cant send *all* the events to *everyone* and let them filter locally. On Sun, Feb 17, 2013 at 11:32 AM, Avi Marcus <avi@avimarcus.net> wrote:
Please excuse my ignorance, but can you explain again why zmq won't work?
Regarding boasting about messaging capability.. it was made to be quite lightweight - supposedly it can easily saturate a 1gigabit link with message size over 150 bytes each: http://www.zeromq.org/results:10gbe-tests
If you want to use it for a queue, then it's lack of persistence and assurance that messages get through might not be helpful, but for a broadcast to whoever wants to listen it sounds great for heartbeats, events, etc.
-Avi Marcus
1-718-989-9485 (USA) 1-866-202-5850 (USA & Canada Toll Free) 02-372-1570 (Israel) 020-3298-2875 (UK)
On Sun, Feb 17, 2013 at 5:50 PM, Eliot Gable <egable+freeswitch@gmail.com>wrote:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
There are some drawbacks to it, namely:
1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly.
2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible.
3) They boast about how it can handle up to 8,000 1KB messages per second. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some.
Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin.
There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster.
There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org>wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH™ Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
------------------------------ *From:* Eliot Gable [mailto:egable+freeswitch@gmail.com] *To:* FreeSWITCH Users Help [mailto: freeswitch-users@lists.freeswitch.org] *Sent:* Tue, 12 Feb 2013 05:49:13 -0800 *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable
"We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower
"I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower
"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Regarding #3, it seems they got to it for zeromq v3.
From ØMQ 3.x, filtering happens at the publisher side, when using a connected protocol (tcp: *or ipc:*). Using the epgm:// protocol, filtering happens at the subscriber side.
In ØMQ/2.x, all filtering happened at the subscriber side. -Avi On Sun, Feb 17, 2013 at 7:51 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It has a strict philosophy on not using locks or mutexes and only doing com over socket. When you create an instance of it, it uses its own threads internally and that model gets messed up severely if you call fork from the process (even when you are doing it for something else like system command). Its basically an engine waiting for a host program rather than something you can integrated into a more complex application. Also the multicast has issues, there is no way to specificy host local scope for hops. Finally, their philosophy is to ignore the dilemma of sub/pub. There are no subscriptions to events, there is only send all events to everyone and let them filter locally. Its easy to believe this is adequate when doing pre-determined test scenarios but we know in reality that in a busy FS box we, cant send *all* the events to *everyone* and let them filter locally.
On Sun, Feb 17, 2013 at 11:32 AM, Avi Marcus <avi@avimarcus.net> wrote:
Please excuse my ignorance, but can you explain again why zmq won't work?
Regarding boasting about messaging capability.. it was made to be quite lightweight - supposedly it can easily saturate a 1gigabit link with message size over 150 bytes each: http://www.zeromq.org/results:10gbe-tests
If you want to use it for a queue, then it's lack of persistence and assurance that messages get through might not be helpful, but for a broadcast to whoever wants to listen it sounds great for heartbeats, events, etc.
-Avi Marcus
1-718-989-9485 (USA) 1-866-202-5850 (USA & Canada Toll Free) 02-372-1570 (Israel) 020-3298-2875 (UK)
On Sun, Feb 17, 2013 at 5:50 PM, Eliot Gable <egable+freeswitch@gmail.com
wrote:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
There are some drawbacks to it, namely:
1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly.
2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible.
3) They boast about how it can handle up to 8,000 1KB messages per second. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some.
Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin.
There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster.
There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org
wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH™ Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
------------------------------ *From:* Eliot Gable [mailto:egable+freeswitch@gmail.com] *To:* FreeSWITCH Users Help [mailto: freeswitch-users@lists.freeswitch.org] *Sent:* Tue, 12 Feb 2013 05:49:13 -0800 *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable
"We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower
"I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower
"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Tony, ZMQ is fixing the Pub/Sub part filtering on version 3 which I think is out now. Sent from my iPhone On Feb 17, 2013, at 2:51 PM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
It has a strict philosophy on not using locks or mutexes and only doing com over socket. When you create an instance of it, it uses its own threads internally and that model gets messed up severely if you call fork from the process (even when you are doing it for something else like system command). Its basically an engine waiting for a host program rather than something you can integrated into a more complex application. Also the multicast has issues, there is no way to specificy host local scope for hops. Finally, their philosophy is to ignore the dilemma of sub/pub. There are no subscriptions to events, there is only send all events to everyone and let them filter locally. Its easy to believe this is adequate when doing pre-determined test scenarios but we know in reality that in a busy FS box we, cant send *all* the events to *everyone* and let them filter locally.
On Sun, Feb 17, 2013 at 11:32 AM, Avi Marcus <avi@avimarcus.net> wrote:
Please excuse my ignorance, but can you explain again why zmq won't work?
Regarding boasting about messaging capability.. it was made to be quite lightweight - supposedly it can easily saturate a 1gigabit link with message size over 150 bytes each: http://www.zeromq.org/results:10gbe-tests
If you want to use it for a queue, then it's lack of persistence and assurance that messages get through might not be helpful, but for a broadcast to whoever wants to listen it sounds great for heartbeats, events, etc.
-Avi Marcus
1-718-989-9485 (USA) 1-866-202-5850 (USA & Canada Toll Free) 02-372-1570 (Israel) 020-3298-2875 (UK)
On Sun, Feb 17, 2013 at 5:50 PM, Eliot Gable <egable+freeswitch@gmail.com> wrote:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
There are some drawbacks to it, namely:
1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly.
2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible.
3) They boast about how it can handle up to 8,000 1KB messages per second. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some.
Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin.
There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster.
There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org> wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH™ Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
From: Eliot Gable [mailto:egable+freeswitch@gmail.com] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Tue, 12 Feb 2013 05:49:13 -0800 Subject: [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable
"We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower
"I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower
"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Sun, Feb 17, 2013 at 7:59 PM, João Mesquita <jmesquita@freeswitch.org>wrote:
Tony, ZMQ is fixing the Pub/Sub part filtering on version 3 which I think is out now.
For those who want to read up on version 3.x of zmq: http://www.zeromq.org/topics:planning#toc1 -- Michael S Collins Twitter: @mercutioviz http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org
2013/2/17 Eliot Gable <egable+freeswitch@gmail.com>:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
My understanding of ZMQ and Spread is that they solve very different problems and they operate on different levels. ZMQ is on much lower levels and does not
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
I did use Spread in a big project, comprising of 40+ servers for cluster membership. It worked very well most of the time, but failed without proper understanding on our side. It took some config files voo-doo (tuning some timeout knobs) to get it running reliably.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
ZooKeeper? Something based on Paxos: http://libpaxos.sourceforge.net/ ? http://openreplica.org/ ? Never used any of them though. -- Marcin Gozdalik
Sent previous mail too fast :/ 2013/2/17 Marcin Gozdalik <gozdal@gmail.com>:
2013/2/17 Eliot Gable <egable+freeswitch@gmail.com>:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
My understanding of ZMQ and Spread is that they solve very different problems and they operate on different levels. ZMQ is on much lower levels and does not...
...guarantee anything like Spread: "Spread services range from reliable message passing to fully ordered messages with delivery guarantees, even in case of computer failures and network partitions". ZMQ is basically just a socket, but a bit easier to use.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
I did use Spread in a big project, comprising of 40+ servers for cluster membership. It worked very well most of the time, but failed without proper understanding on our side. It took some config files voo-doo (tuning some timeout knobs) to get it running reliably.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
ZooKeeper? Something based on Paxos: http://libpaxos.sourceforge.net/ ? http://openreplica.org/ ?
Never used any of them though.
-- Marcin Gozdalik
This is just my two cents but could we not use snmp as a basis for something like this since it would allow monitoring eternally using ones own systems, or allow each FreeSWITCH server to monitor the other servers over snmp. At least I think it could be an option to explore anyways. On Feb 17, 2013 3:12 PM, "Marcin Gozdalik" <gozdal@gmail.com> wrote:
Sent previous mail too fast :/
2013/2/17 Marcin Gozdalik <gozdal@gmail.com>:
2013/2/17 Eliot Gable <egable+freeswitch@gmail.com>:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
My understanding of ZMQ and Spread is that they solve very different problems and they operate on different levels. ZMQ is on much lower levels and does not...
...guarantee anything like Spread: "Spread services range from reliable message passing to fully ordered messages with delivery guarantees, even in case of computer failures and network partitions". ZMQ is basically just a socket, but a bit easier to use.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
I did use Spread in a big project, comprising of 40+ servers for cluster membership. It worked very well most of the time, but failed without proper understanding on our side. It took some config files voo-doo (tuning some timeout knobs) to get it running reliably.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
ZooKeeper? Something based on Paxos: http://libpaxos.sourceforge.net/ ? http://openreplica.org/ ?
Never used any of them though.
-- Marcin Gozdalik
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Sun, Feb 17, 2013 at 4:50 PM, Roger Castaldo <roger.castaldo@gmail.com>wrote:
This is just my two cents but could we not use snmp as a basis for something like this since it would allow monitoring eternally using ones own systems, or allow each FreeSWITCH server to monitor the other servers over snmp. At least I think it could be an option to explore anyways.
SNMP is not intended as an HA solution. It is slow, and unwieldy to use. Fail over time using SNMP would be measured in minutes. We are going for sub-second fail over. SNMP is really only intended for external monitoring. SNMP also does not provide a solution for the vast majority of issues which an HA solution must cover.
It may be worth mentioning there's a mod_snmp module for external monitoring. While that's also possible via ESL, XML RPC etc where it's better tested, SNMP at least allows integration with tools like Nagios. But as Eliot says, SNMP is slow and unwieldy and requires high timeouts. It's not something you're going to be able to use for failover. -Steve On 17 February 2013 22:51, Eliot Gable <egable+freeswitch@gmail.com> wrote:
On Sun, Feb 17, 2013 at 4:50 PM, Roger Castaldo <roger.castaldo@gmail.com>wrote:
This is just my two cents but could we not use snmp as a basis for something like this since it would allow monitoring eternally usiNng ones own systems, or allow each FreeSWITCH server to monitor the other servers over snmp. At least I think it could be an option to explore anyways.
SNMP is not intended as an HA solution. It is slow, and unwieldy to use. Fail over time using SNMP would be measured in minutes. We are going for sub-second fail over. SNMP is really only intended for external monitoring. SNMP also does not provide a solution for the vast majority of issues which an HA solution must cover.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (10)
-
Anthony Minessale
-
Avi Marcus
-
Dave R. Kompel
-
Eliot Gable
-
João Mesquita
-
Ken Rice
-
Marcin Gozdalik
-
Michael Collins
-
Roger Castaldo
-
Steven Ayre