Regarding #3, it seems they got to it for zeromq v3.
From ØMQ 3.x, filtering happens at the publisher side, when using a connected protocol (tcp: *or ipc:*). Using the epgm:// protocol, filtering happens at the subscriber side.
In ØMQ/2.x, all filtering happened at the subscriber side. -Avi On Sun, Feb 17, 2013 at 7:51 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It has a strict philosophy on not using locks or mutexes and only doing com over socket. When you create an instance of it, it uses its own threads internally and that model gets messed up severely if you call fork from the process (even when you are doing it for something else like system command). Its basically an engine waiting for a host program rather than something you can integrated into a more complex application. Also the multicast has issues, there is no way to specificy host local scope for hops. Finally, their philosophy is to ignore the dilemma of sub/pub. There are no subscriptions to events, there is only send all events to everyone and let them filter locally. Its easy to believe this is adequate when doing pre-determined test scenarios but we know in reality that in a busy FS box we, cant send *all* the events to *everyone* and let them filter locally.
On Sun, Feb 17, 2013 at 11:32 AM, Avi Marcus <avi@avimarcus.net> wrote:
Please excuse my ignorance, but can you explain again why zmq won't work?
Regarding boasting about messaging capability.. it was made to be quite lightweight - supposedly it can easily saturate a 1gigabit link with message size over 150 bytes each: http://www.zeromq.org/results:10gbe-tests
If you want to use it for a queue, then it's lack of persistence and assurance that messages get through might not be helpful, but for a broadcast to whoever wants to listen it sounds great for heartbeats, events, etc.
-Avi Marcus
1-718-989-9485 (USA) 1-866-202-5850 (USA & Canada Toll Free) 02-372-1570 (Israel) 020-3298-2875 (UK)
On Sun, Feb 17, 2013 at 5:50 PM, Eliot Gable <egable+freeswitch@gmail.com
wrote:
ZMQ also does not work with a fork, which is needed in order to execute any system commands (like iptables or anything else which has no programming API). That pretty much eliminates ZMQ as a possibility.
I did some research this weekend, and of all the possibilities I could find, the one that caught my attention the most was Spread: http://www.spread.org/
There are some drawbacks to it, namely:
1) It requires any marketing material mentioning FreeSWITCH or any project / solution utilizing FreeSWITCH to also include a prepared statement about the use of the Spread toolkit. This is a fairly major licensing issue, as all commercial solutions utilizing FreeSWITCH as a core component would then also need to mention the use of the Spread toolkit within FreeSWITCH. It is essentially a "viral clause," which would be very tedious for people to make sure they honor correctly.
2) The toolkit uses a stand-alone daemon which would then need to be monitored separately from FreeSWITCH and would be another point of failure, adding complexity to the system. Basically, if that daemon were to crash, FreeSWITCH would need to know about it and would need to either respawn it or shut down. This would bring the need for Pacemaker or something similar back into the picture for any viable HA solution. Alternatively, we could write some code similar to daemontools into FreeSWTICH which respawns the daemon if it dies, but we would have to test the impact of this respawning on the overall cluster to determine if it impacts visibility of the node at any point in time. Another alternative would be to wrap the daemon into a thread inside FreeSWITCH such that if the daemon caused a segfault or something, it would force FreeSWITCH to terminate, as well. I have not done a code review of the daemon yet to determine if this is viable alternative, but assuming they have not coded it on LSD or something, it is more than likely possible.
3) They boast about how it can handle up to 8,000 1KB messages per second. I don't consider that boast-worthy. When I worked at Broadvox a few years ago, I had a FS pair which ran around 380 calls per second (760 sessions per second). Each call generates dozens of events. That hardware was getting dated when I left Broadvox, and today's hardware along with the performance improvements done to FS since then means we could conceivably have a single node which runs over 1k calls per second firing dozens of events per call. That means a single box could completely consume the message bandwidth of the entire Spread network. Imagine trying to have 64 such boxes running. We are really in need of a solution which boasts hundreds of thousands of messages per second. Spread seems like it might be off by an order of magnitude and then some.
Despite these issues, Spread still seems to come closer to our needs than any other solution I found. FYI, I also looked at the Corosync IPC system, and was not at all impressed. On paper, Spread exceeds Corosync's capabilities by a fair margin.
There are some strategies for mitigating issue #3 with Spread, as well. For example, we could limit messages across the Spread network to things like heartbeats and / or other HA and synchronization related messages. Basically, think of it like a D-channel on a PRI. For sending high packet per second streams of messages, we could do standard unicast connections or even try straight up mutlicasting to all nodes on the LAN. Sending heartbeats every 10ms across the Spread network would put a 64-node cluster at 6,400 messages per second just with heartbeats. That would still leave a decent amount of message bandwidth available for other types of negotiation messages and should still allow for sub-second fail-over detection and reaction. Of course, this is all assuming we can actually get 8,000 1KB messages per second out of a 64-node cluster.
There are likely lots of things that impact how many messages per second Spread can handle. A lot of it has to do with network latency and CPU power. Spread uses acknowledgements and message reordering to ensure delivery in a way that accounts for things like node membership changes during the time the message is in transit and whether the message has been received by all nodes in the cluster. Network latency is probably one of the biggest factors in how many messages per second it can handle. On a faster network link, the messages per second would be higher and on a slower network, it would be lower. Obviously, CPU processing time and scheduling is important, as well. If one system is extremely overloaded and the Spread daemon is being starved for CPU resources, that will add extra latency in processing and also reduce message throughput. Obviously, this could also impact whether the node is seen as visible. So, this is one more argument for why we would need to try to run the daemon as a thread under FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and cannot be starved for resources by FreeSWITCH itself. It also would exhibit the same amount of resource starvation FreeSWITCH experiences on the node, so would more accurately reflect the state of FreeSWITCH on the node.
If anyone has any other suggestions than Spread, I would like to hear it. Also, some feedback on item #1 would be great, as I cannot really judge for everyone else how willing they are to accept such a licensing clause.
On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita@freeswitch.org
wrote:
I have used ZeroMQ in the past for this sorts of things but it really won't be able to detect failures really fast. It is not made for this. Maybe we can gather the requirements for such message bus? Zmq for example provides you with this cool interface to build messaging protocols on top of it but it does not provide reliability when it comes to endpoint to endpoint connection without a heartbeat implemented on the user end. Can this be used for FS as well? Anyhow, just throwing some ideas...
João Mesquita FreeSWITCH™ Solutions
On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** I've done a few experments with using both Redis, and the evil "Microsoft Azure Service bus" (the server on prem based version) to extend the eventing system to have global PUB/SUB. This way things like registrations, and Limit stuff could be made global.
I'm looking for a way, in my carrier switch implmentation, to implment both HA Failover and Scaleout clustering.
--Dave
------------------------------ *From:* Eliot Gable [mailto:egable+freeswitch@gmail.com] *To:* FreeSWITCH Users Help [mailto: freeswitch-users@lists.freeswitch.org] *Sent:* Tue, 12 Feb 2013 05:49:13 -0800 *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store
Tony and Mike and I had a discussion last night about FreeSWITCH with regards to implementing some form of core message bus or shared key-value store. We discussed a few different options, but did not really settle on anything. If you are writing modules or using FreeSWITCH in a multi-node setting, please share what features / functionality you would like to see implemented in this regard, how you would use it, and why you want to see the specific mechanism of your choice rather than some alternative. Also, please consider and mention whether "cluster awareness" is something that factors into your use case. By this, I mean having each FS node have some idea about the state / status of each other node in terms of taking calls vs acting as a standby or slave node, etc.
-- Eliot Gable
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Eliot Gable
"We do not inherit the Earth from our ancestors: we borrow it from our children." ~David Brower
"I decided the words were too conservative for me. We're not borrowing from our children, we're stealing from them--and it's not even considered to be a crime." ~David Brower
"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live; not live to eat.) ~Marcus Tullius Cicero
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org