[Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value Store

Sun Feb 17 18:50:58 MSK 2013

ZMQ also does not work with a fork, which is needed in order to execute any
system commands (like iptables or anything else which has no programming
API). That pretty much eliminates ZMQ as a possibility.

I did some research this weekend, and of all the possibilities I could
find, the one that caught my attention the most was Spread:
http://www.spread.org/

There are some drawbacks to it, namely:

1) It requires any marketing material mentioning FreeSWITCH or any project
/ solution utilizing FreeSWITCH to also include a prepared statement about
the use of the Spread toolkit. This is a fairly major licensing issue, as
all commercial solutions utilizing FreeSWITCH as a core component would
then also need to mention the use of the Spread toolkit within FreeSWITCH.
It is essentially a "viral clause," which would be very tedious for people
to make sure they honor correctly.

2) The toolkit uses a stand-alone daemon which would then need to be
monitored separately from FreeSWITCH and would be another point of failure,
adding complexity to the system. Basically, if that daemon were to crash,
FreeSWITCH would need to know about it and would need to either respawn it
or shut down. This would bring the need for Pacemaker or something similar
back into the picture for any viable HA solution. Alternatively, we could
write some code similar to daemontools into FreeSWTICH which respawns the
daemon if it dies, but we would have to test the impact of this respawning
on the overall cluster to determine if it impacts visibility of the node at
any point in time. Another alternative would be to wrap the daemon into a
thread inside FreeSWITCH such that if the daemon caused a segfault or
something, it would force FreeSWITCH to terminate, as well. I have not done
a code review of the daemon yet to determine if this is viable alternative,
but assuming they have not coded it on LSD or something, it is more than
likely possible.

3) They boast about how it can handle up to 8,000 1KB messages per second.
I don't consider that boast-worthy. When I worked at Broadvox a few years
ago, I had a FS pair which ran around 380 calls per second (760 sessions
per second). Each call generates dozens of events. That hardware was
getting dated when I left Broadvox, and today's hardware along with the
performance improvements done to FS since then means we could conceivably
have a single node which runs over 1k calls per second firing dozens of
events per call. That means a single box could completely consume the
message bandwidth of the entire Spread network. Imagine trying to have 64
such boxes running. We are really in need of a solution which boasts
hundreds of thousands of messages per second. Spread seems like it might be
off by an order of magnitude and then some.

Despite these issues, Spread still seems to come closer to our needs than
any other solution I found. FYI, I also looked at the Corosync IPC system,
and was not at all impressed. On paper, Spread exceeds Corosync's
capabilities by a fair margin.

There are some strategies for mitigating issue #3 with Spread, as well. For
example, we could limit messages across the Spread network to things like
heartbeats and / or other HA and synchronization related messages.
Basically, think of it like a D-channel on a PRI. For sending high packet
per second streams of messages, we could do standard unicast connections or
even try straight up mutlicasting to all nodes on the LAN. Sending
heartbeats every 10ms across the Spread network would put a 64-node cluster
at 6,400 messages per second just with heartbeats. That would still leave a
decent amount of message bandwidth available for other types of negotiation
messages and should still allow for sub-second fail-over detection and
reaction. Of course, this is all assuming we can actually get 8,000 1KB
messages per second out of a 64-node cluster.

There are likely lots of things that impact how many messages per second
Spread can handle. A lot of it has to do with network latency and CPU
power. Spread uses acknowledgements and message reordering to ensure
delivery in a way that accounts for things like node membership changes
during the time the message is in transit and whether the message has been
received by all nodes in the cluster. Network latency is probably one of
the biggest factors in how many messages per second it can handle. On a
faster network link, the messages per second would be higher and on a
slower network, it would be lower. Obviously, CPU processing time and
scheduling is important, as well. If one system is extremely overloaded and
the Spread daemon is being starved for CPU resources, that will add extra
latency in processing and also reduce message throughput. Obviously, this
could also impact whether the node is seen as visible. So, this is one more
argument for why we would need to try to run the daemon as a thread under
FreeSWITCH. Then it has the same scheduling priority as FreeSWITCH and
cannot be starved for resources by FreeSWITCH itself. It also would exhibit
the same amount of resource starvation FreeSWITCH experiences on the node,
so would more accurately reflect the state of FreeSWITCH on the node.

If anyone has any other suggestions than Spread, I would like to hear it.
Also, some feedback on item #1 would be great, as I cannot really judge for
everyone else how willing they are to accept such a licensing clause.

On Tue, Feb 12, 2013 at 8:48 PM, João Mesquita <jmesquita at freeswitch.org>wrote:

> I have used ZeroMQ in the past for this sorts of things but it really
> won't be able to detect failures really fast. It is not made for this.
> Maybe we can gather the requirements for such message bus? Zmq for example
> provides you with this cool interface to build messaging protocols on top
> of it but it does not provide reliability when it comes to endpoint to
> endpoint connection without a heartbeat implemented on the user end. Can
> this be used for FS as well? Anyhow, just throwing some ideas...
>
> João Mesquita
> FreeSWITCH™ Solutions
>
>
> On Tue, Feb 12, 2013 at 5:42 PM, Dave R. Kompel <drk at drkngs.net> wrote:
>
>> **
>> I've done a few experments with using both Redis, and the evil "Microsoft
>> Azure Service bus" (the server on prem based version) to extend the
>> eventing system to have global PUB/SUB. This way things like registrations,
>> and Limit stuff could be made global.
>>
>> I'm looking for a way, in my carrier switch implmentation, to implment
>> both HA Failover and Scaleout clustering.
>>
>> --Dave
>>
>>  ------------------------------
>> *From:* Eliot Gable [mailto:egable+freeswitch at gmail.com]
>> *To:* FreeSWITCH Users Help [mailto:freeswitch-users at lists.freeswitch.org
>> ]
>> *Sent:* Tue, 12 Feb 2013 05:49:13 -0800
>> *Subject:* [Freeswitch-users] FreeSWITCH Message Bus / Shared Key Value
>> Store
>>
>>
>> Tony and Mike and I had a discussion last night about FreeSWITCH with
>> regards to implementing some form of core message bus or shared key-value
>> store. We discussed a few different options, but did not really settle on
>> anything. If you are writing modules or using FreeSWITCH in a multi-node
>> setting, please share what features / functionality you would like to see
>> implemented in this regard, how you would use it, and why you want to see
>> the specific mechanism of your choice rather than some alternative. Also,
>> please consider and mention whether "cluster awareness" is something that
>> factors into your use case. By this, I mean having each FS node have some
>> idea about the state / status of each other node in terms of taking calls
>> vs acting as a standby or slave node, etc.
>>
>> --
>> Eliot Gable
>>
>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>

-- 
Eliot Gable

"We do not inherit the Earth from our ancestors: we borrow it from our
children." ~David Brower

"I decided the words were too conservative for me. We're not borrowing from
our children, we're stealing from them--and it's not even considered to be
a crime." ~David Brower

"Esse oportet ut vivas, non vivere ut edas." (Thou shouldst eat to live;
not live to eat.) ~Marcus Tullius Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130217/38d99963/attachment-0001.html 