[Freeswitch-users] mod_gsmopen - deadlock tty

Luiz Fernando Softov fernando at softov.com.br
Sun Sep 27 06:53:35 MSD 2015


Hi, i sent a e-mail to you, with the diff, files and give you access to web
interface of the system.

I will wait you to read, confirm and reply my e-mail.

Thanks a lot for the help.

2015-09-26 15:28 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:

> As per chat, I'm "gmaruzz" in both FreeSWITCH's hipchat and in IRC
> #freeswitch channel at irc.freenode.net
>
> -giovanni
>
> On Sat, Sep 26, 2015 at 9:25 PM, Giovanni Maruzzelli <gmaruzz at gmail.com>
> wrote:
>
>> On Fri, Sep 25, 2015 at 11:30 PM, Luiz Fernando Softov <
>> fernando at softov.com.br> wrote:
>>
>>> Now, i make some changes, and until now it's running fine
>>>
>>> I think the problem was when you declare interface as dead, like line
>>> 674 of gsmopen_protocol.cpp
>>>
>>>
>>> https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/src/mod/endpoints/mod_gsmopen/gsmopen_protocol.cpp
>>>
>>> alarm_event(tech_pvt, ALARM_FAILED_INTERFACE, "gsmopen_serial_monitor
>>> failed, declaring interface dead");
>>> tech_pvt->active = 0;
>>> ->>>> tech_pvt->name[0] = '\0';
>>>
>>> You make a \0 in interface name
>>>
>>> So, when someone call remove_interface() in mod_gsmopen.cpp
>>>
>>>
>>> https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/src/mod/endpoints/mod_gsmopen/mod_gsmopen.cpp
>>>
>>> To grab the tech_pvt, the function check for strlen or compare
>>> GSMOPEN_INTERFACES[interface_id].name (line 322 and 329)
>>>
>>> Then, we will not found interface (tech_pvt), because "name" is nothing
>>>
>>>
>>>
>> continuing, in line 336 it's check if there is a tech_pvt
>>>
>>> if (!tech_pvt) {
>>> DEBUGA_GSMOPEN("interface '%s' does not exist\n", GSMOPEN_P_LOG,
>>> the_interface);
>>>   goto end;
>>> }
>>>
>>> This check will jump the Close in line 385, causing a deadlock opened fd.
>>> res = tech_pvt->serialPort_serial_control->Close();
>>>
>>> So i comment every line when you set tech_pvt->name[0] = '\0';
>>>
>>> It's correct? Since when interface failed, and it's declared as died,
>>> the serial isn't closed, just marked as died, and the tty still opened
>>> causing a deadlock, if no one close this.
>>>
>>>
>> your reasoning seems sound, is the system still running ok?
>>
>>
>> Have you changed anything else?
>>
>> Can you send me (to my mail, as an attach) a git diff? Or open a jira
>> attaching that diff?
>>
>> Btw, yes, I'm interested in understanding better how your system works
>> (daemon, attach-detach, and all).
>>
>> Also, big thanks for taking the time to look into the source code!!!
>>
>> -giovanni
>>
>>
>>
>>
>>
>>>
>>>
>>>
>>>
>>> 2015-09-25 14:10 GMT-04:00 Luiz Fernando Softov <fernando at softov.com.br>
>>> :
>>>
>>>> I make a daemon, witch manage USB using kqueue and other things, so,
>>>> every time i plug or unplug a modem, i make a event, like ARRIVE and LEAVE.
>>>>
>>>> This daemon have a list of modems, correlate with ttys ids - U3G (U0.0,
>>>> U0.1, U0.2) and UMODEM (U0 U1 U2)
>>>>
>>>> So, when a modem ARRIVE, and it's "attached" (configured), i call a
>>>> "gsm reload".
>>>> When a modem LEAVE, i call a gsm remove interface_x.
>>>>
>>>> Using mod_xml_curl, i have dynamic config.
>>>>
>>>> I use events, like gsmopen_dump list, to manage and controll
>>>> interfaces, like when user "detach" a modem.
>>>>
>>>> If you want, send me a e-mail, i will give access to my system and you
>>>> can see what i'm talking about.
>>>>
>>>>
>>>> Anyway, i found things like in function
>>>>
>>>> int gsmopen_serial_write_AT_noack(private_t *tech_pvt, const char *data)
>>>>
>>>> // You make
>>>> LOKKA(tech_pvt->controldev_lock);
>>>>
>>>> Then call gsmopen_serial_write_AT(tech_pvt, data), and
>>>> inside gsmopen_serial_write_AT, when something is wrong, you
>>>> make UNLOCKA(tech_pvt->controldev_lock); and return -1
>>>>
>>>> But, you check if (gsmopen_serial_write_AT(tech_pvt, data) != (int)
>>>> strlen(data)) {
>>>>
>>>> Then, you make again UNLOCKA(tech_pvt->controldev_lock);
>>>>
>>>> Why make two times UNLOCKA(tech_pvt->controldev_lock); ?
>>>>
>>>> It's like
>>>> LOKKA(tech_pvt->controldev_lock);
>>>> UNLOCKA(tech_pvt->controldev_lock);
>>>> UNLOCKA(tech_pvt->controldev_lock);
>>>>
>>>>
>>>> Another is in
>>>> int gsmopen_serial_read_AT(private_t *tech_pvt, int look_for_ack, int
>>>> timeout_usec, int timeout_sec, const char *expected_string, int expect_crlf)
>>>> You make a LOKKA(tech_pvt->controldev_lock);, but there is some returns
>>>> (inside try catch), without UNLOCKA.
>>>>
>>>> Sorry if i make some mistakes in my explanation, but i just want to
>>>> undestand how this funcions works.
>>>>
>>>> I disagree with "Shaabir" about this module is not useable. I just
>>>> think this need some changes to work fine.
>>>> I already test this module with 40 modems for 7 days, and it's work
>>>> fine, until i grab a deadlock in USB.
>>>>
>>>> Giovanni, if you have time to chat with me, i will be grateful.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2015-09-25 12:41 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>>>
>>>>> for plugging and unplugging modems, that's a very difficult thing to
>>>>> manage, this is the best I can come out with (and probably does not work in
>>>>> freebsd):
>>>>>
>>>>>
>>>>> http://wiki.freeswitch.org/wiki/GSMopen#IMEI_and_IMSI_automatic_device_discovery_.28only_on_Linux.29_2
>>>>>
>>>>> you're welcome to find a similar mechanism, and file a jira with a
>>>>> pull request, so it will be inserted in manline.
>>>>>
>>>>> If your modifications only apply to freebsd, please use #ifdefs, so it
>>>>> still continue to compile and work in Linux.
>>>>>
>>>>> -giovanni
>>>>>
>>>>>
>>>>> Sincerely,
>>>>>
>>>>> Giovanni Maruzzelli
>>>>> Cell : +39-347-2665618
>>>>>
>>>>> On Fri, Sep 25, 2015 at 6:24 PM, Luiz Fernando Softov <
>>>>> fernando at softov.com.br> wrote:
>>>>>
>>>>>> I'm using a USB with power supply.
>>>>>>
>>>>>> But sometimes it's just happen. Sometimes user remove and plug modem
>>>>>> again.
>>>>>>
>>>>>> I can't use Debian, because our system depends of FreeBSD, and we
>>>>>> have a lot of source (daemons) running, based, on that platform. And we use
>>>>>> kqueue, I think Debian don't have kqueue, it's like epool, but better, much
>>>>>> better.
>>>>>>
>>>>>> Anyway, I don't That it's a kernel problem, because, since I have
>>>>>> another daemon, attached in ttyx.0 (mod_gsmopen attach to ttyx.3), and when
>>>>>> modem power off or "reset", only ttyx.3 still locked.
>>>>>>
>>>>>> I'm trying to read the source to figure what's happening, but it's
>>>>>> hard to read, not because the source is difficult, but because isn't
>>>>>> organized.
>>>>>>
>>>>>> I'm thinking in rewrite the module, but if in future someone change
>>>>>> something in original module I will need to check my source.
>>>>>> There is another thing, I don't know how things like apr_thread
>>>>>> works, and what functions like switch_sleep(1000000) do and why using that.
>>>>>>
>>>>>> Today I make some alterations where the module make a lock, LOKKA,
>>>>>> there is some functions who don't have unlock, and another who only have
>>>>>> unlock, sometimes two unlocks. So I will wait to see what will happen.
>>>>>>
>>>>>> Now, here it's 3:00 am, and I'm going to sleep, almost 19 hours awake
>>>>>> up, trying to debug this module and another things.
>>>>>>
>>>>>>
>>>>>> Maybe you or someone have a time, to chat, or reply some e-mails, or
>>>>>> maybe here, to explain some questions about the source (module)?
>>>>>>
>>>>>> There's some lines marked with TODO, why?
>>>>>> And there is another version of this module?
>>>>>> Which the maximum number of modems that you already usage?
>>>>>>
>>>>>> 2015-09-25 7:53 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>>>>>
>>>>>>> if you found problems, please open a jira issue, with all info to
>>>>>>> replicate the problem, as explained in gsmopen page.
>>>>>>>
>>>>>>> It would be useful for us developers, and to me personally, being
>>>>>>> the person in charge of gsmopen.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>> -giovanni
>>>>>>>
>>>>>>> Sincerely,
>>>>>>>
>>>>>>> Giovanni Maruzzelli
>>>>>>> Cell : +39-347-2665618
>>>>>>>
>>>>>>> On Fri, Sep 25, 2015 at 1:44 PM, Shabbir abbasi <
>>>>>>> shabbirabbasi92 at gmail.com> wrote:
>>>>>>>
>>>>>>>> mod_gsmopen  needs more development  work, i have tested it
>>>>>>>> sometimes ago and found it is not useable , so now i am using  asterisk
>>>>>>>> with chan_dongle better then gsmopen without usb HUB
>>>>>>>>
>>>>>>>> On Fri, Sep 25, 2015 at 11:11 AM, Giovanni Maruzzelli <
>>>>>>>> gmaruzz at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> This seems to be a kernel problem.
>>>>>>>>>
>>>>>>>>> To have the most success chances: use a USB hub with power supply
>>>>>>>>> (eg, a USB hub that get electrical power from the wall socket).
>>>>>>>>> Also, use the reference.platform, debian 8 jessie, instead of
>>>>>>>>> freebsd.
>>>>>>>>>
>>>>>>>>> -goovanni
>>>>>>>>>
>>>>>>>>> sent from my mobile,
>>>>>>>>> Giovanni Maruzzelli
>>>>>>>>> cell: +39 347 266 56 18
>>>>>>>>> On Sep 25, 2015 7:19 AM, "Luiz Fernando Softov" <
>>>>>>>>> fernando at softov.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm using freeswitch 1.15 in Freebsd 10.1, with mod_gsmopen
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH Version 1.5.15b+git~20150407T231134Z~d55c4a053b~64bit
>>>>>>>>>> (git d55c4a0 2015-04-07 23:11:34Z 64bit), whats happen, its the same with
>>>>>>>>>> another versions like, 1.6, 1.4.19, etc
>>>>>>>>>>
>>>>>>>>>> After sometime running freeswitch, i have this TTYs.
>>>>>>>>>>
>>>>>>>>>> crw-------   1 root  wheel      0x8e Sep 24 15:30 ttyU1.3
>>>>>>>>>> crw-------   1 root  wheel      0x8f Sep 24 15:30 ttyU1.3.init
>>>>>>>>>> crw-------   1 root  wheel      0x90 Sep 24 15:30 ttyU1.3.lock
>>>>>>>>>> crw-------   1 root  wheel      0x8e Sep 24 15:30 ttyU2.3
>>>>>>>>>> crw-------   1 root  wheel      0x8f Sep 24 15:30 ttyU2.3.init
>>>>>>>>>> crw-------   1 root  wheel      0x90 Sep 24 15:30 ttyU2.3.lock
>>>>>>>>>> crw-------   1 root  wheel      0x8e Sep 24 15:30 ttyU3.3
>>>>>>>>>> crw-------   1 root  wheel      0x8f Sep 24 15:30 ttyU3.3.init
>>>>>>>>>> crw-------   1 root  wheel      0x90 Sep 24 15:30 ttyU3.3.lock
>>>>>>>>>> crw-------   1 root  wheel      0x76 Sep 25 00:46 ttyU4.0
>>>>>>>>>> crw-------   1 root  wheel      0x77 Sep 24 22:28 ttyU4.0.init
>>>>>>>>>> crw-------   1 root  wheel      0x78 Sep 24 22:28 ttyU4.0.lock
>>>>>>>>>> crw-------   1 root  wheel      0x7c Sep 24 22:28 ttyU4.1
>>>>>>>>>> crw-------   1 root  wheel      0x7d Sep 24 22:28 ttyU4.1.init
>>>>>>>>>> crw-------   1 root  wheel      0x7e Sep 24 22:28 ttyU4.1.lock
>>>>>>>>>> crw-------   1 root  wheel      0x82 Sep 24 22:28 ttyU4.2
>>>>>>>>>> crw-------   1 root  wheel      0x83 Sep 24 22:28 ttyU4.2.init
>>>>>>>>>> crw-------   1 root  wheel      0x84 Sep 24 22:28 ttyU4.2.lock
>>>>>>>>>> crw-------   1 root  wheel      0x8e Sep 24 22:28 ttyU4.3
>>>>>>>>>> crw-------   1 root  wheel      0x8f Sep 24 22:28 ttyU4.3.init
>>>>>>>>>> crw-------   1 root  wheel      0x90 Sep 24 22:28 ttyU4.3.lock
>>>>>>>>>> crw-------   1 root  wheel      0xa8 Sep 25 00:46 ttyU5.0
>>>>>>>>>> crw-------   1 root  wheel      0xa9 Sep 24 22:28 ttyU5.0.init
>>>>>>>>>> crw-------   1 root  wheel      0xaa Sep 24 22:28 ttyU5.0.lock
>>>>>>>>>> crw-------   1 root  wheel      0xae Sep 24 22:28 ttyU5.1
>>>>>>>>>> crw-------   1 root  wheel      0xaf Sep 24 22:28 ttyU5.1.init
>>>>>>>>>> crw-------   1 root  wheel      0xb6 Sep 24 22:28 ttyU5.1.lock
>>>>>>>>>> crw-------   1 root  wheel      0xba Sep 24 22:28 ttyU5.2
>>>>>>>>>> crw-------   1 root  wheel      0xbb Sep 24 22:28 ttyU5.2.init
>>>>>>>>>> crw-------   1 root  wheel      0xbc Sep 24 22:28 ttyU5.2.lock
>>>>>>>>>> crw-------   1 root  wheel      0xc0 Sep 24 22:28 ttyU5.3
>>>>>>>>>> crw-------   1 root  wheel      0xc1 Sep 24 22:28 ttyU5.3.init
>>>>>>>>>> crw-------   1 root  wheel      0xc2 Sep 24 22:28 ttyU5.3.lock
>>>>>>>>>>
>>>>>>>>>> As you can see, ttyU1.3, ttyU2.3, ttyU3.3 are USB deadlocks
>>>>>>>>>>
>>>>>>>>>> I try to unload mod_gsmopen, but i receive a message that is in
>>>>>>>>>> use
>>>>>>>>>> so i kill freeswitch, then ttyU1.3, ttyU2.3, ttyU3.3, are free
>>>>>>>>>> and disappear (as well .init and .lock)
>>>>>>>>>>
>>>>>>>>>> I try to figure out whats happening, with logs, alarms, but i
>>>>>>>>>> can't.
>>>>>>>>>> Someone have the same problem, or know the solution?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ps.
>>>>>>>>>> Reading the list, i realize that few members use the module, or
>>>>>>>>>> don't have problems.
>>>>>>>>>> If i am correct, is there any other way to use GSM modems or GSM,
>>>>>>>>>> which is low cost?
>>>>>>>>>> Is there anyone who can guide me in some things about the code or
>>>>>>>>>> tips?
>>>>>>>>>>
>>>>>>>>>> ps 2.
>>>>>>>>>> sorry about my english
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Luiz Fernando Softov
>>>>>>>>>> http://www.softov.com.br
>>>>>>>>>> fernando at softov.com.br
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://confluence.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://confluence.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Luiz Fernando Softov
>>>>>> http://www.softov.com.br
>>>>>> fernando at softov.com.br
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://confluence.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Luiz Fernando Softov
>>>> http://www.softov.com.br
>>>> fernando at softov.com.br
>>>>
>>>
>>>
>>>
>>> --
>>> Luiz Fernando Softov
>>> http://www.softov.com.br
>>> fernando at softov.com.br
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>>
>> --
>> Sincerely,
>>
>> Giovanni Maruzzelli
>> Cell : +39-347-2665618
>>
>
>
>
> --
> Sincerely,
>
> Giovanni Maruzzelli
> Cell : +39-347-2665618
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>



-- 
Luiz Fernando Softov
http://www.softov.com.br
fernando at softov.com.br
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150926/d4f27894/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list