[Freeswitch-users] mod_gsmopen - deadlock tty
Luiz Fernando Softov
fernando at softov.com.br
Tue Dec 15 17:37:02 MSK 2015
I have a look in this patch, you can't just do a reconnect, since the tty
port can be changed (eg: ttyU1 -> ttyU4).
I have found a lot of errors in the original source (master), like
in gsmopen_protocol.cpp, function gsmopen_serial_init
It's suppose to return the fd of serial, but just return 0 or 1,
to controldevfd
Then in mod_gsmopen.cpp, line 2651
close(tech_pvt->controldevfd);
So, sometimes it make a close of a fd 0/-1 without check this before, and
it's invalid, because the fd can be 1, 2, 3, ...
It's just one of many errors i found, now i'm running this in a lot of
clients with 4, 10 and 32 modems.
Sometimes, if the machine is poor, or if the user don't use a HUB, the
modems can reset itself, because of the power supply and i reattach the
modem
with the new info (tty).
Now, sometimes, i got channels that can't be closed.
freeswitch at internal> uuid_kill e66e1072-4fa2-e511-a6df-6c626dfc249d
-ERR No such channel!
But, i can't know if this is because of my patch, or the system, since it's
happen with user to user call too.
If you wan't, can make contact with me by e-mail, i will be happy to help
with the adjustments of this module, since i need this to work in my system.
Ps, sorry about my english.
2015-12-08 9:09 GMT-03:00 Shabbir abbasi <shabbirabbasi92 at gmail.com>:
> Hi:
> i have made patch for auto reconnect if dongle disconnected and display
> status of dongle connected or disconnected
> https://freeswitch.org/jira/browse/FS-8626
>
> i think after this patch you dont need a seprate daemon to run
> thank u
>
> On Sun, Sep 27, 2015 at 7:53 AM, Luiz Fernando Softov <
> fernando at softov.com.br> wrote:
>
>> Hi, i sent a e-mail to you, with the diff, files and give you access to
>> web interface of the system.
>>
>> I will wait you to read, confirm and reply my e-mail.
>>
>> Thanks a lot for the help.
>>
>> 2015-09-26 15:28 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>
>>> As per chat, I'm "gmaruzz" in both FreeSWITCH's hipchat and in IRC
>>> #freeswitch channel at irc.freenode.net
>>>
>>> -giovanni
>>>
>>> On Sat, Sep 26, 2015 at 9:25 PM, Giovanni Maruzzelli <gmaruzz at gmail.com>
>>> wrote:
>>>
>>>> On Fri, Sep 25, 2015 at 11:30 PM, Luiz Fernando Softov <
>>>> fernando at softov.com.br> wrote:
>>>>
>>>>> Now, i make some changes, and until now it's running fine
>>>>>
>>>>> I think the problem was when you declare interface as dead, like line
>>>>> 674 of gsmopen_protocol.cpp
>>>>>
>>>>>
>>>>> https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/src/mod/endpoints/mod_gsmopen/gsmopen_protocol.cpp
>>>>>
>>>>> alarm_event(tech_pvt, ALARM_FAILED_INTERFACE, "gsmopen_serial_monitor
>>>>> failed, declaring interface dead");
>>>>> tech_pvt->active = 0;
>>>>> ->>>> tech_pvt->name[0] = '\0';
>>>>>
>>>>> You make a \0 in interface name
>>>>>
>>>>> So, when someone call remove_interface() in mod_gsmopen.cpp
>>>>>
>>>>>
>>>>> https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/src/mod/endpoints/mod_gsmopen/mod_gsmopen.cpp
>>>>>
>>>>> To grab the tech_pvt, the function check for strlen or compare
>>>>> GSMOPEN_INTERFACES[interface_id].name (line 322 and 329)
>>>>>
>>>>> Then, we will not found interface (tech_pvt), because "name" is nothing
>>>>>
>>>>>
>>>>>
>>>> continuing, in line 336 it's check if there is a tech_pvt
>>>>>
>>>>> if (!tech_pvt) {
>>>>> DEBUGA_GSMOPEN("interface '%s' does not exist\n", GSMOPEN_P_LOG,
>>>>> the_interface);
>>>>> goto end;
>>>>> }
>>>>>
>>>>> This check will jump the Close in line 385, causing a deadlock opened
>>>>> fd.
>>>>> res = tech_pvt->serialPort_serial_control->Close();
>>>>>
>>>>> So i comment every line when you set tech_pvt->name[0] = '\0';
>>>>>
>>>>> It's correct? Since when interface failed, and it's declared as died,
>>>>> the serial isn't closed, just marked as died, and the tty still opened
>>>>> causing a deadlock, if no one close this.
>>>>>
>>>>>
>>>> your reasoning seems sound, is the system still running ok?
>>>>
>>>>
>>>> Have you changed anything else?
>>>>
>>>> Can you send me (to my mail, as an attach) a git diff? Or open a jira
>>>> attaching that diff?
>>>>
>>>> Btw, yes, I'm interested in understanding better how your system works
>>>> (daemon, attach-detach, and all).
>>>>
>>>> Also, big thanks for taking the time to look into the source code!!!
>>>>
>>>> -giovanni
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2015-09-25 14:10 GMT-04:00 Luiz Fernando Softov <
>>>>> fernando at softov.com.br>:
>>>>>
>>>>>> I make a daemon, witch manage USB using kqueue and other things, so,
>>>>>> every time i plug or unplug a modem, i make a event, like ARRIVE and LEAVE.
>>>>>>
>>>>>> This daemon have a list of modems, correlate with ttys ids - U3G
>>>>>> (U0.0, U0.1, U0.2) and UMODEM (U0 U1 U2)
>>>>>>
>>>>>> So, when a modem ARRIVE, and it's "attached" (configured), i call a
>>>>>> "gsm reload".
>>>>>> When a modem LEAVE, i call a gsm remove interface_x.
>>>>>>
>>>>>> Using mod_xml_curl, i have dynamic config.
>>>>>>
>>>>>> I use events, like gsmopen_dump list, to manage and controll
>>>>>> interfaces, like when user "detach" a modem.
>>>>>>
>>>>>> If you want, send me a e-mail, i will give access to my system and
>>>>>> you can see what i'm talking about.
>>>>>>
>>>>>>
>>>>>> Anyway, i found things like in function
>>>>>>
>>>>>> int gsmopen_serial_write_AT_noack(private_t *tech_pvt, const char
>>>>>> *data)
>>>>>>
>>>>>> // You make
>>>>>> LOKKA(tech_pvt->controldev_lock);
>>>>>>
>>>>>> Then call gsmopen_serial_write_AT(tech_pvt, data), and
>>>>>> inside gsmopen_serial_write_AT, when something is wrong, you
>>>>>> make UNLOCKA(tech_pvt->controldev_lock); and return -1
>>>>>>
>>>>>> But, you check if (gsmopen_serial_write_AT(tech_pvt, data) != (int)
>>>>>> strlen(data)) {
>>>>>>
>>>>>> Then, you make again UNLOCKA(tech_pvt->controldev_lock);
>>>>>>
>>>>>> Why make two times UNLOCKA(tech_pvt->controldev_lock); ?
>>>>>>
>>>>>> It's like
>>>>>> LOKKA(tech_pvt->controldev_lock);
>>>>>> UNLOCKA(tech_pvt->controldev_lock);
>>>>>> UNLOCKA(tech_pvt->controldev_lock);
>>>>>>
>>>>>>
>>>>>> Another is in
>>>>>> int gsmopen_serial_read_AT(private_t *tech_pvt, int look_for_ack, int
>>>>>> timeout_usec, int timeout_sec, const char *expected_string, int expect_crlf)
>>>>>> You make a LOKKA(tech_pvt->controldev_lock);, but there is some
>>>>>> returns (inside try catch), without UNLOCKA.
>>>>>>
>>>>>> Sorry if i make some mistakes in my explanation, but i just want to
>>>>>> undestand how this funcions works.
>>>>>>
>>>>>> I disagree with "Shaabir" about this module is not useable. I just
>>>>>> think this need some changes to work fine.
>>>>>> I already test this module with 40 modems for 7 days, and it's work
>>>>>> fine, until i grab a deadlock in USB.
>>>>>>
>>>>>> Giovanni, if you have time to chat with me, i will be grateful.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-09-25 12:41 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>>>>>
>>>>>>> for plugging and unplugging modems, that's a very difficult thing to
>>>>>>> manage, this is the best I can come out with (and probably does not work in
>>>>>>> freebsd):
>>>>>>>
>>>>>>>
>>>>>>> http://wiki.freeswitch.org/wiki/GSMopen#IMEI_and_IMSI_automatic_device_discovery_.28only_on_Linux.29_2
>>>>>>>
>>>>>>> you're welcome to find a similar mechanism, and file a jira with a
>>>>>>> pull request, so it will be inserted in manline.
>>>>>>>
>>>>>>> If your modifications only apply to freebsd, please use #ifdefs, so
>>>>>>> it still continue to compile and work in Linux.
>>>>>>>
>>>>>>> -giovanni
>>>>>>>
>>>>>>>
>>>>>>> Sincerely,
>>>>>>>
>>>>>>> Giovanni Maruzzelli
>>>>>>> Cell : +39-347-2665618
>>>>>>>
>>>>>>> On Fri, Sep 25, 2015 at 6:24 PM, Luiz Fernando Softov <
>>>>>>> fernando at softov.com.br> wrote:
>>>>>>>
>>>>>>>> I'm using a USB with power supply.
>>>>>>>>
>>>>>>>> But sometimes it's just happen. Sometimes user remove and plug
>>>>>>>> modem again.
>>>>>>>>
>>>>>>>> I can't use Debian, because our system depends of FreeBSD, and we
>>>>>>>> have a lot of source (daemons) running, based, on that platform. And we use
>>>>>>>> kqueue, I think Debian don't have kqueue, it's like epool, but better, much
>>>>>>>> better.
>>>>>>>>
>>>>>>>> Anyway, I don't That it's a kernel problem, because, since I have
>>>>>>>> another daemon, attached in ttyx.0 (mod_gsmopen attach to ttyx.3), and when
>>>>>>>> modem power off or "reset", only ttyx.3 still locked.
>>>>>>>>
>>>>>>>> I'm trying to read the source to figure what's happening, but it's
>>>>>>>> hard to read, not because the source is difficult, but because isn't
>>>>>>>> organized.
>>>>>>>>
>>>>>>>> I'm thinking in rewrite the module, but if in future someone change
>>>>>>>> something in original module I will need to check my source.
>>>>>>>> There is another thing, I don't know how things like apr_thread
>>>>>>>> works, and what functions like switch_sleep(1000000) do and why using that.
>>>>>>>>
>>>>>>>> Today I make some alterations where the module make a lock, LOKKA,
>>>>>>>> there is some functions who don't have unlock, and another who only have
>>>>>>>> unlock, sometimes two unlocks. So I will wait to see what will happen.
>>>>>>>>
>>>>>>>> Now, here it's 3:00 am, and I'm going to sleep, almost 19 hours
>>>>>>>> awake up, trying to debug this module and another things.
>>>>>>>>
>>>>>>>>
>>>>>>>> Maybe you or someone have a time, to chat, or reply some e-mails,
>>>>>>>> or maybe here, to explain some questions about the source (module)?
>>>>>>>>
>>>>>>>> There's some lines marked with TODO, why?
>>>>>>>> And there is another version of this module?
>>>>>>>> Which the maximum number of modems that you already usage?
>>>>>>>>
>>>>>>>> 2015-09-25 7:53 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>>>>>>>
>>>>>>>>> if you found problems, please open a jira issue, with all info to
>>>>>>>>> replicate the problem, as explained in gsmopen page.
>>>>>>>>>
>>>>>>>>> It would be useful for us developers, and to me personally, being
>>>>>>>>> the person in charge of gsmopen.
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>>
>>>>>>>>> -giovanni
>>>>>>>>>
>>>>>>>>> Sincerely,
>>>>>>>>>
>>>>>>>>> Giovanni Maruzzelli
>>>>>>>>> Cell : +39-347-2665618
>>>>>>>>>
>>>>>>>>> On Fri, Sep 25, 2015 at 1:44 PM, Shabbir abbasi <
>>>>>>>>> shabbirabbasi92 at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> mod_gsmopen needs more development work, i have tested it
>>>>>>>>>> sometimes ago and found it is not useable , so now i am using asterisk
>>>>>>>>>> with chan_dongle better then gsmopen without usb HUB
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 25, 2015 at 11:11 AM, Giovanni Maruzzelli <
>>>>>>>>>> gmaruzz at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> This seems to be a kernel problem.
>>>>>>>>>>>
>>>>>>>>>>> To have the most success chances: use a USB hub with power
>>>>>>>>>>> supply (eg, a USB hub that get electrical power from the wall socket).
>>>>>>>>>>> Also, use the reference.platform, debian 8 jessie, instead of
>>>>>>>>>>> freebsd.
>>>>>>>>>>>
>>>>>>>>>>> -goovanni
>>>>>>>>>>>
>>>>>>>>>>> sent from my mobile,
>>>>>>>>>>> Giovanni Maruzzelli
>>>>>>>>>>> cell: +39 347 266 56 18
>>>>>>>>>>> On Sep 25, 2015 7:19 AM, "Luiz Fernando Softov" <
>>>>>>>>>>> fernando at softov.com.br> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm using freeswitch 1.15 in Freebsd 10.1, with mod_gsmopen
>>>>>>>>>>>>
>>>>>>>>>>>> FreeSWITCH Version
>>>>>>>>>>>> 1.5.15b+git~20150407T231134Z~d55c4a053b~64bit (git d55c4a0 2015-04-07
>>>>>>>>>>>> 23:11:34Z 64bit), whats happen, its the same with another versions like,
>>>>>>>>>>>> 1.6, 1.4.19, etc
>>>>>>>>>>>>
>>>>>>>>>>>> After sometime running freeswitch, i have this TTYs.
>>>>>>>>>>>>
>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 15:30 ttyU1.3
>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 15:30 ttyU1.3.init
>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 15:30 ttyU1.3.lock
>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 15:30 ttyU2.3
>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 15:30 ttyU2.3.init
>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 15:30 ttyU2.3.lock
>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 15:30 ttyU3.3
>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 15:30 ttyU3.3.init
>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 15:30 ttyU3.3.lock
>>>>>>>>>>>> crw------- 1 root wheel 0x76 Sep 25 00:46 ttyU4.0
>>>>>>>>>>>> crw------- 1 root wheel 0x77 Sep 24 22:28 ttyU4.0.init
>>>>>>>>>>>> crw------- 1 root wheel 0x78 Sep 24 22:28 ttyU4.0.lock
>>>>>>>>>>>> crw------- 1 root wheel 0x7c Sep 24 22:28 ttyU4.1
>>>>>>>>>>>> crw------- 1 root wheel 0x7d Sep 24 22:28 ttyU4.1.init
>>>>>>>>>>>> crw------- 1 root wheel 0x7e Sep 24 22:28 ttyU4.1.lock
>>>>>>>>>>>> crw------- 1 root wheel 0x82 Sep 24 22:28 ttyU4.2
>>>>>>>>>>>> crw------- 1 root wheel 0x83 Sep 24 22:28 ttyU4.2.init
>>>>>>>>>>>> crw------- 1 root wheel 0x84 Sep 24 22:28 ttyU4.2.lock
>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 22:28 ttyU4.3
>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 22:28 ttyU4.3.init
>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 22:28 ttyU4.3.lock
>>>>>>>>>>>> crw------- 1 root wheel 0xa8 Sep 25 00:46 ttyU5.0
>>>>>>>>>>>> crw------- 1 root wheel 0xa9 Sep 24 22:28 ttyU5.0.init
>>>>>>>>>>>> crw------- 1 root wheel 0xaa Sep 24 22:28 ttyU5.0.lock
>>>>>>>>>>>> crw------- 1 root wheel 0xae Sep 24 22:28 ttyU5.1
>>>>>>>>>>>> crw------- 1 root wheel 0xaf Sep 24 22:28 ttyU5.1.init
>>>>>>>>>>>> crw------- 1 root wheel 0xb6 Sep 24 22:28 ttyU5.1.lock
>>>>>>>>>>>> crw------- 1 root wheel 0xba Sep 24 22:28 ttyU5.2
>>>>>>>>>>>> crw------- 1 root wheel 0xbb Sep 24 22:28 ttyU5.2.init
>>>>>>>>>>>> crw------- 1 root wheel 0xbc Sep 24 22:28 ttyU5.2.lock
>>>>>>>>>>>> crw------- 1 root wheel 0xc0 Sep 24 22:28 ttyU5.3
>>>>>>>>>>>> crw------- 1 root wheel 0xc1 Sep 24 22:28 ttyU5.3.init
>>>>>>>>>>>> crw------- 1 root wheel 0xc2 Sep 24 22:28 ttyU5.3.lock
>>>>>>>>>>>>
>>>>>>>>>>>> As you can see, ttyU1.3, ttyU2.3, ttyU3.3 are USB deadlocks
>>>>>>>>>>>>
>>>>>>>>>>>> I try to unload mod_gsmopen, but i receive a message that is in
>>>>>>>>>>>> use
>>>>>>>>>>>> so i kill freeswitch, then ttyU1.3, ttyU2.3, ttyU3.3, are free
>>>>>>>>>>>> and disappear (as well .init and .lock)
>>>>>>>>>>>>
>>>>>>>>>>>> I try to figure out whats happening, with logs, alarms, but i
>>>>>>>>>>>> can't.
>>>>>>>>>>>> Someone have the same problem, or know the solution?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ps.
>>>>>>>>>>>> Reading the list, i realize that few members use the module, or
>>>>>>>>>>>> don't have problems.
>>>>>>>>>>>> If i am correct, is there any other way to use GSM modems or
>>>>>>>>>>>> GSM, which is low cost?
>>>>>>>>>>>> Is there anyone who can guide me in some things about the code
>>>>>>>>>>>> or tips?
>>>>>>>>>>>>
>>>>>>>>>>>> ps 2.
>>>>>>>>>>>> sorry about my english
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Luiz Fernando Softov
>>>>>>>>>>>> http://www.softov.com.br
>>>>>>>>>>>> fernando at softov.com.br
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>>
>>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Luiz Fernando Softov
>>>>>>>> http://www.softov.com.br
>>>>>>>> fernando at softov.com.br
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://confluence.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://confluence.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Luiz Fernando Softov
>>>>>> http://www.softov.com.br
>>>>>> fernando at softov.com.br
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Luiz Fernando Softov
>>>>> http://www.softov.com.br
>>>>> fernando at softov.com.br
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sincerely,
>>>>
>>>> Giovanni Maruzzelli
>>>> Cell : +39-347-2665618
>>>>
>>>
>>>
>>>
>>> --
>>> Sincerely,
>>>
>>> Giovanni Maruzzelli
>>> Cell : +39-347-2665618
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>>
>> --
>> Luiz Fernando Softov
>> http://www.softov.com.br
>> fernando at softov.com.br
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151215/8e13a94f/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list