[Freeswitch-users] mod_gsmopen - deadlock tty
Shabbir abbasi
shabbirabbasi92 at gmail.com
Wed Dec 16 07:14:23 MSK 2015
i am still working on it , and i found problums in my work and corrected it
like in previus commit i was found that when i sends a ussd commands to a
disconnected modem then cli gets hanged due to there was no check whethere
modem is disconnected or connected,now i have added a check to check status
connected or not,
"How many modems are you using?"
i am just playing with it with only two modems,i am not a Developer just
trying to learn, i am using centos 7 64 bit,
my system is not in production , just for experimental,
and Discovery function in centos working fine
On Tue, Dec 15, 2015 at 9:09 PM, Luiz Fernando Softov <
fernando at softov.com.br> wrote:
> My bad, i looked better and see that you have made a discovery function.
>
> How many modems are you using?
> How many calls simultaneous?
> It's working fine? no calls stuck after you unplug/plug modem?
>
> I have found errors in source (master). And have made a lot a tests, like:
> unplug a modem in a middle of call
> - try to make a call with a modem that no support voice
> - send a sms at the same time you made a call
> - receive a call from modem A and route to modem B
>
> There is more tests that i have made, and made changes to correct it.
>
> Anyway, since i use xml_curl to grab the attached modems, and some modems
> can be removed/dettached or just have failure of power supply i
> reload/dettached/remove in according with the system/user interaction.
>
> And, that find_ttyusb_devices, don't work in FreeBSD, witch is the SO that
> i use *i can't user another system, i have made a lot of modifications in
> kernel and have a complete custom system based on that called BrbOS, with
> kernel encryption, custom drivers and other things.
>
> I make the control of calls using event_socket, and make custom
> routes according with the time, cost, user, configuration, usage,
> portability, destination...
>
> I need the system working that way, receive events when a modem cannot
> make calls, and control the modem to reload/attach when i need.
> So, because of it i create a daemon that i can control and know when a
> modem changes.
>
> I have found, in freeBSD and debian, a commom problem
>
> Its like, when i start the system for the first time
>
> Lets consider this
> ModemA (MA) - (ttyU0, ttyU1, ttyU2, ttyU3)
> ModemB (MB) - (ttyU4, ttyU5, ttyU6, ttyU7)
> ModemC (MC) - (ttyU8, ttyU9, ttyU10)
> ModemD (MD) - (ttyU11, ttyU12, ttyU13)
>
>
> MA was unplugged/have power supply problem
>
> ModemA (MA) -
> ModemB (MB) - (ttyU4, ttyU5, ttyU6, ttyU7)
> ModemC (MC) - (ttyU8, ttyU9, ttyU10)
> ModemD (MD) - (ttyU11, ttyU12, ttyU13)
>
>
> MC was unplugged/have power supply problem
>
> ModemA (MA) -
> ModemB (MB) - (ttyU4, ttyU5, ttyU6, ttyU7)
> ModemC (MC) -
> ModemD (MD) - (ttyU11, ttyU12, ttyU13)
>
> MC was plugged/redicovered
>
> ModemA (MA) -
> ModemB (MB) - (ttyU4, ttyU5, ttyU6, ttyU7)
> ModemC (MC) - (ttyU0, ttyU1, ttyU2)
> ModemD (MD) - (ttyU11, ttyU12, ttyU13)
>
> MA was plugged/redicovered
>
> ModemA (MA) - (ttyU3, ttyU8, ttyU9, ttyU10)
> ModemB (MB) - (ttyU4, ttyU5, ttyU6, ttyU7)
> ModemC (MC) - (ttyU0, ttyU1, ttyU2)
> ModemD (MD) - (ttyU11, ttyU12, ttyU13)
>
> As you see, sometimes the modems don't get the tty sequence 1.2.3.4. They
> get the sequence of free tty ports
> And, it's just in this test, when i have modems with 4/3 ttys
> Sometimes, the modem just RESET, and the tty was not free yet, so the
> modem jump a number in sequence
>
>
> MD was unplugged/have power supply problem
>
> ModemA (MA) - (ttyU3, ttyU8, ttyU9, ttyU10)
> ModemB (MB) - (ttyU4, ttyU5, ttyU6, ttyU7)
> ModemC (MC) - (ttyU0, ttyU1, ttyU2)
> ModemD (MD) -
>
>
> MD was on, but ttyU12 still oppened by FreeSwitch (mod_gsmopen)
>
> ModemA (MA) - (ttyU3, ttyU8, ttyU9, ttyU10)
> ModemB (MB) - (ttyU4, ttyU5, ttyU6, ttyU7)
> ModemC (MC) - (ttyU0, ttyU1, ttyU2)
> ModemD (MD) - (ttyU11, ttyU13, ttyU14)
>
>
> As you see, the discovery will not work sometimes, you need to change to
> get all ttyports
> not getting as (ttyFOUND + 1) and (ttyFOUND + 2)
>
>
> And, like i said before, if you wan't to made changes i'm able to help or
> to rewrite the module, just need someone to help
> If you wan't, send me a e-mail or just reply here
>
> 2015-12-08 9:09 GMT-03:00 Shabbir abbasi <shabbirabbasi92 at gmail.com>:
>
>> Hi:
>> i have made patch for auto reconnect if dongle disconnected and
>> display status of dongle connected or disconnected
>> https://freeswitch.org/jira/browse/FS-8626
>>
>> i think after this patch you dont need a seprate daemon to run
>> thank u
>>
>> On Sun, Sep 27, 2015 at 7:53 AM, Luiz Fernando Softov <
>> fernando at softov.com.br> wrote:
>>
>>> Hi, i sent a e-mail to you, with the diff, files and give you access to
>>> web interface of the system.
>>>
>>> I will wait you to read, confirm and reply my e-mail.
>>>
>>> Thanks a lot for the help.
>>>
>>> 2015-09-26 15:28 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>>
>>>> As per chat, I'm "gmaruzz" in both FreeSWITCH's hipchat and in IRC
>>>> #freeswitch channel at irc.freenode.net
>>>>
>>>> -giovanni
>>>>
>>>> On Sat, Sep 26, 2015 at 9:25 PM, Giovanni Maruzzelli <gmaruzz at gmail.com
>>>> > wrote:
>>>>
>>>>> On Fri, Sep 25, 2015 at 11:30 PM, Luiz Fernando Softov <
>>>>> fernando at softov.com.br> wrote:
>>>>>
>>>>>> Now, i make some changes, and until now it's running fine
>>>>>>
>>>>>> I think the problem was when you declare interface as dead, like line
>>>>>> 674 of gsmopen_protocol.cpp
>>>>>>
>>>>>>
>>>>>> https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/src/mod/endpoints/mod_gsmopen/gsmopen_protocol.cpp
>>>>>>
>>>>>> alarm_event(tech_pvt, ALARM_FAILED_INTERFACE, "gsmopen_serial_monitor
>>>>>> failed, declaring interface dead");
>>>>>> tech_pvt->active = 0;
>>>>>> ->>>> tech_pvt->name[0] = '\0';
>>>>>>
>>>>>> You make a \0 in interface name
>>>>>>
>>>>>> So, when someone call remove_interface() in mod_gsmopen.cpp
>>>>>>
>>>>>>
>>>>>> https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/src/mod/endpoints/mod_gsmopen/mod_gsmopen.cpp
>>>>>>
>>>>>> To grab the tech_pvt, the function check for strlen or compare
>>>>>> GSMOPEN_INTERFACES[interface_id].name (line 322 and 329)
>>>>>>
>>>>>> Then, we will not found interface (tech_pvt), because "name" is
>>>>>> nothing
>>>>>>
>>>>>>
>>>>>>
>>>>> continuing, in line 336 it's check if there is a tech_pvt
>>>>>>
>>>>>> if (!tech_pvt) {
>>>>>> DEBUGA_GSMOPEN("interface '%s' does not exist\n", GSMOPEN_P_LOG,
>>>>>> the_interface);
>>>>>> goto end;
>>>>>> }
>>>>>>
>>>>>> This check will jump the Close in line 385, causing a deadlock opened
>>>>>> fd.
>>>>>> res = tech_pvt->serialPort_serial_control->Close();
>>>>>>
>>>>>> So i comment every line when you set tech_pvt->name[0] = '\0';
>>>>>>
>>>>>> It's correct? Since when interface failed, and it's declared as died,
>>>>>> the serial isn't closed, just marked as died, and the tty still opened
>>>>>> causing a deadlock, if no one close this.
>>>>>>
>>>>>>
>>>>> your reasoning seems sound, is the system still running ok?
>>>>>
>>>>>
>>>>> Have you changed anything else?
>>>>>
>>>>> Can you send me (to my mail, as an attach) a git diff? Or open a jira
>>>>> attaching that diff?
>>>>>
>>>>> Btw, yes, I'm interested in understanding better how your system works
>>>>> (daemon, attach-detach, and all).
>>>>>
>>>>> Also, big thanks for taking the time to look into the source code!!!
>>>>>
>>>>> -giovanni
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-09-25 14:10 GMT-04:00 Luiz Fernando Softov <
>>>>>> fernando at softov.com.br>:
>>>>>>
>>>>>>> I make a daemon, witch manage USB using kqueue and other things, so,
>>>>>>> every time i plug or unplug a modem, i make a event, like ARRIVE and LEAVE.
>>>>>>>
>>>>>>> This daemon have a list of modems, correlate with ttys ids - U3G
>>>>>>> (U0.0, U0.1, U0.2) and UMODEM (U0 U1 U2)
>>>>>>>
>>>>>>> So, when a modem ARRIVE, and it's "attached" (configured), i call a
>>>>>>> "gsm reload".
>>>>>>> When a modem LEAVE, i call a gsm remove interface_x.
>>>>>>>
>>>>>>> Using mod_xml_curl, i have dynamic config.
>>>>>>>
>>>>>>> I use events, like gsmopen_dump list, to manage and controll
>>>>>>> interfaces, like when user "detach" a modem.
>>>>>>>
>>>>>>> If you want, send me a e-mail, i will give access to my system and
>>>>>>> you can see what i'm talking about.
>>>>>>>
>>>>>>>
>>>>>>> Anyway, i found things like in function
>>>>>>>
>>>>>>> int gsmopen_serial_write_AT_noack(private_t *tech_pvt, const char
>>>>>>> *data)
>>>>>>>
>>>>>>> // You make
>>>>>>> LOKKA(tech_pvt->controldev_lock);
>>>>>>>
>>>>>>> Then call gsmopen_serial_write_AT(tech_pvt, data), and
>>>>>>> inside gsmopen_serial_write_AT, when something is wrong, you
>>>>>>> make UNLOCKA(tech_pvt->controldev_lock); and return -1
>>>>>>>
>>>>>>> But, you check if (gsmopen_serial_write_AT(tech_pvt, data) != (int)
>>>>>>> strlen(data)) {
>>>>>>>
>>>>>>> Then, you make again UNLOCKA(tech_pvt->controldev_lock);
>>>>>>>
>>>>>>> Why make two times UNLOCKA(tech_pvt->controldev_lock); ?
>>>>>>>
>>>>>>> It's like
>>>>>>> LOKKA(tech_pvt->controldev_lock);
>>>>>>> UNLOCKA(tech_pvt->controldev_lock);
>>>>>>> UNLOCKA(tech_pvt->controldev_lock);
>>>>>>>
>>>>>>>
>>>>>>> Another is in
>>>>>>> int gsmopen_serial_read_AT(private_t *tech_pvt, int look_for_ack,
>>>>>>> int timeout_usec, int timeout_sec, const char *expected_string, int
>>>>>>> expect_crlf)
>>>>>>> You make a LOKKA(tech_pvt->controldev_lock);, but there is some
>>>>>>> returns (inside try catch), without UNLOCKA.
>>>>>>>
>>>>>>> Sorry if i make some mistakes in my explanation, but i just want to
>>>>>>> undestand how this funcions works.
>>>>>>>
>>>>>>> I disagree with "Shaabir" about this module is not useable. I just
>>>>>>> think this need some changes to work fine.
>>>>>>> I already test this module with 40 modems for 7 days, and it's work
>>>>>>> fine, until i grab a deadlock in USB.
>>>>>>>
>>>>>>> Giovanni, if you have time to chat with me, i will be grateful.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2015-09-25 12:41 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>>>>>>
>>>>>>>> for plugging and unplugging modems, that's a very difficult thing
>>>>>>>> to manage, this is the best I can come out with (and probably does not work
>>>>>>>> in freebsd):
>>>>>>>>
>>>>>>>>
>>>>>>>> http://wiki.freeswitch.org/wiki/GSMopen#IMEI_and_IMSI_automatic_device_discovery_.28only_on_Linux.29_2
>>>>>>>>
>>>>>>>> you're welcome to find a similar mechanism, and file a jira with a
>>>>>>>> pull request, so it will be inserted in manline.
>>>>>>>>
>>>>>>>> If your modifications only apply to freebsd, please use #ifdefs, so
>>>>>>>> it still continue to compile and work in Linux.
>>>>>>>>
>>>>>>>> -giovanni
>>>>>>>>
>>>>>>>>
>>>>>>>> Sincerely,
>>>>>>>>
>>>>>>>> Giovanni Maruzzelli
>>>>>>>> Cell : +39-347-2665618
>>>>>>>>
>>>>>>>> On Fri, Sep 25, 2015 at 6:24 PM, Luiz Fernando Softov <
>>>>>>>> fernando at softov.com.br> wrote:
>>>>>>>>
>>>>>>>>> I'm using a USB with power supply.
>>>>>>>>>
>>>>>>>>> But sometimes it's just happen. Sometimes user remove and plug
>>>>>>>>> modem again.
>>>>>>>>>
>>>>>>>>> I can't use Debian, because our system depends of FreeBSD, and we
>>>>>>>>> have a lot of source (daemons) running, based, on that platform. And we use
>>>>>>>>> kqueue, I think Debian don't have kqueue, it's like epool, but better, much
>>>>>>>>> better.
>>>>>>>>>
>>>>>>>>> Anyway, I don't That it's a kernel problem, because, since I have
>>>>>>>>> another daemon, attached in ttyx.0 (mod_gsmopen attach to ttyx.3), and when
>>>>>>>>> modem power off or "reset", only ttyx.3 still locked.
>>>>>>>>>
>>>>>>>>> I'm trying to read the source to figure what's happening, but it's
>>>>>>>>> hard to read, not because the source is difficult, but because isn't
>>>>>>>>> organized.
>>>>>>>>>
>>>>>>>>> I'm thinking in rewrite the module, but if in future someone
>>>>>>>>> change something in original module I will need to check my source.
>>>>>>>>> There is another thing, I don't know how things like apr_thread
>>>>>>>>> works, and what functions like switch_sleep(1000000) do and why using that.
>>>>>>>>>
>>>>>>>>> Today I make some alterations where the module make a lock, LOKKA,
>>>>>>>>> there is some functions who don't have unlock, and another who only have
>>>>>>>>> unlock, sometimes two unlocks. So I will wait to see what will happen.
>>>>>>>>>
>>>>>>>>> Now, here it's 3:00 am, and I'm going to sleep, almost 19 hours
>>>>>>>>> awake up, trying to debug this module and another things.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Maybe you or someone have a time, to chat, or reply some e-mails,
>>>>>>>>> or maybe here, to explain some questions about the source (module)?
>>>>>>>>>
>>>>>>>>> There's some lines marked with TODO, why?
>>>>>>>>> And there is another version of this module?
>>>>>>>>> Which the maximum number of modems that you already usage?
>>>>>>>>>
>>>>>>>>> 2015-09-25 7:53 GMT-04:00 Giovanni Maruzzelli <gmaruzz at gmail.com>:
>>>>>>>>>
>>>>>>>>>> if you found problems, please open a jira issue, with all info to
>>>>>>>>>> replicate the problem, as explained in gsmopen page.
>>>>>>>>>>
>>>>>>>>>> It would be useful for us developers, and to me personally, being
>>>>>>>>>> the person in charge of gsmopen.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance,
>>>>>>>>>>
>>>>>>>>>> -giovanni
>>>>>>>>>>
>>>>>>>>>> Sincerely,
>>>>>>>>>>
>>>>>>>>>> Giovanni Maruzzelli
>>>>>>>>>> Cell : +39-347-2665618
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 25, 2015 at 1:44 PM, Shabbir abbasi <
>>>>>>>>>> shabbirabbasi92 at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> mod_gsmopen needs more development work, i have tested it
>>>>>>>>>>> sometimes ago and found it is not useable , so now i am using asterisk
>>>>>>>>>>> with chan_dongle better then gsmopen without usb HUB
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 25, 2015 at 11:11 AM, Giovanni Maruzzelli <
>>>>>>>>>>> gmaruzz at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> This seems to be a kernel problem.
>>>>>>>>>>>>
>>>>>>>>>>>> To have the most success chances: use a USB hub with power
>>>>>>>>>>>> supply (eg, a USB hub that get electrical power from the wall socket).
>>>>>>>>>>>> Also, use the reference.platform, debian 8 jessie, instead of
>>>>>>>>>>>> freebsd.
>>>>>>>>>>>>
>>>>>>>>>>>> -goovanni
>>>>>>>>>>>>
>>>>>>>>>>>> sent from my mobile,
>>>>>>>>>>>> Giovanni Maruzzelli
>>>>>>>>>>>> cell: +39 347 266 56 18
>>>>>>>>>>>> On Sep 25, 2015 7:19 AM, "Luiz Fernando Softov" <
>>>>>>>>>>>> fernando at softov.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm using freeswitch 1.15 in Freebsd 10.1, with mod_gsmopen
>>>>>>>>>>>>>
>>>>>>>>>>>>> FreeSWITCH Version
>>>>>>>>>>>>> 1.5.15b+git~20150407T231134Z~d55c4a053b~64bit (git d55c4a0 2015-04-07
>>>>>>>>>>>>> 23:11:34Z 64bit), whats happen, its the same with another versions like,
>>>>>>>>>>>>> 1.6, 1.4.19, etc
>>>>>>>>>>>>>
>>>>>>>>>>>>> After sometime running freeswitch, i have this TTYs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 15:30 ttyU1.3
>>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 15:30 ttyU1.3.init
>>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 15:30 ttyU1.3.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 15:30 ttyU2.3
>>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 15:30 ttyU2.3.init
>>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 15:30 ttyU2.3.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 15:30 ttyU3.3
>>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 15:30 ttyU3.3.init
>>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 15:30 ttyU3.3.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0x76 Sep 25 00:46 ttyU4.0
>>>>>>>>>>>>> crw------- 1 root wheel 0x77 Sep 24 22:28 ttyU4.0.init
>>>>>>>>>>>>> crw------- 1 root wheel 0x78 Sep 24 22:28 ttyU4.0.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0x7c Sep 24 22:28 ttyU4.1
>>>>>>>>>>>>> crw------- 1 root wheel 0x7d Sep 24 22:28 ttyU4.1.init
>>>>>>>>>>>>> crw------- 1 root wheel 0x7e Sep 24 22:28 ttyU4.1.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0x82 Sep 24 22:28 ttyU4.2
>>>>>>>>>>>>> crw------- 1 root wheel 0x83 Sep 24 22:28 ttyU4.2.init
>>>>>>>>>>>>> crw------- 1 root wheel 0x84 Sep 24 22:28 ttyU4.2.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0x8e Sep 24 22:28 ttyU4.3
>>>>>>>>>>>>> crw------- 1 root wheel 0x8f Sep 24 22:28 ttyU4.3.init
>>>>>>>>>>>>> crw------- 1 root wheel 0x90 Sep 24 22:28 ttyU4.3.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0xa8 Sep 25 00:46 ttyU5.0
>>>>>>>>>>>>> crw------- 1 root wheel 0xa9 Sep 24 22:28 ttyU5.0.init
>>>>>>>>>>>>> crw------- 1 root wheel 0xaa Sep 24 22:28 ttyU5.0.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0xae Sep 24 22:28 ttyU5.1
>>>>>>>>>>>>> crw------- 1 root wheel 0xaf Sep 24 22:28 ttyU5.1.init
>>>>>>>>>>>>> crw------- 1 root wheel 0xb6 Sep 24 22:28 ttyU5.1.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0xba Sep 24 22:28 ttyU5.2
>>>>>>>>>>>>> crw------- 1 root wheel 0xbb Sep 24 22:28 ttyU5.2.init
>>>>>>>>>>>>> crw------- 1 root wheel 0xbc Sep 24 22:28 ttyU5.2.lock
>>>>>>>>>>>>> crw------- 1 root wheel 0xc0 Sep 24 22:28 ttyU5.3
>>>>>>>>>>>>> crw------- 1 root wheel 0xc1 Sep 24 22:28 ttyU5.3.init
>>>>>>>>>>>>> crw------- 1 root wheel 0xc2 Sep 24 22:28 ttyU5.3.lock
>>>>>>>>>>>>>
>>>>>>>>>>>>> As you can see, ttyU1.3, ttyU2.3, ttyU3.3 are USB deadlocks
>>>>>>>>>>>>>
>>>>>>>>>>>>> I try to unload mod_gsmopen, but i receive a message that is
>>>>>>>>>>>>> in use
>>>>>>>>>>>>> so i kill freeswitch, then ttyU1.3, ttyU2.3, ttyU3.3, are free
>>>>>>>>>>>>> and disappear (as well .init and .lock)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I try to figure out whats happening, with logs, alarms, but i
>>>>>>>>>>>>> can't.
>>>>>>>>>>>>> Someone have the same problem, or know the solution?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ps.
>>>>>>>>>>>>> Reading the list, i realize that few members use the module,
>>>>>>>>>>>>> or don't have problems.
>>>>>>>>>>>>> If i am correct, is there any other way to use GSM modems or
>>>>>>>>>>>>> GSM, which is low cost?
>>>>>>>>>>>>> Is there anyone who can guide me in some things about the code
>>>>>>>>>>>>> or tips?
>>>>>>>>>>>>>
>>>>>>>>>>>>> ps 2.
>>>>>>>>>>>>> sorry about my english
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Luiz Fernando Softov
>>>>>>>>>>>>> http://www.softov.com.br
>>>>>>>>>>>>> fernando at softov.com.br
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>>
>>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Luiz Fernando Softov
>>>>>>>>> http://www.softov.com.br
>>>>>>>>> fernando at softov.com.br
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://confluence.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Luiz Fernando Softov
>>>>>>> http://www.softov.com.br
>>>>>>> fernando at softov.com.br
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Luiz Fernando Softov
>>>>>> http://www.softov.com.br
>>>>>> fernando at softov.com.br
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://confluence.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sincerely,
>>>>>
>>>>> Giovanni Maruzzelli
>>>>> Cell : +39-347-2665618
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sincerely,
>>>>
>>>> Giovanni Maruzzelli
>>>> Cell : +39-347-2665618
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>>
>>> --
>>> Luiz Fernando Softov
>>> http://www.softov.com.br
>>> fernando at softov.com.br
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151216/2c7d779f/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list