[Freeswitch-users] FreeSWITCH Stops Responding

Brian West brian at freeswitch.com
Wed Oct 6 13:33:43 UTC 2021 

Ah, I see where you might have zigged where thou shall zagged, 'Gentoo
servers', RUN!

Seriously, that's a minefield of WTF.

/b


On Tue, Oct 5, 2021 at 7:11 PM Cary Shouse <cary at ytel.com> wrote:

> Hello,
>
> Just sending an update on this. The Debian Freeswitch server we set up had
> the same issue as our Gentoo servers. Looking at the backtrace we noticed
> the first lowlevellock was occurring when or OurOutbound.luac was being
> used. We decided to use a version that was not precompiled. After that
> change the issue has only occurred once more. So there is still a problem
> we are looking into, but the problems have been massively reduced with that
> change.
>
> Part of the backtrace
> Thread 40 (Thread 0x7f2fe191e700 (LWP 15884)):
> #0  __lll_lock_wait_private () at
> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
> #1  0x00007f30512cce53 in __GI__IO_link_in (fp=fp at entry=0x7f2fec53e400)
> at genops.c:93
> #2  0x00007f30512cbb92 in _IO_new_file_init_internal (fp=fp at entry=0x7f2fec53e400)
> at fileops.c:114
> #3  0x00007f30512c0148 in __fopen_internal (filename=0x7f2fec4d5920
> "/usr/local/freeswitch/scripts/360/V3/lua/OurOutbound.luac",
> mode=0x7f2fe37d81b2 "r", is32=1) at iofopen.c:74
> #4  0x00007f2fe37cc123 in luaL_loadfilex () from
> /lib/x86_64-linux-gnu/liblua5.2.so.0
> #5  0x00007f302405796d in lua_parse_and_execute (L=<optimized out>,
> input_code=<optimized out>, session=0x7f2fc91217f8) at mod_lua.cpp:194
> #6  0x00007f3024058191 in lua_function (session=0x7f2fc91217f8,
> data=<optimized out>) at mod_lua.cpp:483
> #7  0x00007f30518920df in switch_core_session_exec (session=session at entry=0x7f2fc91217f8,
> application_interface=application_interface at entry=0x55d5e9528150,
> arg=arg at entry=0x7f2fc96819c0 "360/V3/lua/OurOutbound.luac") at
> src/switch_core_session.c:2900
> #8  0x00007f305189276f in
> switch_core_session_execute_application_get_flags (session=session at entry=0x7f2fc91217f8,
> app=0x7f2fc96819b8 "lua", arg=0x7f2fc96819c0 "360/V3/lua/OurOutbound.luac",
> flags=flags at entry=0x0) at src/switch_core_session.c:2765
> #9  0x00007f3051896ab2 in switch_core_standard_on_execute
> (session=0x7f2fc91217f8) at src/switch_core_state_machine.c:354
> #10 switch_core_session_run (session=0x7f2fc91217f8) at
> src/switch_core_state_machine.c:651
> #11 0x00007f305188face in switch_core_session_thread (thread=<optimized
> out>, obj=0x7f2fc91217f8) at src/switch_core_session.c:1719
> #12 0x00007f305188b4c3 in switch_core_session_thread_pool_worker
> (thread=0x7f3034236c70, obj=<optimized out>) at
> src/switch_core_session.c:1782
> #13 0x00007f3051bc536c in dummy_worker (opaque=0x7f3034236c70) at
> threadproc/unix/thread.c:151
> #14 0x00007f3051793fa3 in start_thread (arg=<optimized out>) at
> pthread_create.c:486
> #15 0x00007f30513494cf in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
> On Thu, Sep 9, 2021 at 1:37 AM Cary Shouse <cary at ytel.com> wrote:
>
>> We have set up a Debian 10 FreeSWITCH-1.10.6 server for
>> testing/debugging.
>>
>> On Sat, Sep 4, 2021 at 5:51 PM David Villasmil <
>> david.villasmil.work at gmail.com> wrote:
>>
>>> That is correct. But we who have worked with FS since around 2002 know
>>> this, newer people get all worked up until they’re told this :)
>>>
>>> It’s a learning thing..
>>>
>>> On Sat, 4 Sep 2021 at 17:31, Ken Rice <krice at freeswitch.org> wrote:
>>>
>>>> “stuck channels” usually arent stuck once you realize things like “show
>>>> calls” and “show channels” are selects against the coredb. this usually
>>>> happens on busy systems timing put waiting for db delete’s to run that fail
>>>> or get skipped.
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Sep 4, 2021, at 05:22, David Villasmil <
>>>> david.villasmil.work at gmail.com> wrote:
>>>>
>>>> 
>>>>
>>>> Hello all,
>>>>
>>>> Interesting point about Gentoo, but I highly doubt it’s OS related.
>>>> I’ve been living this issue (about those “stuck channels”, not a frozen FS,
>>>> that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian,
>>>> Ubuntu, etc.
>>>>
>>>> In my experience this became an issue with the rise of VM. Back in the
>>>> day you would see this every now and then, like months apart (this, or a
>>>> crashed MySQL database), but wasn’t big enough to be a critical problem.
>>>> Today day it’s basically an everyday thing if you have enough nodes.
>>>>
>>>>
>>>> On Sat, 4 Sep 2021 at 00:15, Ken Rice <krice at freeswitch.org> wrote:
>>>>
>>>>> Having massive experience here, you might try replacing your linux
>>>>> distro.
>>>>>
>>>>>  (Yes i know some people are going to take this as brushing off the
>>>>> problem or trying to start a distro war. If this starts a distro war I will
>>>>> moderate you and only allow replies are not headed to a flame war.)
>>>>>
>>>>> However, as someone who has ran an extremely large Gentoo based
>>>>> cluster, we had to abandon gentoo due to random crashes, unexplainable bugs
>>>>> that could not be reproduced or explained.
>>>>>
>>>>> If you can not replace the distro I VERY HiGHLY recommend running
>>>>> vanilla libc and other libs without the gentoo portage included patches.
>>>>> (hint the use flag: vanilla)
>>>>>
>>>>> Ken
>>>>> Sent from my iPhone
>>>>>
>>>>> On Sep 3, 2021, at 14:49, Cary Shouse <cary at ytel.com> wrote:
>>>>>
>>>>> 
>>>>>
>>>>>  We have not been able to reproduce this in testing. It only happens in our production environment.
>>>>>  Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into.  We could get you access to a server if that would help.
>>>>> Server Details:
>>>>> Gentoo-5.10.27
>>>>> RAM: 15GB
>>>>> CPU: 4
>>>>> FreeSWITCH-1-.10.6
>>>>> Sqlite-3.34.1
>>>>>
>>>>>
>>>>> On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian at freeswitch.com>
>>>>> wrote:
>>>>>
>>>>>> And these exhibit the issue on command?
>>>>>>
>>>>>> On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary at ytel.com> wrote:
>>>>>>
>>>>>>> Here are two originate samples.
>>>>>>>
>>>>>>> https://pastebin.com/YKRKrJLu
>>>>>>>
>>>>>>> https://pastebin.com/TEX3yfX6
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian at freeswitch.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Can you try to create one where this happens without all this mess?
>>>>>>>>
>>>>>>>> On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary at ytel.com> wrote:
>>>>>>>>
>>>>>>>>> Here is the LUA code regarding creating sessions and bridging. It
>>>>>>>>> is broken up into 5 sections.
>>>>>>>>> The top part is the common functions, and the 4 sections below are
>>>>>>>>> different call scenarios.
>>>>>>>>> https://pastebin.com/qqGw3tms
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian at freeswitch.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I you do sessions from your lua, I need to see how you do that.
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary at ytel.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, it bridges the inbound call to an outbound one.
>>>>>>>>>>>
>>>>>>>>>>> (ProxyOutbound.luac)https://pastebin.com/RDUMy473
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian at freeswitch.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Guessing you create sessions in lua? ProxyOutbound.luac?
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 31, 2021 at 3:33 PM Brian West <
>>>>>>>>>>>> brian at freeswitch.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> and to make sure when you uuid kill one it says it didn't
>>>>>>>>>>>>> exist but show channels had it still listed correct?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary at ytel.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have got the logs for a few calls that were stuck.
>>>>>>>>>>>>>> (Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
>>>>>>>>>>>>>> https://pastebin.com/NTnte7J7
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
>>>>>>>>>>>>>> https://pastebin.com/sAhFiGjY
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
>>>>>>>>>>>>>> https://pastebin.com/bmcdjzAr
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
>>>>>>>>>>>>>> https://pastebin.com/aGyY3m9F
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)
>>>>>>>>>>>>>> https://pastebin.com/0cnQaRhC
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
>>>>>>>>>>>>>> https://pastebin.com/atX0VXdZ
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please let us know if there is anything else we can provide
>>>>>>>>>>>>>> that would be useful. Thank you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Aug 30, 2021 at 8:20 PM Brian West <
>>>>>>>>>>>>>> brian at freeswitch.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Check the UUID of the session in question, and extract its
>>>>>>>>>>>>>>> logs, and let's diagnose it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary at ytel.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  Yes, that is what we are seeing when running uuid_kill.  We
>>>>>>>>>>>>>>>> cannot "replicate" the issue, but it happens multiple times a day for us
>>>>>>>>>>>>>>>> across our servers.   We have looked through logs across
>>>>>>>>>>>>>>>> multiple systems, taken SIP traces to see if that showed anything.
>>>>>>>>>>>>>>>> Last we got a Core dump of the running process after the
>>>>>>>>>>>>>>>> failure. Nothing seems to stand out as the cause.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  Are there any other debug steps you can think of or what
>>>>>>>>>>>>>>>> other information to look at?  We could capture all traffic on the server,
>>>>>>>>>>>>>>>> or recompile with special debug flags whatever it is that might help.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Steps we have taken:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sofia loglevel all 9
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sofia global siptrace on
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> fsctl loglevel 7
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> # Set environment variable
>>>>>>>>>>>>>>>> TPORT_LOG=1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Aug 28, 2021 at 12:54 AM Brian West <
>>>>>>>>>>>>>>>> brian at freeswitch.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's an issue we've been searching for, if you uuid_kill
>>>>>>>>>>>>>>>>> it, and it says no session, it only means the record was NOT removed from
>>>>>>>>>>>>>>>>> the sqlite database.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Nobody can seem to replicate the steps required to get
>>>>>>>>>>>>>>>>> into that condition, maybe you could look thru the logs and see if you can
>>>>>>>>>>>>>>>>> replicate what ever happened on that lingering UUID
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /b
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary at ytel.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>  We recently started having a problem with our FreeSWITCH
>>>>>>>>>>>>>>>>>> servers where calls are working well, with about 200 calls operating
>>>>>>>>>>>>>>>>>> sometimes for a couple of days, and then something happens and all call
>>>>>>>>>>>>>>>>>> progress stops. Then any new inbound calls that come into the server get
>>>>>>>>>>>>>>>>>> stuck in initial status, new inbound get stuck at RINGING. This happens
>>>>>>>>>>>>>>>>>> until the max sessions are hit and of course nothing else works from there.
>>>>>>>>>>>>>>>>>> Our thought is something is creating a lock and stopping anything else from
>>>>>>>>>>>>>>>>>> progressing.
>>>>>>>>>>>>>>>>>>  We are seeing the stuck sessions using "show channels".
>>>>>>>>>>>>>>>>>> Doing a uuid_kill on the calls seems to make no difference to the output of
>>>>>>>>>>>>>>>>>> "show channels", but we have noticed in the logs that some events do happen
>>>>>>>>>>>>>>>>>> to some of the calls.
>>>>>>>>>>>>>>>>>>  We had ran into a similar issue when writing a custom C
>>>>>>>>>>>>>>>>>> module and made some mistakes with session_rwlock etc. But in this case,
>>>>>>>>>>>>>>>>>> all of this is using LUA.
>>>>>>>>>>>>>>>>>> Some general information:
>>>>>>>>>>>>>>>>>> FreeSWITCH Versions 1.10.6 and 1.10.1
>>>>>>>>>>>>>>>>>> Modules used:
>>>>>>>>>>>>>>>>>> LUA, xml_curl(dialplan), event_socket,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>  Most of this is just application playing audio files and
>>>>>>>>>>>>>>>>>> or creating a new call to bridge to.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>  Really looking for some help on what debugging we can
>>>>>>>>>>>>>>>>>> enable to help identify what is getting stuck to identify where a bug might
>>>>>>>>>>>>>>>>>> be on our usage or possible issue with FreeSWITCH.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>  We have captured a core dump of the running process when
>>>>>>>>>>>>>>>>>> it got into the bad state as well, if that is helpful, we can send it in.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The FreeSWITCH project is sponsored by SignalWire
>>>>>>>>>>>>>>>>>> https://signalwire.com
>>>>>>>>>>>>>>>>>> Enhance your FreeSWITCH install with disruptive priced
>>>>>>>>>>>>>>>>>> SMS and PSTN services.
>>>>>>>>>>>>>>>>>> Build your next product on our scalable cloud platform.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Join our online community to chat in real time
>>>>>>>>>>>>>>>>>> https://signalwire.community
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Professional FreeSWITCH Services
>>>>>>>>>>>>>>>>>> sales at freeswitch.com
>>>>>>>>>>>>>>>>>> https://freeswitch.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>>>>>>>>> https://freeswitch.com/oss
>>>>>>>>>>>>>>>>>> https://freeswitch.org/confluence
>>>>>>>>>>>>>>>>>> https://cluecon.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>>>>>>>>> https://freeswitch.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531
>>>>>>>>>>>>>>>>> Brookfield, WI 53045
>>>>>>>>>>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Mobile: 918-424-9378
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Website: https://www.FreeSWITCH.com
>>>>>>>>>>>>>>>>> <https://www.freeswitch.com/>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>>>>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>>>>>>>>>>> https://twitter.com/freeswitch]
>>>>>>>>>>>>>>>>> <https://twitter.com/freeswitch>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield,
>>>>>>>>>>>>>>> WI 53045
>>>>>>>>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mobile: 918-424-9378
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Website: https://www.FreeSWITCH.com
>>>>>>>>>>>>>>> <https://www.freeswitch.com/>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>>>>>>>>> https://twitter.com/freeswitch]
>>>>>>>>>>>>>>> <https://twitter.com/freeswitch>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI
>>>>>>>>>>>>> 53045
>>>>>>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mobile: 918-424-9378
>>>>>>>>>>>>>
>>>>>>>>>>>>> Website: https://www.FreeSWITCH.com
>>>>>>>>>>>>> <https://www.freeswitch.com/>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>>>>>>> https://twitter.com/freeswitch]
>>>>>>>>>>>>> <https://twitter.com/freeswitch>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>>>>>
>>>>>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>>>>>
>>>>>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI
>>>>>>>>>>>> 53045
>>>>>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>>>>>
>>>>>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>>>>>
>>>>>>>>>>>> Mobile: 918-424-9378
>>>>>>>>>>>>
>>>>>>>>>>>> Website: https://www.FreeSWITCH.com
>>>>>>>>>>>> <https://www.freeswitch.com/>
>>>>>>>>>>>>
>>>>>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>>>>>> https://twitter.com/freeswitch]
>>>>>>>>>>>> <https://twitter.com/freeswitch>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>>>
>>>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI
>>>>>>>>>> 53045
>>>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>>>
>>>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>>>
>>>>>>>>>> Mobile: 918-424-9378
>>>>>>>>>>
>>>>>>>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>>>>>>>
>>>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>
>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>
>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>
>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>
>>>>>>>> Mobile: 918-424-9378
>>>>>>>>
>>>>>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>>>>>
>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Brian West | Co-founder and Developer
>>>>>>
>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>
>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>
>>>>>> Email: brian at freeswitch.com
>>>>>>
>>>>>> Mobile: 918-424-9378
>>>>>>
>>>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>>>
>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>>
>>>>> The FreeSWITCH project is sponsored by SignalWire
>>>>> https://signalwire.com
>>>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>>>> services.
>>>>> Build your next product on our scalable cloud platform.
>>>>>
>>>>> Join our online community to chat in real time
>>>>> https://signalwire.community
>>>>>
>>>>> Professional FreeSWITCH Services
>>>>> sales at freeswitch.com
>>>>> https://freeswitch.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> https://freeswitch.com/oss
>>>>> https://freeswitch.org/confluence
>>>>> https://cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> https://freeswitch.com
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>>
>>>>> The FreeSWITCH project is sponsored by SignalWire
>>>>> https://signalwire.com
>>>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>>>> services.
>>>>> Build your next product on our scalable cloud platform.
>>>>>
>>>>> Join our online community to chat in real time
>>>>> https://signalwire.community
>>>>>
>>>>> Professional FreeSWITCH Services
>>>>> sales at freeswitch.com
>>>>> https://freeswitch.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> https://freeswitch.com/oss
>>>>> https://freeswitch.org/confluence
>>>>> https://cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> https://freeswitch.com
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> David Villasmil
>>>> email: david.villasmil.work at gmail.com
>>>> phone: +34669448337
>>>>
>>>> _________________________________________________________________________
>>>>
>>>> The FreeSWITCH project is sponsored by SignalWire
>>>> https://signalwire.com
>>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>>> services.
>>>> Build your next product on our scalable cloud platform.
>>>>
>>>> Join our online community to chat in real time
>>>> https://signalwire.community
>>>>
>>>> Professional FreeSWITCH Services
>>>> sales at freeswitch.com
>>>> https://freeswitch.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> https://freeswitch.com/oss
>>>> https://freeswitch.org/confluence
>>>> https://cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> https://freeswitch.com
>>>>
>>>>
>>>> _________________________________________________________________________
>>>>
>>>> The FreeSWITCH project is sponsored by SignalWire
>>>> https://signalwire.com
>>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>>> services.
>>>> Build your next product on our scalable cloud platform.
>>>>
>>>> Join our online community to chat in real time
>>>> https://signalwire.community
>>>>
>>>> Professional FreeSWITCH Services
>>>> sales at freeswitch.com
>>>> https://freeswitch.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> https://freeswitch.com/oss
>>>> https://freeswitch.org/confluence
>>>> https://cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> https://freeswitch.com
>>>
>>> --
>>> Regards,
>>>
>>> David Villasmil
>>> email: david.villasmil.work at gmail.com
>>> phone: +34669448337
>>> _________________________________________________________________________
>>>
>>> The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>> services.
>>> Build your next product on our scalable cloud platform.
>>>
>>> Join our online community to chat in real time
>>> https://signalwire.community
>>>
>>> Professional FreeSWITCH Services
>>> sales at freeswitch.com
>>> https://freeswitch.com
>>>
>>> Official FreeSWITCH Sites
>>> https://freeswitch.com/oss
>>> https://freeswitch.org/confluence
>>> https://cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> https://freeswitch.com
>>
>> _________________________________________________________________________
>
> The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
> services.
> Build your next product on our scalable cloud platform.
>
> Join our online community to chat in real time
> https://signalwire.community
>
> Professional FreeSWITCH Services
> sales at freeswitch.com
> https://freeswitch.com
>
> Official FreeSWITCH Sites
> https://freeswitch.com/oss
> https://freeswitch.org/confluence
> https://cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> https://freeswitch.com



-- 

Brian West | Co-founder and Developer

Need Commercial support? email sales at freeswitch.com

FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
<https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>

Email: brian at freeswitch.com

Mobile: 918-424-9378

Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>

[image: https://www.facebook.com/signalwireinc?src=email]
<https://www.facebook.com/freeswitch> [image:
https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20211006/12cecf74/attachment-0001.html>

More information about the FreeSWITCH-users mailing list