<div dir="ltr"><font face="arial, sans-serif">Ah, I see where you might have zigged where thou shall zagged, 'Gentoo servers', RUN!</font><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Seriously, that's a minefield of WTF.</font><br><div><span style="font-family:monospace,monospace"><br></span></div><div><span style="font-family:monospace,monospace">/b</span></div><div><span style="font-family:monospace,monospace"><br></span></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 5, 2021 at 7:11 PM Cary Shouse <<a href="mailto:cary@ytel.com">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><table cellpadding="0" id="gmail-m_-5574547624929765182gmail-undefined" style="width:904px;table-layout:fixed;border-spacing:0px"><tbody><tr><td style="vertical-align:top;width:904px;height:100px"><div id="gmail-m_-5574547624929765182gmail-:2k8" style="border-radius:1px;box-sizing:border-box;padding:0px 0px 16px;zoom:1;border:0px;margin:0px"><div><div id="gmail-m_-5574547624929765182gmail-:2k4" aria-label="Message Body" role="textbox" aria-multiline="true" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:1.5;font-family:Arial,Helvetica,sans-serif;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;border:0px;overflow:visible hidden;width:904px;outline:none;direction:ltr;min-height:85px"></div></div></div></td></tr></tbody></table><div class="gmail_default" style="font-family:monospace,monospace">Hello,</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Just sending an update on this. The Debian Freeswitch server we set up had the same issue as our Gentoo servers. Looking at the backtrace we noticed the first lowlevellock was occurring when or OurOutbound.luac was being used. We decided to use a version that was not precompiled. After that change the issue has only occurred once more. So there is still a problem we are looking into, but the problems have been massively reduced with that change.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Part of the backtrace</div><div class="gmail_default" style="font-family:monospace,monospace">Thread 40 (Thread 0x7f2fe191e700 (LWP 15884)):<br>#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63<br>#1  0x00007f30512cce53 in __GI__IO_link_in (fp=fp@entry=0x7f2fec53e400) at genops.c:93<br>#2  0x00007f30512cbb92 in _IO_new_file_init_internal (fp=fp@entry=0x7f2fec53e400) at fileops.c:114<br>#3  0x00007f30512c0148 in __fopen_internal (filename=0x7f2fec4d5920 "/usr/local/freeswitch/scripts/360/V3/lua/OurOutbound.luac", mode=0x7f2fe37d81b2 "r", is32=1) at iofopen.c:74<br>#4  0x00007f2fe37cc123 in luaL_loadfilex () from /lib/x86_64-linux-gnu/liblua5.2.so.0<br>#5  0x00007f302405796d in lua_parse_and_execute (L=<optimized out>, input_code=<optimized out>, session=0x7f2fc91217f8) at mod_lua.cpp:194<br>#6  0x00007f3024058191 in lua_function (session=0x7f2fc91217f8, data=<optimized out>) at mod_lua.cpp:483<br>#7  0x00007f30518920df in switch_core_session_exec (session=session@entry=0x7f2fc91217f8, application_interface=application_interface@entry=0x55d5e9528150, arg=arg@entry=0x7f2fc96819c0 "360/V3/lua/OurOutbound.luac") at src/switch_core_session.c:2900<br>#8  0x00007f305189276f in switch_core_session_execute_application_get_flags (session=session@entry=0x7f2fc91217f8, app=0x7f2fc96819b8 "lua", arg=0x7f2fc96819c0 "360/V3/lua/OurOutbound.luac", flags=flags@entry=0x0) at src/switch_core_session.c:2765<br>#9  0x00007f3051896ab2 in switch_core_standard_on_execute (session=0x7f2fc91217f8) at src/switch_core_state_machine.c:354<br>#10 switch_core_session_run (session=0x7f2fc91217f8) at src/switch_core_state_machine.c:651<br>#11 0x00007f305188face in switch_core_session_thread (thread=<optimized out>, obj=0x7f2fc91217f8) at src/switch_core_session.c:1719<br>#12 0x00007f305188b4c3 in switch_core_session_thread_pool_worker (thread=0x7f3034236c70, obj=<optimized out>) at src/switch_core_session.c:1782<br>#13 0x00007f3051bc536c in dummy_worker (opaque=0x7f3034236c70) at threadproc/unix/thread.c:151<br>#14 0x00007f3051793fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486<br>#15 0x00007f30513494cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 9, 2021 at 1:37 AM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">We have set up a Debian 10 FreeSWITCH-1.10.6 server for testing/debugging. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 4, 2021 at 5:51 PM David Villasmil <<a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">That is correct. But we who have worked with FS since around 2002 know this, newer people get all worked up until they’re told this :)</div><div dir="auto"><br></div><div dir="auto">It’s a learning thing..</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 4 Sep 2021 at 17:31, Ken Rice <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">“stuck channels” usually arent stuck once you realize things like “show calls” and “show channels” are selects against the coredb. this usually happens on busy systems timing put waiting for db delete’s to run that fail or get skipped.<br><br><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On Sep 4, 2021, at 05:22, David Villasmil <<a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a>> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"></div></blockquote></div><div dir="auto"><blockquote type="cite"><div dir="ltr"><div dir="auto">Hello all,</div><div dir="auto"><br></div><div dir="auto">Interesting point about Gentoo, but I highly doubt it’s OS related. I’ve been living this issue (about those “stuck channels”, not a frozen FS, that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian, Ubuntu, etc. </div><div dir="auto"><br></div><div dir="auto">In my experience this became an issue with the rise of VM. Back in the day you would see this every now and then, like months apart (this, or a crashed MySQL database), but wasn’t big enough to be a critical problem. Today day it’s basically an everyday thing if you have enough nodes.</div><div dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 4 Sep 2021 at 00:15, Ken Rice <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Having massive experience here, you might try replacing your linux distro.<div><br></div><div> (Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)<div><br></div><div>However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained. </div><div><br></div><div>If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)<br><br>Ken<br><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On Sep 3, 2021, at 14:49, Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"></div></blockquote></div></div></div><div dir="auto"><div><div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace"> We have not been able to reproduce this in testing. It only happens in our production environment.
 Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into.  We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1</pre></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 2, 2021 at 7:24 PM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">And these exhibit the issue on command?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Here are two originate samples.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><a href="https://pastebin.com/YKRKrJLu" target="_blank">https://pastebin.com/YKRKrJLu</a><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br><a href="https://pastebin.com/TEX3yfX6" target="_blank">https://pastebin.com/TEX3yfX6</a><br><br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 1, 2021 at 8:00 PM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Can you try to create one where this happens without all this mess?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default"><font color="#1d1c1d" face="Slack-Lato, appleLogo, sans-serif"><span style="font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.<br>The top part is the common functions, and the 4 sections below are different call scenarios.</span></font></div><div class="gmail_default" style="font-family:monospace,monospace"><div style="box-sizing:inherit;max-width:none;margin-bottom:4px;display:inline;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><div style="box-sizing:inherit;max-width:none;display:inline"><div style="box-sizing:inherit;width:100%;display:inline"><div style="box-sizing:inherit;display:inline"><div dir="auto" style="box-sizing:inherit;width:100%;margin-top:0px;display:inline;line-height:1.46668"><div style="box-sizing:inherit;display:inline"><a href="https://pastebin.com/qqGw3tms" rel="noopener noreferrer" style="box-sizing:inherit" target="_blank">https://pastebin.com/qqGw3tms</a></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 1, 2021 at 3:53 PM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I you do sessions from your lua, I need to see how you do that.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">Yes, it bridges the inbound call to an outbound one.<br><br></pre><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">(ProxyOutbound.luac)
<a href="https://pastebin.com/RDUMy473" rel="noopener noreferrer" style="box-sizing:inherit;color:inherit;text-decoration-line:none" target="_blank">https://pastebin.com/RDUMy473</a></pre></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 31, 2021 at 8:39 PM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Guessing you create sessions in lua? <span style="color:rgb(0,0,0);white-space:pre-wrap">ProxyOutbound.luac?</span></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 31, 2021 at 3:33 PM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Hello,</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">We have got the logs for a few calls that were stuck.</div><div class="gmail_default" style="font-family:monospace,monospace">(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)<br><a href="https://pastebin.com/NTnte7J7" target="_blank">https://pastebin.com/NTnte7J7</a></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)<br><a href="https://pastebin.com/sAhFiGjY" target="_blank">https://pastebin.com/sAhFiGjY</a><br><br>(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)<br><a href="https://pastebin.com/bmcdjzAr" target="_blank">https://pastebin.com/bmcdjzAr</a><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)<br></div><div class="gmail_default" style="font-family:monospace,monospace"><a href="https://pastebin.com/aGyY3m9F" target="_blank">https://pastebin.com/aGyY3m9F</a><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)<br></div><div class="gmail_default" style="font-family:monospace,monospace"><a href="https://pastebin.com/0cnQaRhC" target="_blank">https://pastebin.com/0cnQaRhC</a><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)<br><a href="https://pastebin.com/atX0VXdZ" target="_blank">https://pastebin.com/atX0VXdZ</a><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Please let us know if there is anything else we can provide that would be useful. Thank you.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 30, 2021 at 8:20 PM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Check the UUID of the session in question, and extract its logs, and let's diagnose it</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:monospace,monospace"><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt"> </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">Yes, that is what we are
seeing when running uuid_kill.</span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">  </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">We cannot
"replicate" the issue, but it happens multiple times a day for us
across our servers.</span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">   </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">We have looked
through logs across multiple systems, taken SIP traces to see if that showed
anything.</span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">  </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">Last we got a Core dump of the
running process after the failure. Nothing seems to stand out as the cause.</span></div>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"> </span></p>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"> Are there any other debug
steps you can think of or what other information to look at?  We could capture all traffic on the server,
or recompile with special debug flags whatever it is that might help. </span></p>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"> </span></p>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">Steps we have taken:</span></p>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">sofia loglevel all 9 </span></p>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">sofia global siptrace on</span></p>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">fsctl loglevel 7</span></p>

<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"># Set environment variable</span></p>

<div style="font-family:monospace,monospace"><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">TPORT_LOG=1</span></div><div style="font-family:monospace,monospace"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 28, 2021 at 12:54 AM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.<div><br></div><div>Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID</div><div><br></div><div>/b</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:monospace,monospace"> We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.<br> We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.<br> We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  <br>Some general information:<br>FreeSWITCH Versions 1.10.6 and 1.10.1<br>Modules used:<br>LUA, xml_curl(dialplan), event_socket,<br><br> Most of this is just application playing audio files and or creating a new call to bridge to.<br><br> Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.<br><br> We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.<br><br>Thank you for your help.<br></div></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
<span>_________________________________________________________________________</span><br><span></span><br><span>The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" target="_blank">https://signalwire.com</a></span><br><span>Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.</span><br><span>Build your next product on our scalable cloud platform.</span><br><span></span><br><span>Join our online community to chat in real time <a href="https://signalwire.community" target="_blank">https://signalwire.community</a></span><br><span></span><br><span>Professional FreeSWITCH Services</span><br><span><a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a></span><br><span><a href="https://freeswitch.com" target="_blank">https://freeswitch.com</a></span><br><span></span><br><span>Official FreeSWITCH Sites</span><br><span><a href="https://freeswitch.com/oss" target="_blank">https://freeswitch.com/oss</a></span><br><span><a href="https://freeswitch.org/confluence" target="_blank">https://freeswitch.org/confluence</a></span><br><span><a href="https://cluecon.com" target="_blank">https://cluecon.com</a></span><br><span></span><br><span>FreeSWITCH-users mailing list</span><br><span><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a></span><br><span><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a></span><br><span>UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a></span><br><span><a href="https://freeswitch.com" target="_blank">https://freeswitch.com</a></span></div></blockquote></div></div></div>_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div></div>-- <br><div dir="ltr"><div dir="ltr"><div>Regards,</div><div><br></div>David Villasmil<div>email: <a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a></div><div>phone: +34669448337</div></div></div>
<span>_________________________________________________________________________</span><br><span></span><br><span>The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" target="_blank">https://signalwire.com</a></span><br><span>Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.</span><br><span>Build your next product on our scalable cloud platform.</span><br><span></span><br><span>Join our online community to chat in real time <a href="https://signalwire.community" target="_blank">https://signalwire.community</a></span><br><span></span><br><span>Professional FreeSWITCH Services</span><br><span><a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a></span><br><span><a href="https://freeswitch.com" target="_blank">https://freeswitch.com</a></span><br><span></span><br><span>Official FreeSWITCH Sites</span><br><span><a href="https://freeswitch.com/oss" target="_blank">https://freeswitch.com/oss</a></span><br><span><a href="https://freeswitch.org/confluence" target="_blank">https://freeswitch.org/confluence</a></span><br><span><a href="https://cluecon.com" target="_blank">https://cluecon.com</a></span><br><span></span><br><span>FreeSWITCH-users mailing list</span><br><span><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a></span><br><span><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a></span><br><span>UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a></span><br><span><a href="https://freeswitch.com" target="_blank">https://freeswitch.com</a></span></div></blockquote></div>_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div></div>-- <br><div dir="ltr"><div dir="ltr"><div>Regards,</div><div><br></div>David Villasmil<div>email: <a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a></div><div>phone: +34669448337</div></div></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div>
</blockquote></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" src="https://lh6.googleusercontent.com/AYfRoSNaDNtMPRMevPn_GqcVEMd5NDRFi0GlluGUWzV6I5TAY_3T2-Tt0IuIXeUtEdYsgNsM8DOYKRKhjmrG_-n2Ga-LCnoNk46sO8VyEma1sBFYdiGJcLRUvkrD1CYHN79qimeg" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" src="https://lh3.googleusercontent.com/W4SqXyybH2qdAozvtoKjcz736qOjk9LHDwldvs1ahc-WVU0putVMSsUH474KDrJ32jsqi6JDjyUWxqeEkN5I1xSlC5ShYrd1b8NIMUkDzDrtbWQfa6A_90UcygqesBtRLgeFirKa" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>