<div dir="ltr">Check the UUID of the session in question, and extract its logs, and let's diagnose it</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <<a href="mailto:cary@ytel.com">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt"> </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">Yes, that is what we are
seeing when running uuid_kill.</span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt"> </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">We cannot
"replicate" the issue, but it happens multiple times a day for us
across our servers.</span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt"> </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">We have looked
through logs across multiple systems, taken SIP traces to see if that showed
anything.</span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt"> </span><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">Last we got a Core dump of the
running process after the failure. Nothing seems to stand out as the cause.</span></div>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"> </span></p>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"> Are there any other debug
steps you can think of or what other information to look at? We could capture all traffic on the server,
or recompile with special debug flags whatever it is that might help. </span></p>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"> </span></p>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">Steps we have taken:</span></p>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">sofia loglevel all 9 </span></p>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">sofia global siptrace on</span></p>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)">fsctl loglevel 7</span></p>
<p class="MsoNormal" style="margin:3pt 0in;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:9pt;font-family:"Courier New";color:rgb(29,28,29)"># Set environment variable</span></p>
<div class="gmail_default" style="font-family:monospace,monospace"><span style="color:rgb(29,28,29);font-family:"Courier New";font-size:9pt">TPORT_LOG=1</span></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 28, 2021 at 12:54 AM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.<div><br></div><div>Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID</div><div><br></div><div>/b</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <<a href="mailto:cary@ytel.com" target="_blank">cary@ytel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"> We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.<br> We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.<br> We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA. <br>Some general information:<br>FreeSWITCH Versions 1.10.6 and 1.10.1<br>Modules used:<br>LUA, xml_curl(dialplan), event_socket,<br><br> Most of this is just application playing audio files and or creating a new call to bridge to.<br><br> Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.<br><br> We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.<br><br>Thank you for your help.<br></div></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" src="https://lh6.googleusercontent.com/AYfRoSNaDNtMPRMevPn_GqcVEMd5NDRFi0GlluGUWzV6I5TAY_3T2-Tt0IuIXeUtEdYsgNsM8DOYKRKhjmrG_-n2Ga-LCnoNk46sO8VyEma1sBFYdiGJcLRUvkrD1CYHN79qimeg" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" src="https://lh3.googleusercontent.com/W4SqXyybH2qdAozvtoKjcz736qOjk9LHDwldvs1ahc-WVU0putVMSsUH474KDrJ32jsqi6JDjyUWxqeEkN5I1xSlC5ShYrd1b8NIMUkDzDrtbWQfa6A_90UcygqesBtRLgeFirKa" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>