<div dir="ltr">We were able to isolate this issue a bit and found:<div><ul><li>This issue persists across FS versions, so it doesn't seem to be a v1.8 issue</li><li>ESL client in fact does return a value each time; it's not that it "drops". The issue is that with a minority of calls the originate command never reaches FS in the sense that we see no trace of it in logs.</li><li>The number of calls affected seems to depend on the amount of parallelism we hit FS with. It's <1% of calls if we send up to 10 of calls per second for an entire day totalling in couple hundred thousand calls. But it goes as high as 15% if we send 100+ calls per second.</li><li>We've tried FS inbuilt pyESL lib (both built from source and package), as well as greenswitch. The issue persists across ESL clients so we don't feel it's a client related issue.</li></ul><div>Any suggestions on how to debug further?</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 23, 2019 at 10:12 AM Neil Patel <<a href="mailto:neilp@cs.stanford.edu" target="_blank">neilp@cs.stanford.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Deepika,<div><br></div><div>I don't think this is the issue as we've had this pattern of code for years and only recently (after upgrading to v1.8) have we seen ESL connection instability. Note we took this pattern from Newfies <a href="https://github.com/newfies-dialer/newfies-dialer/blob/8168b3dd43e9f5ce73a2645b3229def1b2815d47/lua/dial.py#L31" target="_blank">which does exact same thing</a>. Areski hasn't responded yet but I believe he would say this is not an issue.</div><div><br></div><div>We've adjusted our code to abort the attempt and retry if ev=None, but there remains a portion of calls for which ev returns success but there is actually no call originated and no trace of it in FS logs or in CDR.</div><div><br></div><div>Any other ideas?</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 22, 2019 at 9:51 PM Deepika Yadav <<a href="mailto:deepikay@iiitd.ac.in" target="_blank">deepikay@iiitd.ac.in</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Neil, <div><br></div><div>Just a guess, shouldn't you put a significant sleep before "c.disconnect()". Since the previous command starts background api, it would be appropriate to introduce some time gap before disconnecting ESL.</div><div><br></div><div>Regards,</div><div><div>Deepika</div><div>PhD Scholar</div><div><a href="http://iiitd.ac.in/" target="_blank">IIIT Delhi</a>, India</div><div>Webpage: <a href="https://deepikay.wixsite.com/deepika" target="_blank">https://deepikay.wixsite.com/deepika</a></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 22, 2019 at 9:37 PM Neil Patel <<a href="mailto:neilp@cs.stanford.edu" target="_blank">neilp@cs.stanford.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Brian,<div><br></div><div>Sorry, I didn't give enough context.<div><br></div><div>The code below is for the origination of calls, which we are doing through ESL and where the connection is being unexpectedly dropped, or even if not dropped, no record of the origination appears in either FS logs or CDR. </div><div><br></div><div>CDRs are being generated to sqlite and we are reading them from there asynchronously.</div><div><br></div><div>Also forgot to link the documentation that suggests building from source is the way, <a href="https://freeswitch.org/confluence/display/FREESWITCH/Python+ESL" target="_blank">it's here</a></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 22, 2019 at 6:25 PM Brian West <<a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You're doing the CDR collection inline or what are you speaking of exactly?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 22, 2019 at 2:29 AM Neil Patel <<a href="mailto:neilp@cs.stanford.edu" target="_blank">neilp@cs.stanford.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hi all, I have a fresh server running FS 1.8 from package on Debian 9. It runs a python app connecting to FS using ESL that executes this code per call attempt:</div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>c = ESLconnection(str(settings.ESL_HOSTNAME), str(settings.ESL_PORT), str(settings.ESL_PASSWORD))</div></div><div><div>if not c.connected():</div></div><div><div> abort()</div></div><div><div>dial_command = "originate …"</div></div><div><div>ev = c.api("bgapi", str(dial_command))</div></div><div><div>c.disconnect()</div></div><div><div>if ev:</div></div><div><div> result = ev.serialize()</div></div><div><div> logger.debug('API result = {}'.format(result))</div></div></blockquote><div dir="ltr"><div><br></div><div>When we ran 350K call test, 70K calls got to this code and failed the final if condition, meaning ev was None. No CDR generated.</div><div><br></div><div>This is totally an unexpected error case for us after tens of millions of calls over last 3 years on our other servers. Any idea why this is happening? The only difference I can see is we’ve not seen this issue on FS 1.6 (built from source), so maybe it’s introduced in 1.8 (run from package)? FYI we've built pyESL from source, which is what's recommended here. Any chance there is an incompatibility between pyESL built from source and FS 1.8 installed from package?</div><div><br></div><div>Thanks,</div><div>Neil</div><div><br></div><div>This question <a href="https://signalwire-community.slack.com/archives/CDC8H14PN/p1555508784091900" target="_blank">reproduced from Slack</a> where I didn't get a response</div><div><br></div></div></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_4289240009111212867gmail-m_5062451601498149054gmail-m_-877056013464699165gmail-m_-7376309891724622588gmail-m_-100448535362159612gmail-m_8133051631785743498gmail-m_-3113894553185291788gmail-m_8978286192596984981gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><font color="#000000"><br></font></div><div style="font-size:12.8px"><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Brian West | Co-founder and Developer</font></span></p><p style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Need Commercial support? email <a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a> </font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">FreeSWITCH Solutions | <a href="https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">17345 Civic Drive #2531 Brookfield, WI 53045</a></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Email: </span><span style="color:rgb(17,85,204);font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:brian@freeswitch.com" target="_blank">brian@freeswitch.com</a></span></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#000000">Mobile: <span title="Call with Google Voice">918-424-9378</span></font></span></p><p dir="ltr" style="font-size:12.8px;line-height:1.38;margin-top:0pt;margin-bottom:0pt;margin-left:4.5pt"><font color="#000000"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website: </span><a href="https://www.freeswitch.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:8pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">https://www.FreeSWITCH.com</span></a></font></p><p dir="ltr" style="font-size:12.8px;line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;font-size:9pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://www.facebook.com/freeswitch" target="_blank"><img alt="https://www.facebook.com/signalwireinc?src=email" src="https://lh6.googleusercontent.com/AYfRoSNaDNtMPRMevPn_GqcVEMd5NDRFi0GlluGUWzV6I5TAY_3T2-Tt0IuIXeUtEdYsgNsM8DOYKRKhjmrG_-n2Ga-LCnoNk46sO8VyEma1sBFYdiGJcLRUvkrD1CYHN79qimeg" width="31" height="31" style="border: none;"></a> </span><span style="background-color:transparent;font-size:9pt;font-family:Tahoma;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap"><a href="https://twitter.com/freeswitch" target="_blank"><img alt="https://twitter.com/freeswitch" src="https://lh3.googleusercontent.com/W4SqXyybH2qdAozvtoKjcz736qOjk9LHDwldvs1ahc-WVU0putVMSsUH474KDrJ32jsqi6JDjyUWxqeEkN5I1xSlC5ShYrd1b8NIMUkDzDrtbWQfa6A_90UcygqesBtRLgeFirKa" width="31" height="31" style="border: none;"></a></span><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_4289240009111212867gmail-m_5062451601498149054gmail-m_-877056013464699165gmail-m_-7376309891724622588gmail-m_-100448535362159612gmail_signature"><div dir="ltr"><div><div dir="ltr">Regards,<div>Deepika</div><div>PhD Scholar</div><div><a href="http://iiitd.ac.in/" target="_blank">IIIT Delhi</a>, India</div><div>Webpage: <a href="https://deepikay.wixsite.com/deepika" target="_blank">https://deepikay.wixsite.com/deepika</a><br></div></div></div></div></div>
_________________________________________________________________________<br>
<br>
The FreeSWITCH project is sponsored by SignalWire <a href="https://signalwire.com" rel="noreferrer" target="_blank">https://signalwire.com</a><br>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.<br>
Build your next product on our scalable cloud platform.<br>
<br>
Join our online community to chat in real time <a href="https://signalwire.community" rel="noreferrer" target="_blank">https://signalwire.community</a><br>
<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a></blockquote></div>
</blockquote></div>