[Freeswitch-users] ESL connection lost to FS

Tue May 7 12:31:54 UTC 2019

We were able to isolate this issue a bit and found:

   - This issue persists across FS versions, so it doesn't seem to be a
   v1.8 issue
   - ESL client in fact does return a value each time; it's not that it
   "drops". The issue is that with a minority of calls the originate command
   never reaches FS in the sense that we see no trace of it in logs.
   - The number of calls affected seems to depend on the amount of
   parallelism we hit FS with. It's <1% of calls if we send up to 10 of calls
   per second for an entire day totalling in couple hundred thousand calls.
   But it goes as high as 15% if we send 100+ calls per second.
   - We've tried FS inbuilt pyESL lib (both built from source and package),
   as well as greenswitch. The issue persists across ESL clients so we don't
   feel it's a client related issue.

Any suggestions on how to debug further?

On Tue, Apr 23, 2019 at 10:12 AM Neil Patel <neilp at cs.stanford.edu> wrote:

> Deepika,
>
> I don't think this is the issue as we've had this pattern of code for
> years and only recently (after upgrading to v1.8) have we seen ESL
> connection instability. Note we took this pattern from Newfies which does
> exact same thing
> <https://github.com/newfies-dialer/newfies-dialer/blob/8168b3dd43e9f5ce73a2645b3229def1b2815d47/lua/dial.py#L31>.
> Areski hasn't responded yet but I believe he would say this is not an issue.
>
> We've adjusted our code to abort the attempt and retry if ev=None, but
> there remains a portion of calls for which ev returns success but there is
> actually no call originated and no trace of it in FS logs or in CDR.
>
> Any other ideas?
>
>
> On Mon, Apr 22, 2019 at 9:51 PM Deepika Yadav <deepikay at iiitd.ac.in>
> wrote:
>
>> Hi Neil,
>>
>> Just a guess, shouldn't you put a significant sleep before
>> "c.disconnect()". Since the previous command starts background api, it
>> would be appropriate to introduce some time gap before disconnecting ESL.
>>
>> Regards,
>> Deepika
>> PhD Scholar
>> IIIT Delhi <http://iiitd.ac.in/>, India
>> Webpage: https://deepikay.wixsite.com/deepika
>>
>> On Mon, Apr 22, 2019 at 9:37 PM Neil Patel <neilp at cs.stanford.edu> wrote:
>>
>>> Hi Brian,
>>>
>>> Sorry, I didn't give enough context.
>>>
>>> The code below is for the origination of calls, which we are doing
>>> through ESL and where the connection is being unexpectedly dropped, or even
>>> if not dropped, no record of the origination appears in either FS logs or
>>> CDR.
>>>
>>> CDRs are being generated to sqlite and we are reading them from there
>>> asynchronously.
>>>
>>> Also forgot to link the documentation that suggests building from source
>>> is the way, it's here
>>> <https://freeswitch.org/confluence/display/FREESWITCH/Python+ESL>
>>>
>>> On Mon, Apr 22, 2019 at 6:25 PM Brian West <brian at freeswitch.com> wrote:
>>>
>>>> You're doing the CDR collection inline or what are you speaking of
>>>> exactly?
>>>>
>>>> On Mon, Apr 22, 2019 at 2:29 AM Neil Patel <neilp at cs.stanford.edu>
>>>> wrote:
>>>>
>>>>> Hi all, I have a fresh server running FS 1.8 from package on Debian 9.
>>>>> It runs a python app connecting to FS using ESL that executes this code per
>>>>> call attempt:
>>>>>
>>>>> c = ESLconnection(str(settings.ESL_HOSTNAME), str(settings.ESL_PORT),
>>>>> str(settings.ESL_PASSWORD))
>>>>> if not c.connected():
>>>>>     abort()
>>>>> dial_command = "originate …"
>>>>> ev = c.api("bgapi", str(dial_command))
>>>>> c.disconnect()
>>>>> if ev:
>>>>>     result = ev.serialize()
>>>>>     logger.debug('API result = {}'.format(result))
>>>>>
>>>>>
>>>>> When we ran 350K call test, 70K calls got to this code and failed the
>>>>> final if condition, meaning ev was None. No CDR generated.
>>>>>
>>>>> This is totally an unexpected error case for us after tens of millions
>>>>> of calls over last 3 years on our other servers. Any idea why this is
>>>>> happening? The only difference I can see is we’ve not seen this issue on FS
>>>>> 1.6 (built from source), so maybe it’s introduced in 1.8 (run from
>>>>> package)? FYI we've built pyESL from source, which is what's recommended
>>>>> here. Any chance there is an incompatibility between pyESL built from
>>>>> source and FS 1.8 installed from package?
>>>>>
>>>>> Thanks,
>>>>> Neil
>>>>>
>>>>> This question reproduced from Slack
>>>>> <https://signalwire-community.slack.com/archives/CDC8H14PN/p1555508784091900>
>>>>> where I didn't get a response
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>>
>>>>> The FreeSWITCH project is sponsored by SignalWire
>>>>> https://signalwire.com
>>>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>>>> services.
>>>>> Build your next product on our scalable cloud platform.
>>>>>
>>>>> Join our online community to chat in real time
>>>>> https://signalwire.community
>>>>>
>>>>> Professional FreeSWITCH Services
>>>>> sales at freeswitch.com
>>>>> https://freeswitch.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> https://freeswitch.com/oss
>>>>> https://freeswitch.org/confluence
>>>>> https://cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> https://freeswitch.com
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Brian West | Co-founder and Developer
>>>>
>>>> Need Commercial support? email sales at freeswitch.com
>>>>
>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>
>>>> Email: brian at freeswitch.com
>>>>
>>>> Mobile: 918-424-9378
>>>>
>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>
>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>> <https://www.facebook.com/freeswitch> [image:
>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>
>>>> _________________________________________________________________________
>>>>
>>>> The FreeSWITCH project is sponsored by SignalWire
>>>> https://signalwire.com
>>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>>> services.
>>>> Build your next product on our scalable cloud platform.
>>>>
>>>> Join our online community to chat in real time
>>>> https://signalwire.community
>>>>
>>>> Professional FreeSWITCH Services
>>>> sales at freeswitch.com
>>>> https://freeswitch.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> https://freeswitch.com/oss
>>>> https://freeswitch.org/confluence
>>>> https://cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> https://freeswitch.com
>>>
>>> _________________________________________________________________________
>>>
>>> The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
>>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>>> services.
>>> Build your next product on our scalable cloud platform.
>>>
>>> Join our online community to chat in real time
>>> https://signalwire.community
>>>
>>> Professional FreeSWITCH Services
>>> sales at freeswitch.com
>>> https://freeswitch.com
>>>
>>> Official FreeSWITCH Sites
>>> https://freeswitch.com/oss
>>> https://freeswitch.org/confluence
>>> https://cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> https://freeswitch.com
>>
>>
>>
>> --
>> Regards,
>> Deepika
>> PhD Scholar
>> IIIT Delhi <http://iiitd.ac.in/>, India
>> Webpage: https://deepikay.wixsite.com/deepika
>> _________________________________________________________________________
>>
>> The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
>> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
>> services.
>> Build your next product on our scalable cloud platform.
>>
>> Join our online community to chat in real time
>> https://signalwire.community
>>
>> Professional FreeSWITCH Services
>> sales at freeswitch.com
>> https://freeswitch.com
>>
>> Official FreeSWITCH Sites
>> https://freeswitch.com/oss
>> https://freeswitch.org/confluence
>> https://cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> https://freeswitch.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20190507/630325c2/attachment-0001.html>