[Freeswitch-users] Sofia freezing for 5 minutes then starting again

Abaci abaci64 at gmail.com
Sun Nov 25 22:26:54 MSK 2012


See also http://jira.freeswitch.org/browse/FS-3328

On 11/25/2012 1:46 PM, Abaci wrote:
> I just remember reading somewhere on a blog something about a similar 
> issue and this was the cause.
> Just found that post 
> http://blog.godson.in/2011/06/pitfalls-to-avoid-while-using.html not 
> sure that's the issue. either way it's a good idea to set the timeout 
> option.
>
> On 11/25/2012 11:32 AM, Ken Rice wrote:
>> Sofia is not single threaded except for in one spot deep in libsofia,
>> >From there, messages are handed off to a number of message queues 
>> for FS
>> core to handle as needed...
>>
>> Check to see if anything that fs is depending on is blocking on info
>> retrieval like the databases or other areas...
>>
>> K
>>
>> On 11/25/12 9:51 AM, "Abaci" <abaci64 at gmail.com> wrote:
>>
>>> you mentioned that you use xml_curl, if your web server hangs it may
>>> hang sofia, iirc sofia is running a single thread and it will wait for
>>> the xml_curl response before continuing to the next request.
>>>
>>> On 11/23/2012 1:13 PM, Steven Ayre wrote:
>>>> Any kind of DB backup running? Or any long-running queries (innotop is
>>>> great for highlighting queries that've been running a while, including
>>>> on non-innodb tables).
>>>>
>>>> A global read lock, or queries waiting for a lock could block a db
>>>> update from the sofia profile thread but still allow read-only queries
>>>> (sofia status) to run.
>>>>
>>>> -Steve
>>>>
>>>>
>>>>
>>>> On 23 November 2012 15:32, Tim St. Pierre 
>>>> <fs-list at communicatefreely.net>
>>>> wrote:
>>>>> Hi Steven,
>>>>>
>>>>> Thanks for the suggestions.  I'm hoping once I get the upgrade 
>>>>> done it will
>>>>> all go away.
>>>>> I have watched it happen at least once.  I was on the phone at the 
>>>>> time.
>>>>> Console activity
>>>>> more or less stopped, except for a few calls hanging up. The console
>>>>> remains responsive,
>>>>> and my call wasn't dropped for at least a minute or two (media 
>>>>> timeout?).  I
>>>>> was able to
>>>>> run sofia status and other commands that use the database, so I'm 
>>>>> assuming
>>>>> that the
>>>>> connection was still working.  All our media is runs through the 
>>>>> box, so I
>>>>> think things
>>>>> are fine on the Ethernet level.  I do see higher load averages - 
>>>>> maybe 3-4,
>>>>> but that's the
>>>>> only obvious indication.  It's not taking CPU beyond 10% or so.
>>>>>
>>>>> We are using MySQL as the core DB and also as the DB backend for 
>>>>> each sofia
>>>>> profile.  This
>>>>> is connecting through ODBC of course.
>>>>>
>>>>> If I can get the other kinks worked out, then I will try 1.2 
>>>>> stable in
>>>>> production and
>>>>> we'll see how it goes.
>>>>>
>>>>> -Tim
>>>>>
>>>>> Steven Ayre wrote:
>>>>>>       Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>>>>
>>>>>>
>>>>>> As you've already acknowledged it's a very old version.
>>>>>>
>>>>>> It's possible that your issue has already been found and fixed, 
>>>>>> but if
>>>>>> it hasn't then the code will have changed significantly since 
>>>>>> then and
>>>>>> you'd really need to reproduce it on the latest code for it to be
>>>>>> investigated.
>>>>>>
>>>>>>
>>>>>> As some general thoughts though, are you able to spot it 
>>>>>> happening while
>>>>>> it's happening or only afterwards?
>>>>>>
>>>>>> If you're able to get on the system during one of those times 
>>>>>> look at
>>>>>> what else is happening. Is the load average/cpu usage/io high? 
>>>>>> Perhaps
>>>>>> something's running that's blocking all access or causing very 
>>>>>> high IO.
>>>>>>
>>>>>> What DB backend are you using for Sofia? Is it possible that that's
>>>>>> hanging for a moment? For example if you're running a backup on 
>>>>>> the DB
>>>>>> that blocks all writes to the DB while Sofia is trying to update 
>>>>>> the DB
>>>>>> that perhaps would cause this.
>>>>>>
>>>>>> Try running a SIP OPTIONS ping your your sofia profile from the
>>>>>> localhost during that time, which should exclude it being any 
>>>>>> issue on
>>>>>> the ethernet.
>>>>>>
>>>>>> -Steve
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 22 November 2012 19:31, Tim St. Pierre 
>>>>>> <fs-list at communicatefreely.net
>>>>>> <mailto:fs-list at communicatefreely.net>> wrote:
>>>>>>
>>>>>>       Hello,
>>>>>>
>>>>>>       I'm having a bit of an odd problem.
>>>>>>
>>>>>>       Intermittently, often every 2-3 days or so, Freeswitch stops
>>>>>>       replying to SIP for about 5
>>>>>>       minutes.  I can't verify if it's EXACTLY 5 minutes, but it 
>>>>>> seems to
>>>>>>       be pretty close.
>>>>>>
>>>>>>       During this time, no new registrations or invites can 
>>>>>> happen, but
>>>>>>       existing calls stay
>>>>>>       connected for at least a minute or two.  In the logs, you 
>>>>>> can see
>>>>>>       calls slowly hanging up
>>>>>>       with "NORMAL_CLEARING".  In 5 minutes, everything starts up 
>>>>>> again
>>>>>>       with no word about it at
>>>>>>       all in the logs.
>>>>>>
>>>>>>       When calls resume, I notice that the number of sessions 
>>>>>> returned by
>>>>>>       the status command is
>>>>>>       one higher than the actual number sessions returned by show
>>>>>>       channels, or by looking in the
>>>>>>       database.  Every time this happens, the discrepancy 
>>>>>> increases by one.
>>>>>>
>>>>>>       The interruption happens on all SIP profiles, but calls 
>>>>>> originated
>>>>>>       from the socket API
>>>>>>       still work, insofar as they return with PROGRESS_TIMEOUT 
>>>>>> since the
>>>>>>       profiles are still
>>>>>>       running, but stuck.
>>>>>>
>>>>>>       We are using ODBC/MySQL for the core database, and the 
>>>>>> database
>>>>>>       server only runs this
>>>>>>       database and some basic PHP/xml-curl stuff.
>>>>>>
>>>>>>       We have 416 endpoints registered, and usually sit at about 30
>>>>>>       sessions during the day.
>>>>>>
>>>>>>       This never happens at night, only during busier times, but not
>>>>>>       necessarily busy hour.
>>>>>>
>>>>>>       I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G 
>>>>>> ram)
>>>>>>
>>>>>>       Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>>>>
>>>>>>       Yes, I know it's old and I'm trying to upgrade, but I'm 
>>>>>> still having
>>>>>>       some problems getting
>>>>>>       all my phones to work properly with 1.2 stable. This is a
>>>>>>       production system, so I can't
>>>>>>       just blindly put out the newest release.  Mostly, I need to 
>>>>>> buy
>>>>>>       myself some time so that I
>>>>>>       can get the kinks worked out of the latest version and then 
>>>>>> upgrade
>>>>>>       the production box.
>>>>>>
>>>>>>       I'm grateful for any insights as to what could be 
>>>>>> happening, even if
>>>>>>       a solution is just a
>>>>>>       temporary workaround.
>>>>>>
>>>>>>       Thanks!
>>>>>>
>>>>>>       -Tim
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>>       Professional FreeSWITCH Consulting Services:
>>>>>>       consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>>>       http://www.freeswitchsolutions.com
>>>>>>
>>>>>>       
>>>>>>       
>>>>>>
>>>>>>       Official FreeSWITCH Sites
>>>>>>       http://www.freeswitch.org
>>>>>>       http://wiki.freeswitch.org
>>>>>>       http://www.cluecon.com
>>>>>>
>>>>>>       FreeSWITCH-users mailing list
>>>>>>       FreeSWITCH-users at lists.freeswitch.org
>>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users 
>>>>>>
>>>>>>       http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------ 
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________ 
>>>>>>
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users 
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>> _________________________________________________________________________ 
>>>>>
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users 
>>>>>
>>>>> http://www.freeswitch.org
>>>> _________________________________________________________________________ 
>>>>
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users 
>>>>
>>>> http://www.freeswitch.org
>>>
>>> _________________________________________________________________________ 
>>>
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users 
>>>
>>> http://www.freeswitch.org
>




Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list