[Freeswitch-users] Sofia freezing for 5 minutes then starting again

Ken Rice krice at freeswitch.org
Sun Nov 25 19:32:44 MSK 2012 

Sofia is not single threaded except for in one spot deep in libsofia,
>From there, messages are handed off to a number of message queues for FS
core to handle as needed...

Check to see if anything that fs is depending on is blocking on info
retrieval like the databases or other areas...

K

On 11/25/12 9:51 AM, "Abaci" <abaci64 at gmail.com> wrote:

> you mentioned that you use xml_curl, if your web server hangs it may
> hang sofia, iirc sofia is running a single thread and it will wait for
> the xml_curl response before continuing to the next request.
> 
> On 11/23/2012 1:13 PM, Steven Ayre wrote:
>> Any kind of DB backup running? Or any long-running queries (innotop is
>> great for highlighting queries that've been running a while, including
>> on non-innodb tables).
>> 
>> A global read lock, or queries waiting for a lock could block a db
>> update from the sofia profile thread but still allow read-only queries
>> (sofia status) to run.
>> 
>> -Steve
>> 
>> 
>> 
>> On 23 November 2012 15:32, Tim St. Pierre <fs-list at communicatefreely.net>
>> wrote:
>>> Hi Steven,
>>> 
>>> Thanks for the suggestions.  I'm hoping once I get the upgrade done it will
>>> all go away.
>>> I have watched it happen at least once.  I was on the phone at the time.
>>> Console activity
>>> more or less stopped, except for a few calls hanging up.  The console
>>> remains responsive,
>>> and my call wasn't dropped for at least a minute or two (media timeout?).  I
>>> was able to
>>> run sofia status and other commands that use the database, so I'm assuming
>>> that the
>>> connection was still working.  All our media is runs through the box, so I
>>> think things
>>> are fine on the Ethernet level.  I do see higher load averages - maybe 3-4,
>>> but that's the
>>> only obvious indication.  It's not taking CPU beyond 10% or so.
>>> 
>>> We are using MySQL as the core DB and also as the DB backend for each sofia
>>> profile.  This
>>> is connecting through ODBC of course.
>>> 
>>> If I can get the other kinks worked out, then I will try 1.2 stable in
>>> production and
>>> we'll see how it goes.
>>> 
>>> -Tim
>>> 
>>> Steven Ayre wrote:
>>>>      Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>> 
>>>> 
>>>> As you've already acknowledged it's a very old version.
>>>> 
>>>> It's possible that your issue has already been found and fixed, but if
>>>> it hasn't then the code will have changed significantly since then and
>>>> you'd really need to reproduce it on the latest code for it to be
>>>> investigated.
>>>> 
>>>> 
>>>> As some general thoughts though, are you able to spot it happening while
>>>> it's happening or only afterwards?
>>>> 
>>>> If you're able to get on the system during one of those times look at
>>>> what else is happening. Is the load average/cpu usage/io high? Perhaps
>>>> something's running that's blocking all access or causing very high IO.
>>>> 
>>>> What DB backend are you using for Sofia? Is it possible that that's
>>>> hanging for a moment? For example if you're running a backup on the DB
>>>> that blocks all writes to the DB while Sofia is trying to update the DB
>>>> that perhaps would cause this.
>>>> 
>>>> Try running a SIP OPTIONS ping your your sofia profile from the
>>>> localhost during that time, which should exclude it being any issue on
>>>> the ethernet.
>>>> 
>>>> -Steve
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 22 November 2012 19:31, Tim St. Pierre <fs-list at communicatefreely.net
>>>> <mailto:fs-list at communicatefreely.net>> wrote:
>>>> 
>>>>      Hello,
>>>> 
>>>>      I'm having a bit of an odd problem.
>>>> 
>>>>      Intermittently, often every 2-3 days or so, Freeswitch stops
>>>>      replying to SIP for about 5
>>>>      minutes.  I can't verify if it's EXACTLY 5 minutes, but it seems to
>>>>      be pretty close.
>>>> 
>>>>      During this time, no new registrations or invites can happen, but
>>>>      existing calls stay
>>>>      connected for at least a minute or two.  In the logs, you can see
>>>>      calls slowly hanging up
>>>>      with "NORMAL_CLEARING".  In 5 minutes, everything starts up again
>>>>      with no word about it at
>>>>      all in the logs.
>>>> 
>>>>      When calls resume, I notice that the number of sessions returned by
>>>>      the status command is
>>>>      one higher than the actual number sessions returned by show
>>>>      channels, or by looking in the
>>>>      database.  Every time this happens, the discrepancy increases by one.
>>>> 
>>>>      The interruption happens on all SIP profiles, but calls originated
>>>>      from the socket API
>>>>      still work, insofar as they return with PROGRESS_TIMEOUT since the
>>>>      profiles are still
>>>>      running, but stuck.
>>>> 
>>>>      We are using ODBC/MySQL for the core database, and the database
>>>>      server only runs this
>>>>      database and some basic PHP/xml-curl stuff.
>>>> 
>>>>      We have 416 endpoints registered, and usually sit at about 30
>>>>      sessions during the day.
>>>> 
>>>>      This never happens at night, only during busier times, but not
>>>>      necessarily busy hour.
>>>> 
>>>>      I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
>>>> 
>>>>      Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>> 
>>>>      Yes, I know it's old and I'm trying to upgrade, but I'm still having
>>>>      some problems getting
>>>>      all my phones to work properly with 1.2 stable.  This is a
>>>>      production system, so I can't
>>>>      just blindly put out the newest release.  Mostly, I need to buy
>>>>      myself some time so that I
>>>>      can get the kinks worked out of the latest version and then upgrade
>>>>      the production box.
>>>> 
>>>>      I'm grateful for any insights as to what could be happening, even if
>>>>      a solution is just a
>>>>      temporary workaround.
>>>> 
>>>>      Thanks!
>>>> 
>>>>      -Tim
>>>> 
>>>>      
>>>> _________________________________________________________________________
>>>>      Professional FreeSWITCH Consulting Services:
>>>>      consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>      http://www.freeswitchsolutions.com
>>>> 
>>>>      
>>>>      
>>>> 
>>>>      Official FreeSWITCH Sites
>>>>      http://www.freeswitch.org
>>>>      http://wiki.freeswitch.org
>>>>      http://www.cluecon.com
>>>> 
>>>>      FreeSWITCH-users mailing list
>>>>      FreeSWITCH-users at lists.freeswitch.org
>>>>      <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>      http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>      
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>      http://www.freeswitch.org
>>>> 
>>>> 
>>>> 
>>>> ------------------------------------------------------------------------
>>>> 
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>> 
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>> 
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>> 
>>> 
>>> 
>>> 
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>> 
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>> 
>> 
>> 
>> 
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>> 
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
> 
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-- 
Ken
http://www.FreeSWITCH.org
http://www.ClueCon.com
http://www.OSTAG.org
irc.freenode.net #freeswitch

Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list