[Freeswitch-users] Sofia freezing for 5 minutes then starting again
Tim St. Pierre
fs-list at communicatefreely.net
Tue Nov 27 05:25:30 MSK 2012
Thanks for all the input everyone!
I just had it happen again, this time at night when there were only 2 sessions active!
I was in doing some other work, so I managed to see it happen and found a few interesting
things.
CPU load next to nothing, load average around 0.12.
I used iSQL to test ODBC connectivity, and I could read and write to the freeswitch
database using the same DSN that freeswitch is using.
I could delete a SIP registration from the API
Since I had things on a higher debug level, I saw this:
freeswitch at stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG Thread Started
As soon as that came up, everything started working again.
What's that about?
-Tim
Ken Rice wrote:
> Sofia is not single threaded except for in one spot deep in libsofia,
>>From there, messages are handed off to a number of message queues for FS
> core to handle as needed...
>
> Check to see if anything that fs is depending on is blocking on info
> retrieval like the databases or other areas...
>
> K
>
> On 11/25/12 9:51 AM, "Abaci" <abaci64 at gmail.com> wrote:
>
>> you mentioned that you use xml_curl, if your web server hangs it may
>> hang sofia, iirc sofia is running a single thread and it will wait for
>> the xml_curl response before continuing to the next request.
>>
>> On 11/23/2012 1:13 PM, Steven Ayre wrote:
>>> Any kind of DB backup running? Or any long-running queries (innotop is
>>> great for highlighting queries that've been running a while, including
>>> on non-innodb tables).
>>>
>>> A global read lock, or queries waiting for a lock could block a db
>>> update from the sofia profile thread but still allow read-only queries
>>> (sofia status) to run.
>>>
>>> -Steve
>>>
>>>
>>>
>>> On 23 November 2012 15:32, Tim St. Pierre <fs-list at communicatefreely.net>
>>> wrote:
>>>> Hi Steven,
>>>>
>>>> Thanks for the suggestions. I'm hoping once I get the upgrade done it will
>>>> all go away.
>>>> I have watched it happen at least once. I was on the phone at the time.
>>>> Console activity
>>>> more or less stopped, except for a few calls hanging up. The console
>>>> remains responsive,
>>>> and my call wasn't dropped for at least a minute or two (media timeout?). I
>>>> was able to
>>>> run sofia status and other commands that use the database, so I'm assuming
>>>> that the
>>>> connection was still working. All our media is runs through the box, so I
>>>> think things
>>>> are fine on the Ethernet level. I do see higher load averages - maybe 3-4,
>>>> but that's the
>>>> only obvious indication. It's not taking CPU beyond 10% or so.
>>>>
>>>> We are using MySQL as the core DB and also as the DB backend for each sofia
>>>> profile. This
>>>> is connecting through ODBC of course.
>>>>
>>>> If I can get the other kinks worked out, then I will try 1.2 stable in
>>>> production and
>>>> we'll see how it goes.
>>>>
>>>> -Tim
>>>>
>>>> Steven Ayre wrote:
>>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>>>
>>>>>
>>>>> As you've already acknowledged it's a very old version.
>>>>>
>>>>> It's possible that your issue has already been found and fixed, but if
>>>>> it hasn't then the code will have changed significantly since then and
>>>>> you'd really need to reproduce it on the latest code for it to be
>>>>> investigated.
>>>>>
>>>>>
>>>>> As some general thoughts though, are you able to spot it happening while
>>>>> it's happening or only afterwards?
>>>>>
>>>>> If you're able to get on the system during one of those times look at
>>>>> what else is happening. Is the load average/cpu usage/io high? Perhaps
>>>>> something's running that's blocking all access or causing very high IO.
>>>>>
>>>>> What DB backend are you using for Sofia? Is it possible that that's
>>>>> hanging for a moment? For example if you're running a backup on the DB
>>>>> that blocks all writes to the DB while Sofia is trying to update the DB
>>>>> that perhaps would cause this.
>>>>>
>>>>> Try running a SIP OPTIONS ping your your sofia profile from the
>>>>> localhost during that time, which should exclude it being any issue on
>>>>> the ethernet.
>>>>>
>>>>> -Steve
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 22 November 2012 19:31, Tim St. Pierre <fs-list at communicatefreely.net
>>>>> <mailto:fs-list at communicatefreely.net>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'm having a bit of an odd problem.
>>>>>
>>>>> Intermittently, often every 2-3 days or so, Freeswitch stops
>>>>> replying to SIP for about 5
>>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to
>>>>> be pretty close.
>>>>>
>>>>> During this time, no new registrations or invites can happen, but
>>>>> existing calls stay
>>>>> connected for at least a minute or two. In the logs, you can see
>>>>> calls slowly hanging up
>>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts up again
>>>>> with no word about it at
>>>>> all in the logs.
>>>>>
>>>>> When calls resume, I notice that the number of sessions returned by
>>>>> the status command is
>>>>> one higher than the actual number sessions returned by show
>>>>> channels, or by looking in the
>>>>> database. Every time this happens, the discrepancy increases by one.
>>>>>
>>>>> The interruption happens on all SIP profiles, but calls originated
>>>>> from the socket API
>>>>> still work, insofar as they return with PROGRESS_TIMEOUT since the
>>>>> profiles are still
>>>>> running, but stuck.
>>>>>
>>>>> We are using ODBC/MySQL for the core database, and the database
>>>>> server only runs this
>>>>> database and some basic PHP/xml-curl stuff.
>>>>>
>>>>> We have 416 endpoints registered, and usually sit at about 30
>>>>> sessions during the day.
>>>>>
>>>>> This never happens at night, only during busier times, but not
>>>>> necessarily busy hour.
>>>>>
>>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
>>>>>
>>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>>>
>>>>> Yes, I know it's old and I'm trying to upgrade, but I'm still having
>>>>> some problems getting
>>>>> all my phones to work properly with 1.2 stable. This is a
>>>>> production system, so I can't
>>>>> just blindly put out the newest release. Mostly, I need to buy
>>>>> myself some time so that I
>>>>> can get the kinks worked out of the latest version and then upgrade
>>>>> the production box.
>>>>>
>>>>> I'm grateful for any insights as to what could be happening, even if
>>>>> a solution is just a
>>>>> temporary workaround.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -Tim
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list