[Freeswitch-users] Sofia freezing for 5 minutes then starting again
Abaci
abaci64 at gmail.com
Sun Nov 25 22:26:54 MSK 2012
See also http://jira.freeswitch.org/browse/FS-3328
On 11/25/2012 1:46 PM, Abaci wrote:
> I just remember reading somewhere on a blog something about a similar
> issue and this was the cause.
> Just found that post
> http://blog.godson.in/2011/06/pitfalls-to-avoid-while-using.html not
> sure that's the issue. either way it's a good idea to set the timeout
> option.
>
> On 11/25/2012 11:32 AM, Ken Rice wrote:
>> Sofia is not single threaded except for in one spot deep in libsofia,
>> >From there, messages are handed off to a number of message queues
>> for FS
>> core to handle as needed...
>>
>> Check to see if anything that fs is depending on is blocking on info
>> retrieval like the databases or other areas...
>>
>> K
>>
>> On 11/25/12 9:51 AM, "Abaci" <abaci64 at gmail.com> wrote:
>>
>>> you mentioned that you use xml_curl, if your web server hangs it may
>>> hang sofia, iirc sofia is running a single thread and it will wait for
>>> the xml_curl response before continuing to the next request.
>>>
>>> On 11/23/2012 1:13 PM, Steven Ayre wrote:
>>>> Any kind of DB backup running? Or any long-running queries (innotop is
>>>> great for highlighting queries that've been running a while, including
>>>> on non-innodb tables).
>>>>
>>>> A global read lock, or queries waiting for a lock could block a db
>>>> update from the sofia profile thread but still allow read-only queries
>>>> (sofia status) to run.
>>>>
>>>> -Steve
>>>>
>>>>
>>>>
>>>> On 23 November 2012 15:32, Tim St. Pierre
>>>> <fs-list at communicatefreely.net>
>>>> wrote:
>>>>> Hi Steven,
>>>>>
>>>>> Thanks for the suggestions. I'm hoping once I get the upgrade
>>>>> done it will
>>>>> all go away.
>>>>> I have watched it happen at least once. I was on the phone at the
>>>>> time.
>>>>> Console activity
>>>>> more or less stopped, except for a few calls hanging up. The console
>>>>> remains responsive,
>>>>> and my call wasn't dropped for at least a minute or two (media
>>>>> timeout?). I
>>>>> was able to
>>>>> run sofia status and other commands that use the database, so I'm
>>>>> assuming
>>>>> that the
>>>>> connection was still working. All our media is runs through the
>>>>> box, so I
>>>>> think things
>>>>> are fine on the Ethernet level. I do see higher load averages -
>>>>> maybe 3-4,
>>>>> but that's the
>>>>> only obvious indication. It's not taking CPU beyond 10% or so.
>>>>>
>>>>> We are using MySQL as the core DB and also as the DB backend for
>>>>> each sofia
>>>>> profile. This
>>>>> is connecting through ODBC of course.
>>>>>
>>>>> If I can get the other kinks worked out, then I will try 1.2
>>>>> stable in
>>>>> production and
>>>>> we'll see how it goes.
>>>>>
>>>>> -Tim
>>>>>
>>>>> Steven Ayre wrote:
>>>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>>>>
>>>>>>
>>>>>> As you've already acknowledged it's a very old version.
>>>>>>
>>>>>> It's possible that your issue has already been found and fixed,
>>>>>> but if
>>>>>> it hasn't then the code will have changed significantly since
>>>>>> then and
>>>>>> you'd really need to reproduce it on the latest code for it to be
>>>>>> investigated.
>>>>>>
>>>>>>
>>>>>> As some general thoughts though, are you able to spot it
>>>>>> happening while
>>>>>> it's happening or only afterwards?
>>>>>>
>>>>>> If you're able to get on the system during one of those times
>>>>>> look at
>>>>>> what else is happening. Is the load average/cpu usage/io high?
>>>>>> Perhaps
>>>>>> something's running that's blocking all access or causing very
>>>>>> high IO.
>>>>>>
>>>>>> What DB backend are you using for Sofia? Is it possible that that's
>>>>>> hanging for a moment? For example if you're running a backup on
>>>>>> the DB
>>>>>> that blocks all writes to the DB while Sofia is trying to update
>>>>>> the DB
>>>>>> that perhaps would cause this.
>>>>>>
>>>>>> Try running a SIP OPTIONS ping your your sofia profile from the
>>>>>> localhost during that time, which should exclude it being any
>>>>>> issue on
>>>>>> the ethernet.
>>>>>>
>>>>>> -Steve
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 22 November 2012 19:31, Tim St. Pierre
>>>>>> <fs-list at communicatefreely.net
>>>>>> <mailto:fs-list at communicatefreely.net>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm having a bit of an odd problem.
>>>>>>
>>>>>> Intermittently, often every 2-3 days or so, Freeswitch stops
>>>>>> replying to SIP for about 5
>>>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but it
>>>>>> seems to
>>>>>> be pretty close.
>>>>>>
>>>>>> During this time, no new registrations or invites can
>>>>>> happen, but
>>>>>> existing calls stay
>>>>>> connected for at least a minute or two. In the logs, you
>>>>>> can see
>>>>>> calls slowly hanging up
>>>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts up
>>>>>> again
>>>>>> with no word about it at
>>>>>> all in the logs.
>>>>>>
>>>>>> When calls resume, I notice that the number of sessions
>>>>>> returned by
>>>>>> the status command is
>>>>>> one higher than the actual number sessions returned by show
>>>>>> channels, or by looking in the
>>>>>> database. Every time this happens, the discrepancy
>>>>>> increases by one.
>>>>>>
>>>>>> The interruption happens on all SIP profiles, but calls
>>>>>> originated
>>>>>> from the socket API
>>>>>> still work, insofar as they return with PROGRESS_TIMEOUT
>>>>>> since the
>>>>>> profiles are still
>>>>>> running, but stuck.
>>>>>>
>>>>>> We are using ODBC/MySQL for the core database, and the
>>>>>> database
>>>>>> server only runs this
>>>>>> database and some basic PHP/xml-curl stuff.
>>>>>>
>>>>>> We have 416 endpoints registered, and usually sit at about 30
>>>>>> sessions during the day.
>>>>>>
>>>>>> This never happens at night, only during busier times, but not
>>>>>> necessarily busy hour.
>>>>>>
>>>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G
>>>>>> ram)
>>>>>>
>>>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
>>>>>>
>>>>>> Yes, I know it's old and I'm trying to upgrade, but I'm
>>>>>> still having
>>>>>> some problems getting
>>>>>> all my phones to work properly with 1.2 stable. This is a
>>>>>> production system, so I can't
>>>>>> just blindly put out the newest release. Mostly, I need to
>>>>>> buy
>>>>>> myself some time so that I
>>>>>> can get the kinks worked out of the latest version and then
>>>>>> upgrade
>>>>>> the production box.
>>>>>>
>>>>>> I'm grateful for any insights as to what could be
>>>>>> happening, even if
>>>>>> a solution is just a
>>>>>> temporary workaround.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> -Tim
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>>
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>> _________________________________________________________________________
>>>>>
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>
>>>>> http://www.freeswitch.org
>>>> _________________________________________________________________________
>>>>
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>
>>>> http://www.freeswitch.org
>>>
>>> _________________________________________________________________________
>>>
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>
>>> http://www.freeswitch.org
>
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list