[Freeswitch-users] Sofia freezing for 5 minutes then starting again
Anthony Minessale
anthony.minessale at gmail.com
Thu Nov 29 10:34:00 MSK 2012
too bad you did not gcore it like I suggested, otherwise who knows...
That's just the message it prints when the message parsing thread goes up.
On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre <
fs-list at communicatefreely.net> wrote:
> Thanks for all the input everyone!
>
> I just had it happen again, this time at night when there were only 2
> sessions active!
>
> I was in doing some other work, so I managed to see it happen and found a
> few interesting
> things.
>
> CPU load next to nothing, load average around 0.12.
> I used iSQL to test ODBC connectivity, and I could read and write to the
> freeswitch
> database using the same DSN that freeswitch is using.
>
> I could delete a SIP registration from the API
>
> Since I had things on a higher debug level, I saw this:
>
> freeswitch at stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG
> Thread Started
>
> As soon as that came up, everything started working again.
>
> What's that about?
>
> -Tim
>
> Ken Rice wrote:
> > Sofia is not single threaded except for in one spot deep in libsofia,
> >>From there, messages are handed off to a number of message queues for FS
> > core to handle as needed...
> >
> > Check to see if anything that fs is depending on is blocking on info
> > retrieval like the databases or other areas...
> >
> > K
> >
> > On 11/25/12 9:51 AM, "Abaci" <abaci64 at gmail.com> wrote:
> >
> >> you mentioned that you use xml_curl, if your web server hangs it may
> >> hang sofia, iirc sofia is running a single thread and it will wait for
> >> the xml_curl response before continuing to the next request.
> >>
> >> On 11/23/2012 1:13 PM, Steven Ayre wrote:
> >>> Any kind of DB backup running? Or any long-running queries (innotop is
> >>> great for highlighting queries that've been running a while, including
> >>> on non-innodb tables).
> >>>
> >>> A global read lock, or queries waiting for a lock could block a db
> >>> update from the sofia profile thread but still allow read-only queries
> >>> (sofia status) to run.
> >>>
> >>> -Steve
> >>>
> >>>
> >>>
> >>> On 23 November 2012 15:32, Tim St. Pierre <
> fs-list at communicatefreely.net>
> >>> wrote:
> >>>> Hi Steven,
> >>>>
> >>>> Thanks for the suggestions. I'm hoping once I get the upgrade done
> it will
> >>>> all go away.
> >>>> I have watched it happen at least once. I was on the phone at the
> time.
> >>>> Console activity
> >>>> more or less stopped, except for a few calls hanging up. The console
> >>>> remains responsive,
> >>>> and my call wasn't dropped for at least a minute or two (media
> timeout?). I
> >>>> was able to
> >>>> run sofia status and other commands that use the database, so I'm
> assuming
> >>>> that the
> >>>> connection was still working. All our media is runs through the box,
> so I
> >>>> think things
> >>>> are fine on the Ethernet level. I do see higher load averages -
> maybe 3-4,
> >>>> but that's the
> >>>> only obvious indication. It's not taking CPU beyond 10% or so.
> >>>>
> >>>> We are using MySQL as the core DB and also as the DB backend for each
> sofia
> >>>> profile. This
> >>>> is connecting through ODBC of course.
> >>>>
> >>>> If I can get the other kinks worked out, then I will try 1.2 stable in
> >>>> production and
> >>>> we'll see how it goes.
> >>>>
> >>>> -Tim
> >>>>
> >>>> Steven Ayre wrote:
> >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
> >>>>>
> >>>>>
> >>>>> As you've already acknowledged it's a very old version.
> >>>>>
> >>>>> It's possible that your issue has already been found and fixed, but
> if
> >>>>> it hasn't then the code will have changed significantly since then
> and
> >>>>> you'd really need to reproduce it on the latest code for it to be
> >>>>> investigated.
> >>>>>
> >>>>>
> >>>>> As some general thoughts though, are you able to spot it happening
> while
> >>>>> it's happening or only afterwards?
> >>>>>
> >>>>> If you're able to get on the system during one of those times look at
> >>>>> what else is happening. Is the load average/cpu usage/io high?
> Perhaps
> >>>>> something's running that's blocking all access or causing very high
> IO.
> >>>>>
> >>>>> What DB backend are you using for Sofia? Is it possible that that's
> >>>>> hanging for a moment? For example if you're running a backup on the
> DB
> >>>>> that blocks all writes to the DB while Sofia is trying to update the
> DB
> >>>>> that perhaps would cause this.
> >>>>>
> >>>>> Try running a SIP OPTIONS ping your your sofia profile from the
> >>>>> localhost during that time, which should exclude it being any issue
> on
> >>>>> the ethernet.
> >>>>>
> >>>>> -Steve
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 22 November 2012 19:31, Tim St. Pierre <
> fs-list at communicatefreely.net
> >>>>> <mailto:fs-list at communicatefreely.net>> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm having a bit of an odd problem.
> >>>>>
> >>>>> Intermittently, often every 2-3 days or so, Freeswitch stops
> >>>>> replying to SIP for about 5
> >>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but it
> seems to
> >>>>> be pretty close.
> >>>>>
> >>>>> During this time, no new registrations or invites can happen,
> but
> >>>>> existing calls stay
> >>>>> connected for at least a minute or two. In the logs, you can
> see
> >>>>> calls slowly hanging up
> >>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts up
> again
> >>>>> with no word about it at
> >>>>> all in the logs.
> >>>>>
> >>>>> When calls resume, I notice that the number of sessions
> returned by
> >>>>> the status command is
> >>>>> one higher than the actual number sessions returned by show
> >>>>> channels, or by looking in the
> >>>>> database. Every time this happens, the discrepancy increases
> by one.
> >>>>>
> >>>>> The interruption happens on all SIP profiles, but calls
> originated
> >>>>> from the socket API
> >>>>> still work, insofar as they return with PROGRESS_TIMEOUT since
> the
> >>>>> profiles are still
> >>>>> running, but stuck.
> >>>>>
> >>>>> We are using ODBC/MySQL for the core database, and the database
> >>>>> server only runs this
> >>>>> database and some basic PHP/xml-curl stuff.
> >>>>>
> >>>>> We have 416 endpoints registered, and usually sit at about 30
> >>>>> sessions during the day.
> >>>>>
> >>>>> This never happens at night, only during busier times, but not
> >>>>> necessarily busy hour.
> >>>>>
> >>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
> >>>>>
> >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
> >>>>>
> >>>>> Yes, I know it's old and I'm trying to upgrade, but I'm still
> having
> >>>>> some problems getting
> >>>>> all my phones to work properly with 1.2 stable. This is a
> >>>>> production system, so I can't
> >>>>> just blindly put out the newest release. Mostly, I need to buy
> >>>>> myself some time so that I
> >>>>> can get the kinks worked out of the latest version and then
> upgrade
> >>>>> the production box.
> >>>>>
> >>>>> I'm grateful for any insights as to what could be happening,
> even if
> >>>>> a solution is just a
> >>>>> temporary workaround.
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> -Tim
> >>>>>
> >>>>>
> >>>>>
> _________________________________________________________________________
> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >>>>> http://www.freeswitchsolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Official FreeSWITCH Sites
> >>>>> http://www.freeswitch.org
> >>>>> http://wiki.freeswitch.org
> >>>>> http://www.cluecon.com
> >>>>>
> >>>>> FreeSWITCH-users mailing list
> >>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>
> >>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> ------------------------------------------------------------------------
> >>>>>
> >>>>>
> _________________________________________________________________________
> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>> consulting at freeswitch.org
> >>>>> http://www.freeswitchsolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Official FreeSWITCH Sites
> >>>>> http://www.freeswitch.org
> >>>>> http://wiki.freeswitch.org
> >>>>> http://www.cluecon.com
> >>>>>
> >>>>> FreeSWITCH-users mailing list
> >>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>
> _________________________________________________________________________
> >>>> Professional FreeSWITCH Consulting Services:
> >>>> consulting at freeswitch.org
> >>>> http://www.freeswitchsolutions.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Official FreeSWITCH Sites
> >>>> http://www.freeswitch.org
> >>>> http://wiki.freeswitch.org
> >>>> http://www.cluecon.com
> >>>>
> >>>> FreeSWITCH-users mailing list
> >>>> FreeSWITCH-users at lists.freeswitch.org
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>> http://www.freeswitch.org
> >>>
> _________________________________________________________________________
> >>> Professional FreeSWITCH Consulting Services:
> >>> consulting at freeswitch.org
> >>> http://www.freeswitchsolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> Official FreeSWITCH Sites
> >>> http://www.freeswitch.org
> >>> http://wiki.freeswitch.org
> >>> http://www.cluecon.com
> >>>
> >>> FreeSWITCH-users mailing list
> >>> FreeSWITCH-users at lists.freeswitch.org
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org
> >> http://www.freeswitchsolutions.com
> >>
> >>
> >>
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://wiki.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
googletalk:conf+888 at conference.freeswitch.org
pstn:+19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20121129/e0e414a9/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list