[Freeswitch-users] Sofia freezing for 5 minutes then starting again
Tim St. Pierre
fs-list at communicatefreely.net
Thu Nov 29 21:09:02 MSK 2012
It's not a matter of not taking your suggestion, it's an issue of not having had the time
yet to learn how gcore works and put it in place on a production system. It's on my
"to-do" list, but I thought I would at least update the thread, hoping maybe that message
meant something to someone.
Anthony Minessale wrote:
> too bad you did not gcore it like I suggested, otherwise who knows...
> That's just the message it prints when the message parsing thread goes up.
>
>
>
> On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre
> <fs-list at communicatefreely.net <mailto:fs-list at communicatefreely.net>>
> wrote:
>
> Thanks for all the input everyone!
>
> I just had it happen again, this time at night when there were only
> 2 sessions active!
>
> I was in doing some other work, so I managed to see it happen and
> found a few interesting
> things.
>
> CPU load next to nothing, load average around 0.12.
> I used iSQL to test ODBC connectivity, and I could read and write to
> the freeswitch
> database using the same DSN that freeswitch is using.
>
> I could delete a SIP registration from the API
>
> Since I had things on a higher debug level, I saw this:
>
> freeswitch at stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144
> MSG Thread Started
>
> As soon as that came up, everything started working again.
>
> What's that about?
>
> -Tim
>
> Ken Rice wrote:
> > Sofia is not single threaded except for in one spot deep in libsofia,
> >>From there, messages are handed off to a number of message queues
> for FS
> > core to handle as needed...
> >
> > Check to see if anything that fs is depending on is blocking on info
> > retrieval like the databases or other areas...
> >
> > K
> >
> > On 11/25/12 9:51 AM, "Abaci" <abaci64 at gmail.com
> <mailto:abaci64 at gmail.com>> wrote:
> >
> >> you mentioned that you use xml_curl, if your web server hangs it may
> >> hang sofia, iirc sofia is running a single thread and it will
> wait for
> >> the xml_curl response before continuing to the next request.
> >>
> >> On 11/23/2012 1:13 PM, Steven Ayre wrote:
> >>> Any kind of DB backup running? Or any long-running queries
> (innotop is
> >>> great for highlighting queries that've been running a while,
> including
> >>> on non-innodb tables).
> >>>
> >>> A global read lock, or queries waiting for a lock could block a db
> >>> update from the sofia profile thread but still allow read-only
> queries
> >>> (sofia status) to run.
> >>>
> >>> -Steve
> >>>
> >>>
> >>>
> >>> On 23 November 2012 15:32, Tim St. Pierre
> <fs-list at communicatefreely.net <mailto:fs-list at communicatefreely.net>>
> >>> wrote:
> >>>> Hi Steven,
> >>>>
> >>>> Thanks for the suggestions. I'm hoping once I get the upgrade
> done it will
> >>>> all go away.
> >>>> I have watched it happen at least once. I was on the phone at
> the time.
> >>>> Console activity
> >>>> more or less stopped, except for a few calls hanging up. The
> console
> >>>> remains responsive,
> >>>> and my call wasn't dropped for at least a minute or two (media
> timeout?). I
> >>>> was able to
> >>>> run sofia status and other commands that use the database, so
> I'm assuming
> >>>> that the
> >>>> connection was still working. All our media is runs through
> the box, so I
> >>>> think things
> >>>> are fine on the Ethernet level. I do see higher load averages
> - maybe 3-4,
> >>>> but that's the
> >>>> only obvious indication. It's not taking CPU beyond 10% or so.
> >>>>
> >>>> We are using MySQL as the core DB and also as the DB backend
> for each sofia
> >>>> profile. This
> >>>> is connecting through ODBC of course.
> >>>>
> >>>> If I can get the other kinks worked out, then I will try 1.2
> stable in
> >>>> production and
> >>>> we'll see how it goes.
> >>>>
> >>>> -Tim
> >>>>
> >>>> Steven Ayre wrote:
> >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20
> -0500)
> >>>>>
> >>>>>
> >>>>> As you've already acknowledged it's a very old version.
> >>>>>
> >>>>> It's possible that your issue has already been found and
> fixed, but if
> >>>>> it hasn't then the code will have changed significantly since
> then and
> >>>>> you'd really need to reproduce it on the latest code for it to be
> >>>>> investigated.
> >>>>>
> >>>>>
> >>>>> As some general thoughts though, are you able to spot it
> happening while
> >>>>> it's happening or only afterwards?
> >>>>>
> >>>>> If you're able to get on the system during one of those times
> look at
> >>>>> what else is happening. Is the load average/cpu usage/io
> high? Perhaps
> >>>>> something's running that's blocking all access or causing
> very high IO.
> >>>>>
> >>>>> What DB backend are you using for Sofia? Is it possible that
> that's
> >>>>> hanging for a moment? For example if you're running a backup
> on the DB
> >>>>> that blocks all writes to the DB while Sofia is trying to
> update the DB
> >>>>> that perhaps would cause this.
> >>>>>
> >>>>> Try running a SIP OPTIONS ping your your sofia profile from the
> >>>>> localhost during that time, which should exclude it being any
> issue on
> >>>>> the ethernet.
> >>>>>
> >>>>> -Steve
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 22 November 2012 19:31, Tim St. Pierre
> <fs-list at communicatefreely.net <mailto:fs-list at communicatefreely.net>
> >>>>> <mailto:fs-list at communicatefreely.net
> <mailto:fs-list at communicatefreely.net>>> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm having a bit of an odd problem.
> >>>>>
> >>>>> Intermittently, often every 2-3 days or so, Freeswitch stops
> >>>>> replying to SIP for about 5
> >>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but
> it seems to
> >>>>> be pretty close.
> >>>>>
> >>>>> During this time, no new registrations or invites can
> happen, but
> >>>>> existing calls stay
> >>>>> connected for at least a minute or two. In the logs,
> you can see
> >>>>> calls slowly hanging up
> >>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts
> up again
> >>>>> with no word about it at
> >>>>> all in the logs.
> >>>>>
> >>>>> When calls resume, I notice that the number of sessions
> returned by
> >>>>> the status command is
> >>>>> one higher than the actual number sessions returned by show
> >>>>> channels, or by looking in the
> >>>>> database. Every time this happens, the discrepancy
> increases by one.
> >>>>>
> >>>>> The interruption happens on all SIP profiles, but calls
> originated
> >>>>> from the socket API
> >>>>> still work, insofar as they return with PROGRESS_TIMEOUT
> since the
> >>>>> profiles are still
> >>>>> running, but stuck.
> >>>>>
> >>>>> We are using ODBC/MySQL for the core database, and the
> database
> >>>>> server only runs this
> >>>>> database and some basic PHP/xml-curl stuff.
> >>>>>
> >>>>> We have 416 endpoints registered, and usually sit at
> about 30
> >>>>> sessions during the day.
> >>>>>
> >>>>> This never happens at night, only during busier times,
> but not
> >>>>> necessarily busy hour.
> >>>>>
> >>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores,
> 4G ram)
> >>>>>
> >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20
> -0500)
> >>>>>
> >>>>> Yes, I know it's old and I'm trying to upgrade, but I'm
> still having
> >>>>> some problems getting
> >>>>> all my phones to work properly with 1.2 stable. This is a
> >>>>> production system, so I can't
> >>>>> just blindly put out the newest release. Mostly, I need
> to buy
> >>>>> myself some time so that I
> >>>>> can get the kinks worked out of the latest version and
> then upgrade
> >>>>> the production box.
> >>>>>
> >>>>> I'm grateful for any insights as to what could be
> happening, even if
> >>>>> a solution is just a
> >>>>> temporary workaround.
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> -Tim
> >>>>>
> >>>>>
> >>>>>
> _________________________________________________________________________
> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>> consulting at freeswitch.org
> <mailto:consulting at freeswitch.org> <mailto:consulting at freeswitch.org
> <mailto:consulting at freeswitch.org>>
> >>>>> http://www.freeswitchsolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Official FreeSWITCH Sites
> >>>>> http://www.freeswitch.org
> >>>>> http://wiki.freeswitch.org
> >>>>> http://www.cluecon.com
> >>>>>
> >>>>> FreeSWITCH-users mailing list
> >>>>> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>>
> >>>>>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>
> >>>>>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> ------------------------------------------------------------------------
> >>>>>
> >>>>>
> _________________________________________________________________________
> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >>>>> http://www.freeswitchsolutions.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Official FreeSWITCH Sites
> >>>>> http://www.freeswitch.org
> >>>>> http://wiki.freeswitch.org
> >>>>> http://www.cluecon.com
> >>>>>
> >>>>> FreeSWITCH-users mailing list
> >>>>> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>
> _________________________________________________________________________
> >>>> Professional FreeSWITCH Consulting Services:
> >>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >>>> http://www.freeswitchsolutions.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Official FreeSWITCH Sites
> >>>> http://www.freeswitch.org
> >>>> http://wiki.freeswitch.org
> >>>> http://www.cluecon.com
> >>>>
> >>>> FreeSWITCH-users mailing list
> >>>> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>> http://www.freeswitch.org
> >>>
> _________________________________________________________________________
> >>> Professional FreeSWITCH Consulting Services:
> >>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >>> http://www.freeswitchsolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> Official FreeSWITCH Sites
> >>> http://www.freeswitch.org
> >>> http://wiki.freeswitch.org
> >>> http://www.cluecon.com
> >>>
> >>> FreeSWITCH-users mailing list
> >>> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >> http://www.freeswitchsolutions.com
> >>
> >>
> >>
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://wiki.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> ------------------------------------------------------------------------
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list