[Freeswitch-users] Sofia freezing for 5 minutes then starting again
Anthony Minessale
anthony.minessale at gmail.com
Thu Nov 29 23:50:42 MSK 2012
cliff notes version would be
gcore `cat /usr/local/freeswitch/log/freeswitch.pid`
Take the resulting core file and follow the backtrace instructions at
http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#Getting_a_Backtrace
On Thu, Nov 29, 2012 at 12:09 PM, Tim St. Pierre <
fs-list at communicatefreely.net> wrote:
> It's not a matter of not taking your suggestion, it's an issue of not
> having had the time
> yet to learn how gcore works and put it in place on a production system.
> It's on my
> "to-do" list, but I thought I would at least update the thread, hoping
> maybe that message
> meant something to someone.
>
>
>
> Anthony Minessale wrote:
> > too bad you did not gcore it like I suggested, otherwise who knows...
> > That's just the message it prints when the message parsing thread goes
> up.
> >
> >
> >
> > On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre
> > <fs-list at communicatefreely.net <mailto:fs-list at communicatefreely.net>>
> > wrote:
> >
> > Thanks for all the input everyone!
> >
> > I just had it happen again, this time at night when there were only
> > 2 sessions active!
> >
> > I was in doing some other work, so I managed to see it happen and
> > found a few interesting
> > things.
> >
> > CPU load next to nothing, load average around 0.12.
> > I used iSQL to test ODBC connectivity, and I could read and write to
> > the freeswitch
> > database using the same DSN that freeswitch is using.
> >
> > I could delete a SIP registration from the API
> >
> > Since I had things on a higher debug level, I saw this:
> >
> > freeswitch at stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144
> > MSG Thread Started
> >
> > As soon as that came up, everything started working again.
> >
> > What's that about?
> >
> > -Tim
> >
> > Ken Rice wrote:
> > > Sofia is not single threaded except for in one spot deep in
> libsofia,
> > >>From there, messages are handed off to a number of message queues
> > for FS
> > > core to handle as needed...
> > >
> > > Check to see if anything that fs is depending on is blocking on
> info
> > > retrieval like the databases or other areas...
> > >
> > > K
> > >
> > > On 11/25/12 9:51 AM, "Abaci" <abaci64 at gmail.com
> > <mailto:abaci64 at gmail.com>> wrote:
> > >
> > >> you mentioned that you use xml_curl, if your web server hangs it
> may
> > >> hang sofia, iirc sofia is running a single thread and it will
> > wait for
> > >> the xml_curl response before continuing to the next request.
> > >>
> > >> On 11/23/2012 1:13 PM, Steven Ayre wrote:
> > >>> Any kind of DB backup running? Or any long-running queries
> > (innotop is
> > >>> great for highlighting queries that've been running a while,
> > including
> > >>> on non-innodb tables).
> > >>>
> > >>> A global read lock, or queries waiting for a lock could block a
> db
> > >>> update from the sofia profile thread but still allow read-only
> > queries
> > >>> (sofia status) to run.
> > >>>
> > >>> -Steve
> > >>>
> > >>>
> > >>>
> > >>> On 23 November 2012 15:32, Tim St. Pierre
> > <fs-list at communicatefreely.net <mailto:fs-list at communicatefreely.net
> >>
> > >>> wrote:
> > >>>> Hi Steven,
> > >>>>
> > >>>> Thanks for the suggestions. I'm hoping once I get the upgrade
> > done it will
> > >>>> all go away.
> > >>>> I have watched it happen at least once. I was on the phone at
> > the time.
> > >>>> Console activity
> > >>>> more or less stopped, except for a few calls hanging up. The
> > console
> > >>>> remains responsive,
> > >>>> and my call wasn't dropped for at least a minute or two (media
> > timeout?). I
> > >>>> was able to
> > >>>> run sofia status and other commands that use the database, so
> > I'm assuming
> > >>>> that the
> > >>>> connection was still working. All our media is runs through
> > the box, so I
> > >>>> think things
> > >>>> are fine on the Ethernet level. I do see higher load averages
> > - maybe 3-4,
> > >>>> but that's the
> > >>>> only obvious indication. It's not taking CPU beyond 10% or so.
> > >>>>
> > >>>> We are using MySQL as the core DB and also as the DB backend
> > for each sofia
> > >>>> profile. This
> > >>>> is connecting through ODBC of course.
> > >>>>
> > >>>> If I can get the other kinks worked out, then I will try 1.2
> > stable in
> > >>>> production and
> > >>>> we'll see how it goes.
> > >>>>
> > >>>> -Tim
> > >>>>
> > >>>> Steven Ayre wrote:
> > >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20
> > -0500)
> > >>>>>
> > >>>>>
> > >>>>> As you've already acknowledged it's a very old version.
> > >>>>>
> > >>>>> It's possible that your issue has already been found and
> > fixed, but if
> > >>>>> it hasn't then the code will have changed significantly since
> > then and
> > >>>>> you'd really need to reproduce it on the latest code for it
> to be
> > >>>>> investigated.
> > >>>>>
> > >>>>>
> > >>>>> As some general thoughts though, are you able to spot it
> > happening while
> > >>>>> it's happening or only afterwards?
> > >>>>>
> > >>>>> If you're able to get on the system during one of those times
> > look at
> > >>>>> what else is happening. Is the load average/cpu usage/io
> > high? Perhaps
> > >>>>> something's running that's blocking all access or causing
> > very high IO.
> > >>>>>
> > >>>>> What DB backend are you using for Sofia? Is it possible that
> > that's
> > >>>>> hanging for a moment? For example if you're running a backup
> > on the DB
> > >>>>> that blocks all writes to the DB while Sofia is trying to
> > update the DB
> > >>>>> that perhaps would cause this.
> > >>>>>
> > >>>>> Try running a SIP OPTIONS ping your your sofia profile from
> the
> > >>>>> localhost during that time, which should exclude it being any
> > issue on
> > >>>>> the ethernet.
> > >>>>>
> > >>>>> -Steve
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On 22 November 2012 19:31, Tim St. Pierre
> > <fs-list at communicatefreely.net <mailto:fs-list at communicatefreely.net
> >
> > >>>>> <mailto:fs-list at communicatefreely.net
> > <mailto:fs-list at communicatefreely.net>>> wrote:
> > >>>>>
> > >>>>> Hello,
> > >>>>>
> > >>>>> I'm having a bit of an odd problem.
> > >>>>>
> > >>>>> Intermittently, often every 2-3 days or so, Freeswitch
> stops
> > >>>>> replying to SIP for about 5
> > >>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but
> > it seems to
> > >>>>> be pretty close.
> > >>>>>
> > >>>>> During this time, no new registrations or invites can
> > happen, but
> > >>>>> existing calls stay
> > >>>>> connected for at least a minute or two. In the logs,
> > you can see
> > >>>>> calls slowly hanging up
> > >>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts
> > up again
> > >>>>> with no word about it at
> > >>>>> all in the logs.
> > >>>>>
> > >>>>> When calls resume, I notice that the number of sessions
> > returned by
> > >>>>> the status command is
> > >>>>> one higher than the actual number sessions returned by
> show
> > >>>>> channels, or by looking in the
> > >>>>> database. Every time this happens, the discrepancy
> > increases by one.
> > >>>>>
> > >>>>> The interruption happens on all SIP profiles, but calls
> > originated
> > >>>>> from the socket API
> > >>>>> still work, insofar as they return with PROGRESS_TIMEOUT
> > since the
> > >>>>> profiles are still
> > >>>>> running, but stuck.
> > >>>>>
> > >>>>> We are using ODBC/MySQL for the core database, and the
> > database
> > >>>>> server only runs this
> > >>>>> database and some basic PHP/xml-curl stuff.
> > >>>>>
> > >>>>> We have 416 endpoints registered, and usually sit at
> > about 30
> > >>>>> sessions during the day.
> > >>>>>
> > >>>>> This never happens at night, only during busier times,
> > but not
> > >>>>> necessarily busy hour.
> > >>>>>
> > >>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores,
> > 4G ram)
> > >>>>>
> > >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20
> > -0500)
> > >>>>>
> > >>>>> Yes, I know it's old and I'm trying to upgrade, but I'm
> > still having
> > >>>>> some problems getting
> > >>>>> all my phones to work properly with 1.2 stable. This is
> a
> > >>>>> production system, so I can't
> > >>>>> just blindly put out the newest release. Mostly, I need
> > to buy
> > >>>>> myself some time so that I
> > >>>>> can get the kinks worked out of the latest version and
> > then upgrade
> > >>>>> the production box.
> > >>>>>
> > >>>>> I'm grateful for any insights as to what could be
> > happening, even if
> > >>>>> a solution is just a
> > >>>>> temporary workaround.
> > >>>>>
> > >>>>> Thanks!
> > >>>>>
> > >>>>> -Tim
> > >>>>>
> > >>>>>
> > >>>>>
> >
> _________________________________________________________________________
> > >>>>> Professional FreeSWITCH Consulting Services:
> > >>>>> consulting at freeswitch.org
> > <mailto:consulting at freeswitch.org> <mailto:consulting at freeswitch.org
> > <mailto:consulting at freeswitch.org>>
> > >>>>> http://www.freeswitchsolutions.com
> > >>>>>
> > >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication
> Server
> > >>>>>
> > >>>>>
> > >>>>> Official FreeSWITCH Sites
> > >>>>> http://www.freeswitch.org
> > >>>>> http://wiki.freeswitch.org
> > >>>>> http://www.cluecon.com
> > >>>>>
> > >>>>> FreeSWITCH-users mailing list
> > >>>>> FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > >>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>>
> > >>>>>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > >>>>>
> > >>>>>
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > >>>>> http://www.freeswitch.org
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> >
> ------------------------------------------------------------------------
> > >>>>>
> > >>>>>
> >
> _________________________________________________________________________
> > >>>>> Professional FreeSWITCH Consulting Services:
> > >>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > >>>>> http://www.freeswitchsolutions.com
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Official FreeSWITCH Sites
> > >>>>> http://www.freeswitch.org
> > >>>>> http://wiki.freeswitch.org
> > >>>>> http://www.cluecon.com
> > >>>>>
> > >>>>> FreeSWITCH-users mailing list
> > >>>>> FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > >>>>>
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > >>>>> http://www.freeswitch.org
> > >>>>
> >
> _________________________________________________________________________
> > >>>> Professional FreeSWITCH Consulting Services:
> > >>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > >>>> http://www.freeswitchsolutions.com
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> Official FreeSWITCH Sites
> > >>>> http://www.freeswitch.org
> > >>>> http://wiki.freeswitch.org
> > >>>> http://www.cluecon.com
> > >>>>
> > >>>> FreeSWITCH-users mailing list
> > >>>> FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > >>>>
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > >>>> http://www.freeswitch.org
> > >>>
> >
> _________________________________________________________________________
> > >>> Professional FreeSWITCH Consulting Services:
> > >>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > >>> http://www.freeswitchsolutions.com
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Official FreeSWITCH Sites
> > >>> http://www.freeswitch.org
> > >>> http://wiki.freeswitch.org
> > >>> http://www.cluecon.com
> > >>>
> > >>> FreeSWITCH-users mailing list
> > >>> FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > >>>
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > >>> http://www.freeswitch.org
> > >>
> > >>
> >
> _________________________________________________________________________
> > >> Professional FreeSWITCH Consulting Services:
> > >> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > >> http://www.freeswitchsolutions.com
> > >>
> > >>
> > >>
> > >>
> > >> Official FreeSWITCH Sites
> > >> http://www.freeswitch.org
> > >> http://wiki.freeswitch.org
> > >> http://www.cluecon.com
> > >>
> > >> FreeSWITCH-users mailing list
> > >> FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > >>
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > >> http://www.freeswitch.org
> > >
> >
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> > --
> > Anthony Minessale II
> >
> > FreeSWITCH http://www.freeswitch.org/
> > ClueCon http://www.cluecon.com/
> > Twitter: http://twitter.com/FreeSWITCH_wire
> >
> > AIM: anthm
> > MSN:anthony_minessale at hotmail.com
> > <mailto:MSN%3Aanthony_minessale at hotmail.com>
> > GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> > <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> > IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
> >
> > FreeSWITCH Developer Conference
> > sip:888 at conference.freeswitch.org
> > <mailto:sip%3A888 at conference.freeswitch.org>
> > googletalk:conf+888 at conference.freeswitch.org
> > <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> > pstn:+19193869900
> >
> >
> > ------------------------------------------------------------------------
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
googletalk:conf+888 at conference.freeswitch.org
pstn:+19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20121129/d00bfb37/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list