cliff notes version would be gcore `cat /usr/local/freeswitch/log/freeswitch.pid` Take the resulting core file and follow the backtrace instructions at http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#Getting_a_Backtrace On Thu, Nov 29, 2012 at 12:09 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
It's not a matter of not taking your suggestion, it's an issue of not having had the time yet to learn how gcore works and put it in place on a production system. It's on my "to-do" list, but I thought I would at least update the thread, hoping maybe that message meant something to someone.
Anthony Minessale wrote:
too bad you did not gcore it like I suggested, otherwise who knows... That's just the message it prints when the message parsing thread goes up.
On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Thanks for all the input everyone!
I just had it happen again, this time at night when there were only 2 sessions active!
I was in doing some other work, so I managed to see it happen and found a few interesting things.
CPU load next to nothing, load average around 0.12. I used iSQL to test ODBC connectivity, and I could read and write to the freeswitch database using the same DSN that freeswitch is using.
I could delete a SIP registration from the API
Since I had things on a higher debug level, I saw this:
freeswitch@stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG Thread Started
As soon as that came up, everything started working again.
What's that about?
-Tim
Ken Rice wrote: > Sofia is not single threaded except for in one spot deep in libsofia, >>From there, messages are handed off to a number of message queues for FS > core to handle as needed... > > Check to see if anything that fs is depending on is blocking on info > retrieval like the databases or other areas... > > K > > On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com <mailto:abaci64@gmail.com>> wrote: > >> you mentioned that you use xml_curl, if your web server hangs it may >> hang sofia, iirc sofia is running a single thread and it will wait for >> the xml_curl response before continuing to the next request. >> >> On 11/23/2012 1:13 PM, Steven Ayre wrote: >>> Any kind of DB backup running? Or any long-running queries (innotop is >>> great for highlighting queries that've been running a while, including >>> on non-innodb tables). >>> >>> A global read lock, or queries waiting for a lock could block a db >>> update from the sofia profile thread but still allow read-only queries >>> (sofia status) to run. >>> >>> -Steve >>> >>> >>> >>> On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net
>>> wrote: >>>> Hi Steven, >>>> >>>> Thanks for the suggestions. I'm hoping once I get the upgrade done it will >>>> all go away. >>>> I have watched it happen at least once. I was on the phone at the time. >>>> Console activity >>>> more or less stopped, except for a few calls hanging up. The console >>>> remains responsive, >>>> and my call wasn't dropped for at least a minute or two (media timeout?). I >>>> was able to >>>> run sofia status and other commands that use the database, so I'm assuming >>>> that the >>>> connection was still working. All our media is runs through the box, so I >>>> think things >>>> are fine on the Ethernet level. I do see higher load averages - maybe 3-4, >>>> but that's the >>>> only obvious indication. It's not taking CPU beyond 10% or so. >>>> >>>> We are using MySQL as the core DB and also as the DB backend for each sofia >>>> profile. This >>>> is connecting through ODBC of course. >>>> >>>> If I can get the other kinks worked out, then I will try 1.2 stable in >>>> production and >>>> we'll see how it goes. >>>> >>>> -Tim >>>> >>>> Steven Ayre wrote: >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500) >>>>> >>>>> >>>>> As you've already acknowledged it's a very old version. >>>>> >>>>> It's possible that your issue has already been found and fixed, but if >>>>> it hasn't then the code will have changed significantly since then and >>>>> you'd really need to reproduce it on the latest code for it to be >>>>> investigated. >>>>> >>>>> >>>>> As some general thoughts though, are you able to spot it happening while >>>>> it's happening or only afterwards? >>>>> >>>>> If you're able to get on the system during one of those times look at >>>>> what else is happening. Is the load average/cpu usage/io high? Perhaps >>>>> something's running that's blocking all access or causing very high IO. >>>>> >>>>> What DB backend are you using for Sofia? Is it possible that that's >>>>> hanging for a moment? For example if you're running a backup on the DB >>>>> that blocks all writes to the DB while Sofia is trying to update the DB >>>>> that perhaps would cause this. >>>>> >>>>> Try running a SIP OPTIONS ping your your sofia profile from the >>>>> localhost during that time, which should exclude it being any issue on >>>>> the ethernet. >>>>> >>>>> -Steve >>>>> >>>>> >>>>> >>>>> >>>>> On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net
>>>>> <mailto:fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I'm having a bit of an odd problem. >>>>> >>>>> Intermittently, often every 2-3 days or so, Freeswitch stops >>>>> replying to SIP for about 5 >>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to >>>>> be pretty close. >>>>> >>>>> During this time, no new registrations or invites can happen, but >>>>> existing calls stay >>>>> connected for at least a minute or two. In the logs, you can see >>>>> calls slowly hanging up >>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts up again >>>>> with no word about it at >>>>> all in the logs. >>>>> >>>>> When calls resume, I notice that the number of sessions returned by >>>>> the status command is >>>>> one higher than the actual number sessions returned by show >>>>> channels, or by looking in the >>>>> database. Every time this happens, the discrepancy increases by one. >>>>> >>>>> The interruption happens on all SIP profiles, but calls originated >>>>> from the socket API >>>>> still work, insofar as they return with PROGRESS_TIMEOUT since the >>>>> profiles are still >>>>> running, but stuck. >>>>> >>>>> We are using ODBC/MySQL for the core database, and the database >>>>> server only runs this >>>>> database and some basic PHP/xml-curl stuff. >>>>> >>>>> We have 416 endpoints registered, and usually sit at about 30 >>>>> sessions during the day. >>>>> >>>>> This never happens at night, only during busier times, but not >>>>> necessarily busy hour. >>>>> >>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram) >>>>> >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500) >>>>> >>>>> Yes, I know it's old and I'm trying to upgrade, but I'm still having >>>>> some problems getting >>>>> all my phones to work properly with 1.2 stable. This is a >>>>> production system, so I can't >>>>> just blindly put out the newest release. Mostly, I need to buy >>>>> myself some time so that I >>>>> can get the kinks worked out of the latest version and then upgrade >>>>> the production box. >>>>> >>>>> I'm grateful for any insights as to what could be happening, even if >>>>> a solution is just a >>>>> temporary workaround. >>>>> >>>>> Thanks! >>>>> >>>>> -Tim >>>>> >>>>> >>>>>
_________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication
Server
>>>>> http://www.cudatel.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://wiki.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>>> <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> >>>>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org >>>>> >>>>> >>>>> >>>>>
------------------------------------------------------------------------
>>>>> >>>>>
_________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>> http://www.cudatel.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://wiki.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org >>>>
_________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>>> http://www.freeswitchsolutions.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://wiki.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org >>>
_________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org >> >>
_________________________________________________________________________
>> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900